How to find all pages on a website

No single method finds every page. The sitemap lists what the owner wants indexed; a crawl finds what is actually linked. Use both and compare.

1. Read the sitemap

curl -s https://example.com/robots.txt | grep -i '^sitemap:'
curl -s --compressed https://example.com/sitemap.xml | grep -oE '<loc>[^<]+</loc>' | sed -E 's#</?loc>##g' | sort -u > sitemap-urls.txt

The first line shows the sitemap location from robots.txt (if there is none, try /sitemap.xml and /sitemap_index.xml). The second writes one URL per line. If the file is a sitemap index, its entries are more sitemaps: run the second line on each. Sitemaps can be very incomplete: docs.python.org's lists only 8 URLs, though the documentation has far more pages.

2. Crawl the links

wget --spider -r -l inf -nd -nv -o crawl.log https://example.com/
grep -oE 'URL: ?[^ ]+' crawl.log | sed -E 's/URL: ?//' | grep -vE '\.(css|js|png|jpe?g|gif|svg|ico|xsd|txt|xml)$' | sort -u > crawled-urls.txt

wget follows every link on the same host without saving pages; the second line keeps the page URLs and drops stylesheets, scripts and images. A crawl only finds pages that are linked with plain HTML links: menus built by JavaScript stay invisible to it.

3. Compare the two lists

comm -13 sitemap-urls.txt crawled-urls.txt   # linked, but missing from the sitemap
comm -23 sitemap-urls.txt crawled-urls.txt   # in the sitemap, but not linked (orphan pages?)

On sitemaps.org the sitemap listed 84 URLs and the crawl found 5 pages. The site's 79 translated pages are reachable only through a JavaScript language menu, so only the sitemap knows them. In the other direction, the crawl found /index.html, a second address for the home page that the sitemap leaves out.

4. Other sources

Google: searching site:example.com shows indexed pages, but the count is an estimate and only covers what Google indexed.
Your own site: Search Console's Pages report and your CMS or server logs list pages that neither a crawl nor the sitemap may show.

Big sites, as a spreadsheet

Our Sitemap URL Status Checker on Apify reads sitemaps and sitemap indexes (gzip included) and returns every URL with its HTTP status and redirect target, as CSV, Excel or JSON. To crawl a site and audit each page it reaches, use the SEO Audit Crawler. Both are billed per URL.

Frequently asked questions

How do I see all pages of a website?

Read its sitemap.xml (location in robots.txt), crawl its links, and combine both lists. Each one alone usually misses pages.

What is an orphan page?

A page that exists (often listed in the sitemap) but no other page links to it. Visitors and crawlers that follow links never reach it.

Can I find pages that are not linked anywhere and not in the sitemap?

Not from outside. Only the site owner's CMS, server logs or Search Console can list those.

Does site: search show every page?

No. It only shows pages Google indexed, and the result count is an estimate.

Written with AI assistance. All commands were run against www.sitemaps.org on 2026-10-01 (sitemap: 84 URLs; crawl: 5 pages; both lists compared). The Apify tool is ours; the APIs are free to call directly.