How to check every URL in a sitemap for errors

A sitemap should list only live, final URLs. Here is how to test every entry and what to do with the ones that fail.

1. A shell loop (small sitemaps)

curl -s --compressed https://www.sitemaps.org/sitemap.xml | grep -oE '<loc>[^<]+</loc>' | sed -E 's#</?loc>##g' |
while read -r url; do
  echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"
  sleep 0.5
done

Each line shows the HTTP status and the URL. Add | grep -v '^200' to see only problems. --compressed handles servers that send the file gzip-encoded. If the file is a sitemap index (its <loc> entries point to more sitemaps), run the loop on each child sitemap; a .xml.gz file needs | gunzip after curl.

2. What to do with each status

• 200: fine.
• 301 / 302 / 307 / 308: replace the entry with the final URL. Sitemaps should list only URLs that answer directly.
• 404 / 410: remove the entry, or restore the page if it should exist.
• 403 / 429 / 5xx: may be bot protection or rate limiting rather than a real problem. Recheck slowly, and check the page in a browser.

3. Large sitemaps

A sitemap can hold up to 50,000 URLs, and an index can point to many of them. At one request every half second, 50,000 URLs take about 7 hours. Run the checks in parallel, but gently, and watch for rate limiting.

Our Sitemap URL Status Checker on Apify reads sitemaps and sitemap indexes (gzip included), checks every URL and returns one row per URL with status and redirect target, as CSV, Excel or JSON. To build a new sitemap from a URL list, use the free sitemap generator.

Written with AI assistance. The shell loop was run against sitemaps.org's own sitemap on 2026-10-01. The Apify tool is ours; the APIs are free to call directly.