How to check if a page is noindex (meta robots, X-Robots-Tag, robots.txt)

A page can be kept out of Google in three places: a meta tag, an HTTP header, or robots.txt. Check all three.

1. Meta robots and X-Robots-Tag (shell)

Save as check.sh and run bash check.sh https://example.com/page:

URL="$1"
curl -sL -o /dev/null -D - "$URL" | grep -i '^x-robots-tag'
curl -sL "$URL" | grep -ioE '<meta[^>]*(robots|googlebot)[^>]*>'

The first line prints an X-Robots-Tag header if the server sends one (common for PDFs and other files, where a meta tag is impossible). The second prints every meta tag that mentions robots or googlebot, whatever the attribute order. No output means neither is set, so the page may be indexed. For example, a Wikipedia edit history page returns <meta name="robots" content="noindex,nofollow,max-image-preview:standard">. Google obeys both robots and googlebot; other search engines read robots and their own names (bingbot).

The tag must be in the HTML the server sends. A noindex added later by JavaScript is unreliable, and this check does not see it.

2. robots.txt (Python)

import sys, urllib.error, urllib.parse, urllib.request
from protego import Protego  # pip install protego
url = sys.argv[1]
robots_url = urllib.parse.urljoin(url, "/robots.txt")
req = urllib.request.Request(robots_url, headers={"User-Agent": "Mozilla/5.0 (robots check)"})
try:
    body = urllib.request.urlopen(req).read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
    if e.code != 404:
        raise
    body = ""  # no robots.txt: everything may be crawled
print("Googlebot may crawl:", Protego.parse(body).can_fetch(url, "Googlebot"))

Run it as python3 robots.py URL after pip install protego (Scrapy's robots.txt parser, which follows Google's rules). Do not use Python's built-in urllib.robotparser for this: it ignores * wildcards and applies the first matching rule instead of the longest one. On 2,258 Googlebot checks built from the robots.txt files of Google, GitHub, Wikipedia, Amazon, the New York Times and the BBC, it disagreed with Google's rules 1,205 times; for example it calls en.wikipedia.org/wiki/Category:Noindexed_pages crawlable, while Wikipedia's robots.txt blocks it for Googlebot. Its read() also uses Python's default user agent and treats a 401/403 answer as "everything blocked". The script above fetches robots.txt itself with a browser-like user agent and treats a missing file (404) as "everything allowed". To test rules without code, paste the file into the robots.txt tester.

3. The robots.txt + noindex trap

robots.txt stops crawling, not indexing. If robots.txt blocks a page, Google never fetches it, so it never sees the noindex tag, and the URL can still appear in results (without a description) when other sites link to it. To remove a page from Google, allow crawling and serve noindex. Wikipedia's history pages do both (blocked in robots.txt and noindex), which works for them because they are not meant to be found at all.

Every page of a site at once

Our SEO Audit Crawler on Apify crawls your site and flags each noindex page (meta robots, googlebot meta or X-Robots-Tag) next to 25+ other checks such as canonical, titles and broken links, as CSV, Excel or JSON. Billed per audited page.

Written with AI assistance. Both scripts were run on 2026-10-01 against Wikipedia (a noindex history page, a blocked category page and a normal article), example.com, and a local test server sending a meta noindex with reversed attributes and an X-Robots-Tag header. The Apify tool is ours; the APIs are free to call directly.