How to block AI crawlers (GPTBot, ClaudeBot, …) with robots.txt

AI companies use separate user-agent tokens for their crawlers, so you can block them without touching Google or Bing. Here are two tested variants: block everything AI, or block model training only.

1. Block all AI crawlers

Several User-agent lines in a row form one group, so one Disallow: / covers all of them:

# Block AI training and AI search crawlers; normal search engines keep crawling
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

User-agent: *
Disallow:

Result in both parsers: the 14 AI agents are blocked from every URL; Googlebot, Bingbot, Applebot and DuckDuckBot are allowed. Applebot-Extended does not affect Applebot: a group applies only to the exact product token.

2. Block model training only

# Block model training only; AI search and user-requested fetches may still read the site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow: /admin/

Here only the six training-related tokens are blocked; ChatGPT search, Claude search, Perplexity and the user-requested fetchers can still read the site, which keeps you visible in AI answers that link to sources. The * group still applies its own rules (here /admin/) to everyone else. Note that a bot with its own group ignores the * group entirely.

3. The tokens

OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (pages a user asks ChatGPT to open).
Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user requests). Older lists also name anthropic-ai.
Google: Google-Extended is not a crawler but a control token for use in Gemini models; Googlebot still crawls and Search is unaffected.
Apple: Applebot-Extended, the same kind of control token for Apple's AI training.
Others: CCBot (Common Crawl, a dataset many models train on), PerplexityBot and Perplexity-User, Bytespider (ByteDance), meta-externalagent (Meta), Amazonbot.

Every token above appears in the robots.txt of at least two of nytimes.com, bbc.co.uk, amazon.com and github.com as of 2026-10-01. Operators add new tokens over time; check their crawler documentation now and then.

4. Test before you publish

Paste your file into the free robots.txt tester, set the user agent (GPTBot, Googlebot, …) and list a few URLs. It shows which rule decides each URL, so a typo in a token or a stray Disallow: / in the wrong group shows up before Google sees it.

5. Limits

robots.txt is a request, not access control. Crawlers that ignore it are not stopped; that takes server-side blocking by user agent or IP, or a bot-management service.
Operators describe user-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) as acting for a person, and some say such fetches may not apply robots.txt. Check each operator's documentation.
Blocking stops future crawling. It does not remove content already collected.

Frequently asked questions

Does blocking GPTBot affect my Google ranking?

No. Google Search uses Googlebot, which has its own token. Google-Extended only controls use in Gemini models and does not change Search crawling or ranking.

Where do I put robots.txt?

At the root of each host: https://example.com/robots.txt. A subdomain (blog.example.com) needs its own file.

Do I need a separate Disallow under each User-agent?

No. Consecutive User-agent lines share the rules that follow them, as in the examples above. Repeating the rules per agent also works.

Written with AI assistance. On 2026-10-01 both robots.txt files below were checked for 18 user agents with our robots.txt tester and with Scrapy's Google-compatible parser (protego): identical results. The user-agent tokens were cross-checked against the live robots.txt files of nytimes.com, bbc.co.uk, amazon.com and github.com. The Apify tool is ours; the APIs are free to call directly.