Glovrex
日本語Contact

Journal

111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.

robots.txt on the Tranco top 1,000 on 2026-10-06 (545 files): which AI tokens control training and which control answers, how often each is blocked, and the mistakes real files make, from orphaned legacy tokens to named groups that silently drop the * rules.

Arne Rei8 min read

This piece was first published on dev.to. That is the canonical version.

111 of the top 1,000 sites fully block GPTBot in robots.txt. 51 of those 111 don’t fully block OAI-SearchBot, and 35 of the 51 never mention OAI-SearchBot at all. GPTBot is OpenAI’s training crawler. OAI-SearchBot is the one that decides whether a page can show up in ChatGPT search. So a third of the sites that said “no” to OpenAI said it only to training, and most of them probably didn’t decide that on purpose. I can’t tell from the file. Nobody can, which is the point of this post.

Below: what each AI token actually controls, the numbers from the top 1,000 sites as of today, the mistakes I found in real files, and how to write your own policy so it says what you mean.

Limits first

  • robots.txt is a request. It doesn’t block anything. A crawler that ignores it gets the page. And it says nothing about what your server or CDN does: in an earlier measurement, 19 of 79 small-business sites refused at least one AI crawler at the server, mostly against their own robots.txt. Everything below is about the file, not about what a bot can actually fetch.
  • List: Tranco list 56WKN, generated 2026-10-05, top 1,000. It’s the same list and the same reachable set as my llms.txt count: 742 of the 1,000 domains answered HTTPS. The rest are mostly infrastructure hostnames with no website.
  • When and how: one GET of https://<domain>/robots.txt per domain on 2026-10-06, 09:59 to 10:00 UTC, from one connection in Japan, with a desktop Chrome user agent, redirects followed. A server can hand a different robots.txt to a different user agent. I didn’t test that.
  • One snapshot. These files change, and AI sections change more often than the rest.
  • Matching: RFC 9309 rules. A bot uses the group that names it; only if no group names it does it fall back to User-agent: *. Blank lines are ignored. “Fully blocked” below means disallowed at both / and a deep path. “Partially” means allowed at / but at least one Disallow applies to it.

Of the 742, 545 served a robots.txt file. 144 had none (76 answered 404 or similar, 68 answered 200 with an HTML page, which a parser reads as a file with no rules). 53 I couldn’t read: 45 answered 401 or 403, 8 returned 5xx, 202 or 429. RFC 9309 says a 4xx means “no restrictions” and a 5xx means “assume everything is disallowed”. I count all 53 as unknown.

What each token controls

This is the part most robots.txt files get wrong, so here it is from each vendor’s own page, read today.

Token Owner What disallowing it does
GPTBot OpenAI Opts out of crawling for model training
OAI-SearchBot OpenAI Opts out of ChatGPT search results. OpenAI: “Each setting is independent of the others”
ChatGPT-User OpenAI User-triggered visits. OpenAI says robots.txt rules “may not apply”
ClaudeBot Anthropic Opts out of crawling for training
Claude-SearchBot, Claude-User Anthropic Search quality, and visits on a user’s request
anthropic-ai (Anthropic, older) Not on Anthropic’s current crawler page, which lists only the three above
PerplexityBot Perplexity Opts out of Perplexity’s search index. Perplexity says it is not used for model training
Perplexity-User Perplexity User-triggered fetches. Perplexity says it “generally ignores robots.txt rules”
Google-Extended Google Not a crawler. Controls whether content Google already crawls can train Gemini. Google: it “does not impact a site’s inclusion in Google Search”
Applebot-Extended Apple Not a crawler either. Controls training use of what Applebot fetched; such pages “can still be included in search results”
CCBot Common Crawl Opts out of the open crawl that many training sets are built from
Bytespider ByteDance Training crawler

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, RFC 9309.

The split that matters: training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) versus answer and search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking a training token doesn’t take you out of that company’s answers. Blocking a search token does, or at least asks to. And the user-triggered agents mostly don’t read robots.txt at all.

The numbers

n = 545 robots.txt files, 2026-10-06.

Token Fully blocked of which by name of which via * Partially Allowed
Bytespider 123 88 35 350 72
CCBot 121 88 33 348 76
GPTBot 111 83 28 351 83
ClaudeBot 109 78 31 353 83
Google-Extended 97 68 29 365 83
PerplexityBot 90 61 29 373 82
Applebot-Extended 90 57 33 380 75
anthropic-ai 84 47 37 385 76
OAI-SearchBot 61 35 26 400 84

A few things in there:

  • “Partially” is mostly noise. For GPTBot, 322 of the 351 come from the * group: ordinary rules like Disallow: /search or /cart that apply to every crawler. Those sites have no AI policy at all. They just have a robots.txt.
  • “Via *” means no AI policy either. 49 files have Disallow: / in their * group. Some are ad and CDN hostnames that block everyone. Others are allowlists from big platforms (facebook.com, x.com, reddit.com, netflix.com, yelp.com): block everyone, then name who gets in. For a bot those files don’t name, “via *” is still a decision, just not one about AI.
  • 169 of 545 files (31%) name at least one of the nine tokens. That’s the real count of sites with any written AI policy.
  • OAI-SearchBot is blocked about half as often as GPTBot (61 vs 111). Some of that is a choice. Some of it isn’t, as the next section shows.
  • In the top 100, 55 domains served a file and 13 of those block GPTBot fully, among them amazon.com, x.com, pinterest.com, tiktok.com, yahoo.com and baidu.com.
  • 40 files block all nine tokens fully. 13 block all four training tokens I grouped (GPTBot, Google-Extended, Applebot-Extended, CCBot) and leave both OAI-SearchBot and PerplexityBot open. That second group is the clean “train on nothing, but answer with my pages” policy. It’s rarer than I expected.

Mistakes in real files

Blocking training and, maybe by accident, leaving answers open

Of the 111 sites that fully block GPTBot, 51 don’t fully block OAI-SearchBot.

The reverse happened once. facebook.com ends its file with User-agent: * / Disallow: / and names the bots it allows. GPTBot is named, with path rules. OAI-SearchBot isn’t, so it falls into the block-everything group. Result: training crawler partly in, search crawler out. When your file is an allowlist, every token you forget is blocked.

Blocking a token nobody sends anymore

4 files block anthropic-ai fully and never mention ClaudeBot: chatgpt.com, espn.com, repubblica.it and elmundo.es. ClaudeBot falls back to their * group and is only partly restricted. If the intent was “no Anthropic training”, the file doesn’t say so for the crawler Anthropic currently lists.

Naming a bot replaces the * group

This one surprised me most. Once a group names GPTBot, GPTBot ignores the * group completely. Whatever you disallowed for everyone no longer applies to it.

23 sites give GPTBot its own group without a full block, and that group lacks Disallow rules their * group has. hp.com has User-agent: GPTBot / Allow: /, and with it 170 paths from its * group (order status pages, site search, login forms, /api/) no longer apply to GPTBot. vimeo.com loses 325, kleinanzeigen.de 215, cisco.com 175, wordpress.org 4 (including /wp-admin/). Same pattern for OAI-SearchBot (28 sites) and PerplexityBot (25).

Maybe some of them want exactly that. Most * rules exist to keep crawlers out of cart and search pages, though, and I doubt anyone meant to let AI crawlers in there and nobody else. If you add a named group, copy your * rules into it.

Tokens that match no crawler

6 files had a user-agent value that is a near-miss of one of the nine:

  • lemonde.fr: User-agent: CCBo (it also has a correct CCBot group earlier, so no harm)
  • usatoday.com: User-agent: Claude, which no Anthropic crawler sends as its token
  • linktr.ee: User-agent: Google with a Content-Signal line. Google’s tokens are Googlebot and Google-Extended
  • bestbuy.com: GPTBot-Web; nbcnews.com: ByteSpider Desktop; bloomberg.com: Bytespider-Image

Matching is on the whole product token, case-insensitive. Claude doesn’t match ClaudeBot. These lines do nothing.

Blank lines inside a group

58 files put a blank line between a User-agent line and its rules. chatgpt.com does it for *, hp.com for every AI token it names. RFC 9309 ignores blank lines, so a compliant parser reads these files the way the authors meant. The old 1994 convention ended a record at a blank line, and a parser written to it would see those rules as belonging to no one. I didn’t test which crawlers still parse that way. Deleting the blank lines costs nothing.

A rule above every group

sophos.com commented out the User-agent: * line at the top of its file and left Allow: / under it, above the first real group. A rule before any User-agent line belongs to no group and applies to nobody. Harmless here, since it was an Allow, but the same edit on a Disallow would silently drop it.

Decide your own policy

Two questions, answered separately.

  1. Can my content train models? Then decide on GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider. Blocking Google-Extended doesn’t remove you from Google Search, and blocking Applebot-Extended doesn’t remove you from Apple’s.
  2. Can AI search products cite and link me? Then decide on OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking these takes you out of those answers, or asks to.

User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) are a third thing, and by their own vendors’ account robots.txt mostly doesn’t govern them. If you need them stopped, that’s a server rule, not a robots.txt line.

“No training, yes to answers” written out, with the * rules repeated where a named group would otherwise drop them:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart
Disallow: /search

User-agent: *
Disallow: /cart
Disallow: /search

Several User-agent lines in a row form one group. Name every token you made a decision about, even when the decision is “allow”, so the next person to edit the file can see it was decided.

Then check what your server actually answers each crawler. robots.txt is only the request; a CDN rule can refuse a bot your file allows, or let in one it disallows. The free checker shows, for GPTBot, ClaudeBot and PerplexityBot, whether your robots.txt allows them and what status your server actually returns to each.

The raw data (one JSON line per domain, every robots.txt I fetched, the probe and analysis scripts) is kept with the date above. Ask in the comments if you want the per-domain table.

If you want someone to go through one site end to end, the paid audit on the same page covers crawler access and machine-readability (robots.txt per bot, server answers, structured data, sitemap and llms.txt) and includes paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings. It tells you whether they can read you.