Are You Blocking Your Own ChatGPT Citations? What the Top-5,000 robots.txt Census Found

Trend watch — published September 14, 2026. This is a fact-check post: we report exactly what the robots.txt census found, verify the crawler roles against OpenAI’s own documentation, separate the “block AI” reflex from the specifics, and connect it to the crawler-block and eligibility-vs-pick theses we keep testing in the GEO Lab.

What did the census actually find?

An independent researcher fetched robots.txt from the 5,000 highest-ranked sites in the Tranco list on September 7, 2026, and classified a crawler as blocked when the most specific matching rule disallows the site root. The headline number: of the 535 sites that block GPTBot, 238 (44.5%) also block OAI-SearchBot. The remaining 297 block GPTBot only — training out, citations preserved. Zoom out and the pattern repeats: 25.2% of the top 5,000 block at least one AI training crawler, 13.3% block at least one AI search crawler, and 296 sites block an AI search crawler while still allowing Googlebot — a configuration that, as the author put it, “reads like an accidental catch-all rather than a decision anyone made on purpose,” concentrated among smaller sites without legal review.

A separate network-scale scan lands on the same finding from the other direction. Looking at robots.txt across Cloudflare’s network (snapshot August 31, 2026), the intended pattern is clear: publishers block training bots hard — GPTBot at a 2.33:1 block-to-allow ratio, ClaudeBot 2.39:1, Bytespider 5.80:1 — while allowing the answering crawlers, with OAI-SearchBot at 0.94:1 (more allow than block) and ChatGPT-User at 1.12:1. The report’s summary: “Publishers are not blocking ‘AI.’ They are blocking training and allowing answering — and they are doing it to the same companies.” Which is exactly why the 238 sites stand out: they wanted the mainstream pattern and missed it.

GPTBot vs OAI-SearchBot: what’s the difference?

OpenAI runs multiple crawlers with distinct jobs, and its own bot documentation spells them out:

• GPTBot — “used to make our generative AI foundation models more useful and safe.” This is the training crawler. Disallow it to opt your content out of model training. Blocking it costs you no ChatGPT visibility, because it isn’t the crawler that surfaces you.
• OAI-SearchBot — “used to surface websites in search results in ChatGPT’s search features.” This is the citation crawler. If you want ChatGPT to fetch, rank, and link your pages when it answers, this one has to be allowed. Block it and you disappear from ChatGPT’s sourced answers.
• ChatGPT-User — “used for certain user actions in ChatGPT and Custom GPTs.” It fires when a user explicitly asks ChatGPT to visit a page; it isn’t automatic crawling and doesn’t drive baseline search visibility.

So “blocking OpenAI” is not one decision — it’s three, and they point in opposite directions. The publishers who got it right are pulling the training lever (block GPTBot) while leaving the citation lever alone (allow OAI-SearchBot). The 238 who slipped pulled both, opting out of training and answers in the same breath. If your robots.txt has a single User-agent stanza that sweeps up every OpenAI token under one Disallow: /, that’s the failure mode.

Why does this mistake happen so often?

Three reasons, and none of them is “the site owner decided ChatGPT shouldn’t cite them.” First, copy-paste robots.txt: blocklists circulate as gists and “protect your content from AI” templates that lump every AI-adjacent user-agent together, from a year ago when OpenAI’s search crawler barely existed in public discussion. Second, the blanket “block AI” reflex — the same instinct we flagged when Cloudflare moved to block AI crawlers by default. There, the danger was that multi-purpose crawlers carry search and AI in one user-agent, so a blunt toggle can delist you from the very index AI answers cite. Here it’s the cleaner cousin: two clearly-separated OpenAI tokens, one of which you want to keep.

Third, and most fixable: people conflate the consent question with the visibility question. “Do I want my content training OpenAI’s models?” and “Do I want ChatGPT to cite me?” are separate decisions with separate switches — GPTBot for the first, OAI-SearchBot for the second. This is the same shape as the llms.txt debate and the Applebot-Extended clarification we covered last week: Google-Extended is a training opt-out that’s separate from Googlebot, and Applebot-Extended is a training opt-out that doesn’t touch search. Every major vendor now ships a dedicated training token precisely so you don’t have to choose between “no training” and “no visibility.” Use the dedicated token; never the blanket block.

But wait — doesn’t blocking hide me anyway?

Here’s the reversal that keeps this from being a tidy “just allow the bot” story. Blocking is not a reliable invisibility switch, because compliance varies wildly by lab — which means an accidental block can hurt you and fail to protect you at the same time. Two data points make it concrete. A study of 31 million AI citations found the “access paradox”: blocking works only on the labs that honor it. When The New York Times blocked all major AI crawlers, ChatGPT and Gemini citations to it went to zero — but Grok still cited it 37,642 times and Google’s AI Overviews 8,007 times, because those systems either ignore the directive or source from a search index you didn’t block. Grok, AI Overviews, and DeepSeek together account for roughly half of news citations in that sample, with no functioning opt-out.

The Google side is its own trap. Because AI Overviews and AI Mode read from Google’s regular search index (Googlebot), and Google-Extended only governs Gemini training, blocking AI crawlers does little to keep you out of Google’s AI answers — one index audit even found that domains blocking at least one AI crawler showed up in AI Mode at 51.9% versus a 15% baseline. So the net picture for the 238 accidental blockers is the worst of both worlds: they’ve suppressed themselves in the engines that do respect robots.txt (ChatGPT, Gemini, Claude) while staying exposed in the ones that don’t. Blocking as a privacy strategy is leaky; blocking as an accident is pure downside. Either way, the takeaway is the same — decide each crawler on purpose.

If I allow OAI-SearchBot, will ChatGPT cite me?

No — and this is the part the “one-line fix” headlines skip. Allowing OAI-SearchBot makes you eligible to be cited; it does not make you cited. Crawlability is a floor, not a lever. It’s the same distinction we’ve measured all summer: being listed is not being picked (our self-ranking experiment), and being cited is not being recommended (our consensus-pick verdict). Fixing your robots.txt just gets you back into the pool. What decides whether ChatGPT actually pulls you into an answer is the upstream work — answer-first structure, earned reputation, and category authority — and it’s engine-specific, because as our cross-engine study showed, citation sets barely overlap and there is no universal GEO strategy.

So treat the census finding as a reason to audit, not to celebrate. An accidental block is a real, silent leak — closing it is necessary. But once it’s closed, you’re back at the start line, not the finish. Don’t let “we unblocked OAI-SearchBot” become the moment you stop working on the content and reputation that do the actual moving.

What should you actually do this week?

1. Read your own robots.txt line by line. Find every User-agent stanza and check what each one disallows. If a single blanket block sweeps up OpenAI’s tokens together, that’s the accidental catch-all the census found in ~44% of GPTBot-blockers.
2. Split the two decisions. To opt out of training, disallow GPTBot (and, if you like, Google-Extended, Applebot-Extended, ClaudeBot). To stay citable in ChatGPT, allow OAI-SearchBot and ChatGPT-User. They are independent — you can have “no training, yes citations.”
3. Do the same for Google. Keep Googlebot allowed if you want to appear in Search and in AI Overviews/AI Mode, which read from the search index; use Google-Extended alone to opt out of Gemini training. Never block Googlebot to “control AI.”
4. Verify, don’t assume. After editing, re-fetch your robots.txt and confirm OAI-SearchBot and Googlebot resolve to allow at the paths that matter. Then instrument the outcome: track AI-referred traffic in GA4 and watch Google’s Generative AI report in Search Console for the surfaces that expose it.
5. Remember crawlable is the floor. Unblocking gets you eligible, not cited. Put the real effort into answer-first content and earned reputation, then apply the per-engine checklist to each surface — because a fixed robots.txt is table stakes, not a strategy.

Bottom line: The most avoidable GEO mistake on the web right now is self-inflicted and silent — 44.5% of the sites opting out of OpenAI’s training crawler are also blocking the crawler that lets ChatGPT cite them, and hundreds more block AI search crawlers while leaving Googlebot open, which almost always means nobody chose it. GPTBot trains; OAI-SearchBot cites; they are separate on purpose. Split the consent decision from the visibility decision, use the dedicated training tokens, and keep the answering crawlers open. Then remember what unblocking does and doesn’t buy you: it puts you back in the pool, not on the podium. We check the claim before we repeat it — and here the honest version is “audit your robots.txt today, but crawlability was never the lever.”

Frequently asked questions

What is the difference between GPTBot and OAI-SearchBot?

They are two separate OpenAI crawlers with different jobs. GPTBot, per OpenAI’s documentation, is “used to make our generative AI foundation models more useful and safe” — it collects content for model training. OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features” — it’s the crawler that lets ChatGPT fetch and cite your pages. Blocking GPTBot opts you out of training; blocking OAI-SearchBot removes you from ChatGPT’s sourced answers. You control them independently in robots.txt.

How do I opt out of AI training without losing ChatGPT citations?

Disallow the training crawler and allow the search crawler as separate stanzas in robots.txt: User-agent: GPTBot with Disallow: / to opt out of training, and a separate User-agent: OAI-SearchBot with Allow: / (or simply no disallow) to stay citable. Avoid a single blanket block that sweeps up both — that’s the mistake the September 2026 census found in 44.5% of sites blocking GPTBot.

Does blocking AI crawlers make me invisible in AI search?

Not reliably. Blocking works only on labs that honor robots.txt — ChatGPT, Gemini, and Claude respect it, but systems like Grok and Google’s AI Overviews often cite blocked sites anyway (AI Overviews and AI Mode read from Google’s search index, not from AI-specific crawlers). So an accidental block can suppress you in the compliant engines while doing nothing to protect you in the non-compliant ones — the worst of both worlds. Decide each crawler deliberately rather than blocking everything.

If I allow OAI-SearchBot, will ChatGPT cite me?

Allowing OAI-SearchBot makes you eligible to be cited, not guaranteed to be. Crawlability is a floor, not a lever. Being fetchable puts you back in the candidate pool; whether ChatGPT actually pulls you into an answer depends on answer-first content, earned reputation, and category authority — and it’s engine-specific, since citation sets across ChatGPT, Gemini, Perplexity, and Google barely overlap. Fix the block, then do the upstream work.

Should I block Googlebot to stop appearing in AI Overviews?

No. Google’s AI Overviews and AI Mode source from the regular search index that Googlebot builds, so blocking Googlebot to “control AI” would remove you from ordinary Google Search as well — a huge cost for little AI-specific benefit. If you only want to opt out of Gemini training, use the dedicated Google-Extended token, which is separate from Googlebot and does not affect Search ranking.