Why this happens by accident

Almost no business decides to hide from ChatGPT. Yet a large share of sites do exactly that, because blocking happens in places owners never look. A theme or SEO plugin ships a restrictive robots.txt. A developer copies a blocklist from a forum thread. Or, most common in 2025 to 2026: the CDN does it automatically.

Jul 1, 2025
Cloudflare began blocking AI crawlers by default for new domains, making “allow AI bots” an opt-in choice most owners never see. Cloudflare

That policy exists for good reasons, publishers wanted leverage over AI training. But for a service business that *wants* to be recommended by AI assistants, the default quietly works against you. Blocking the crawler that feeds ChatGPT is the AI-era equivalent of a noindex tag on Google.

The AI crawlers that matter

Not all bots do the same job. Two categories matter: training crawlers (ingest content to train models) and retrieval crawlers (fetch pages live to answer a user right now). Retrieval bots are the ones that put you in an answer with a clickable citation, so those are the ones you least want to block.

User-agentOperatorJob
GPTBotOpenAITraining crawler
OAI-SearchBotOpenAIRetrieval (ChatGPT Search)
ChatGPT-UserOpenAILive fetch when a user asks
ClaudeBotAnthropicTraining + retrieval
PerplexityBotPerplexityRetrieval
Google-ExtendedGoogleGemini / AI training opt-in
CCBotCommon CrawlOpen dataset many models use
The user-agents to allow if you want to be read by AI assistants. Retrieval bots are the highest priority.

Check it in two minutes

  1. Read your robots.txt

    Visit https://yourdomain.com/robots.txt. Search the text for GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. A line like User-agent: GPTBot followed by Disallow: / means you are blocking that bot. A blanket User-agent: * with Disallow: / blocks everything.

  2. Test the network layer

    A clean robots.txt is not enough, your CDN or firewall can return a 403 before the bot ever sees the file. The reliable test is to fetch a page while pretending to be the bot and read the status code, which is exactly what an AI-visibility scan does for you.

  3. Confirm you are actually being read

    The ground truth lives in your server access logs. Grep for GPTBot, ClaudeBot and ChatGPT-User and you will see whether the assistants are fetching your pages, and how often. On ebikeoracle.com, a site we built and still host, ChatGPT-User is the single most active crawler, ahead of Googlebot.

The fix

At the robots.txt layer, explicitly welcome the AI user-agents instead of relying on a blanket wildcard:

  • Add an explicit User-agent: GPTBot block with Allow: /, and the same for OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended.
  • In Cloudflare (or your WAF), turn off any “block AI scrapers / bot fight mode” rule that catches these user-agents, or add an explicit allow rule for them.
  • Make sure the pages themselves render real HTML on the first request. Many AI crawlers do not run JavaScript, so a page that only assembles content client-side reads as blank even when it is “allowed”.