Should I block the training crawlers?

It is a genuine choice. Blocking GPTBot or ClaudeBot keeps your content out of model training but does not stop the retrieval crawlers that cite you live. Those are separate user-agents. If your goal is AI visibility, keep the retrieval bots allowed; decide on training separately based on how you feel about your content being used to train models.

What is the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI’s training crawler; ChatGPT-User (and OAI-SearchBot) fetch pages live to answer a user and can cite you with a link. For visibility, the retrieval pair matters most. They are what put you in an answer in real time.

Do these bots obey robots.txt?

The major operators (OpenAI, Anthropic, Google, Perplexity, Microsoft) publicly state they respect robots.txt, so your directives are honored by the well-behaved crawlers. The practical risk is not rogue bots. It is accidentally disallowing, or CDN-blocking, the ones you actually want.

The Complete List of AI Crawlers (2026): Who’s Reading Your Site, Footnote Works

The short answer

The AI crawlers that matter fall into three jobs: training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) that ingest content to train models; retrieval crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) that fetch pages live to answer a user and cite with a link; and classic search bots (Googlebot, Bingbot) whose AI features sit on top of search. If you want to be recommended by AI, allow the retrieval crawlers above all. They are the ones that put you in an answer.

Key takeaways

Three jobs: training (ingest for the model), retrieval (fetch live and cite), and classic search (AI features layered on search).
Retrieval bots are the priority: they cite you with a clickable link in real time.
You can allow retrieval while making a separate choice about training; they are different user-agents.
Blocking is often accidental (a CDN default or a copied robots.txt). Confirm what you actually allow.

The three jobs a crawler can have

Every bot below does one of three things. Knowing which is which lets you make a deliberate choice, for example, welcome the retrieval crawlers that drive citations while deciding separately how you feel about training crawlers.

Training, downloads content to help train or update a model. No immediate citation; long-term presence in the model’s knowledge.
Retrieval, fetches your page live to answer a user’s question right now, usually with a clickable citation. This is the one that sends traffic.
Classic search, indexes for a search engine whose AI features (AI Overviews, Copilot) then draw on that index.

The crawlers that matter in 2026

User-agent	Operator	Job
GPTBot	OpenAI	Training
OAI-SearchBot	OpenAI	Retrieval (ChatGPT Search)
ChatGPT-User	OpenAI	Retrieval (live fetch on ask)
ClaudeBot	Anthropic	Training
Claude-SearchBot	Anthropic	Retrieval
Claude-User	Anthropic	Retrieval (live fetch on ask)
PerplexityBot	Perplexity	Retrieval / index
Perplexity-User	Perplexity	Retrieval (live fetch on ask)
Google-Extended	Google	Training opt-in (Gemini)
Googlebot	Google	Search (feeds AI Overviews)
GoogleOther	Google	Assorted Google fetches
Bingbot	Microsoft	Search (feeds Copilot)
Applebot-Extended	Apple	Training opt-in
Amazonbot	Amazon	Retrieval / assistant
CCBot	Common Crawl	Open dataset many models use
Bytespider	ByteDance	Training

The user-agents worth knowing. If you want AI visibility, prioritise allowing the retrieval crawlers.

The robots.txt to welcome them

To be maximally readable by AI assistants, allow the retrieval and training crawlers explicitly rather than relying on a blanket wildcard (which a plugin or CDN can quietly override):

Add an explicit User-agent: / Allow: / block for each of OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot, ClaudeBot and Google-Extended.
Keep Googlebot and Bingbot allowed: their AI surfaces depend on the classic index.
Declare your Sitemap: at the top so every crawler finds your URLs.
Then verify at the network layer: a CDN or WAF can still 403 a bot your robots.txt “allows”. A scan tests both together.

Want to know exactly which of these can read your site right now? Run the free scan, it fetches your site as each crawler would and reports what gets through.

Questions

Should I block the training crawlers?: It is a genuine choice. Blocking GPTBot or ClaudeBot keeps your content out of model training but does not stop the retrieval crawlers that cite you live. Those are separate user-agents. If your goal is AI visibility, keep the retrieval bots allowed; decide on training separately based on how you feel about your content being used to train models.
What is the difference between GPTBot and ChatGPT-User?: GPTBot is OpenAI’s training crawler; ChatGPT-User (and OAI-SearchBot) fetch pages live to answer a user and can cite you with a link. For visibility, the retrieval pair matters most. They are what put you in an answer in real time.
Do these bots obey robots.txt?: The major operators (OpenAI, Anthropic, Google, Perplexity, Microsoft) publicly state they respect robots.txt, so your directives are honored by the well-behaved crawlers. The practical risk is not rogue bots. It is accidentally disallowing, or CDN-blocking, the ones you actually want.

Cite or link this article

Found this useful? Link to it.

This article is free to quote and reference. A link helps more people find accurate, evidence-cited answers, and you're welcome to reuse it with attribution under CC BY 4.0. Grab a ready-made link or citation:

Link (HTML)

<a href="https://footnoteworks.com/blog/the-complete-list-of-ai-crawlers">The Complete List of AI Crawlers (2026): Who’s Reading Your Site, Footnote Works</a>

Link (Markdown)

[The Complete List of AI Crawlers (2026): Who’s Reading Your Site, Footnote Works](https://footnoteworks.com/blog/the-complete-list-of-ai-crawlers)

Citation

John Weeks. “The Complete List of AI Crawlers (2026): Who’s Reading Your Site.” Footnote Works, 2026, https://footnoteworks.com/blog/the-complete-list-of-ai-crawlers.

See where you stand

Can AI actually read your site?

Run the free scan. It fetches your site the way an AI crawler does and reports exactly what the machines can and cannot read. No signup.

Run the free scan or book the free call

John Weeks , AI agency. We build the systems firms run on, automate the work around them, and prove it with first-party data. About the studio.

The Complete List of AI Crawlers (2026): Who’s Reading Your Site