Updated 29 October 2026

GPTBot, ClaudeBot and the rest: every AI crawler explained

12 user agent names from 6 engines, what each one is for, and what you actually lose by turning it away. This is the same list our scan engine uses.

Check your site for free

The list

User agentEngineWhat it doesIf you block it
GPTBot ChatGPT Collects pages to train OpenAI models. Keeps your content out of training. Does not remove you from ChatGPT search.
OAI-SearchBot ChatGPT Builds the index behind ChatGPT search. Removes you from ChatGPT search results. Usually the one you did not mean to block.
ChatGPT-User ChatGPT Fetches a page live when a user asks ChatGPT to open a link. A user who pastes your link gets nothing back.
ClaudeBot Claude Collects pages for Anthropic. Keeps your content out of training data.
Claude-SearchBot Claude Reads the web for searches made in Claude. Your pages are left out of what Claude finds when it searches.
Claude-User Claude Fetches a page when somebody asks Claude about it. Claude cannot open your pages when asked to.
Google-Extended Google AI Controls whether your pages train Gemini and appear in AI answers. Does not affect ordinary Google Search ranking. It is a separate switch.
PerplexityBot Perplexity Indexes pages so Perplexity can cite them. Perplexity stops citing you, and it cites heavily.
Perplexity-User Perplexity Fetches a page live for a specific question. Live lookups of your site fail.
Applebot Apple Intelligence Apple’s long-standing crawler, behind Siri and Spotlight. You lose Siri and Spotlight results as well as the AI features.
Applebot-Extended Apple Intelligence Controls training use only, not search. Keeps you out of Apple model training while keeping Siri results.
CCBot Common Crawl Builds the open dataset that many models are trained on. Removes you from a dataset used far beyond any one company.

By engine, and what each decision costs

ChatGPT

  • GPTBot. Collects pages to train OpenAI models. Keeps your content out of training. Does not remove you from ChatGPT search.
  • OAI-SearchBot. Builds the index behind ChatGPT search. Removes you from ChatGPT search results. Usually the one you did not mean to block.
  • ChatGPT-User. Fetches a page live when a user asks ChatGPT to open a link. A user who pastes your link gets nothing back.

Claude

  • ClaudeBot. Collects pages for Anthropic. Keeps your content out of training data.
  • Claude-SearchBot. Reads the web for searches made in Claude. Your pages are left out of what Claude finds when it searches.
  • Claude-User. Fetches a page when somebody asks Claude about it. Claude cannot open your pages when asked to.

Google AI

  • Google-Extended. Controls whether your pages train Gemini and appear in AI answers. Does not affect ordinary Google Search ranking. It is a separate switch.

Perplexity

  • PerplexityBot. Indexes pages so Perplexity can cite them. Perplexity stops citing you, and it cites heavily.
  • Perplexity-User. Fetches a page live for a specific question. Live lookups of your site fail.

Apple Intelligence

  • Applebot. Apple’s long-standing crawler, behind Siri and Spotlight. You lose Siri and Spotlight results as well as the AI features.
  • Applebot-Extended. Controls training use only, not search. Keeps you out of Apple model training while keeping Siri results.

Common Crawl

  • CCBot. Builds the open dataset that many models are trained on. Removes you from a dataset used far beyond any one company.

The distinction almost everyone gets wrong

OpenAI
GPTBot trains. OAI-SearchBot searches. Training agents and search agents are not the same thing. GPTBot feeds model training. OAI-SearchBot feeds ChatGPT search. Blocking the first keeps your writing out of the next model while leaving you findable; blocking the second makes you invisible in the product people actually use.
Google
Google-Extended is not Googlebot. Google splits the same way. Google-Extended governs Gemini and AI answers. It has no effect on ordinary Search, which Googlebot handles. They are separate switches and you can set them differently.

Blocking training while staying findable

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Everything not named stays allowed, so OAI-SearchBot, PerplexityBot and the live fetchers can still find you.

Why your rules might not be the problem

Three things block crawlers that robots.txt has nothing to do with:

  • Bot protection challenging anything it does not recognise, before the file is ever read.
  • Rate limiting that treats a crawler like an attack.
  • Content that needs JavaScript, which several of these agents do not run. The page loads and is empty.

The crawler checker tests the first thing. The full check tests all of them.

Questions about AI crawlers

What is GPTBot?

OpenAI’s crawler for collecting web pages to train its models. It obeys robots.txt. It is not the agent behind ChatGPT search, which is OAI-SearchBot, and confusing the two is the most common mistake on this subject.

What is the robots.txt rule for GPTBot?

To block it: User-agent: GPTBot then Disallow: /. To allow it, name no rule for it at all, because the file works by exception.

Does blocking GPTBot hurt my SEO?

No. Googlebot is a separate agent and ordinary search ranking is unaffected. What you lose is your content being used in model training.

Which crawler should I never block?

If you want to be found in AI answers: OAI-SearchBot and PerplexityBot. They are the ones building the indexes that answers are drawn from today.

How often does this list change?

Engines add and rename agents a few times a year. This list is the one our scan engine uses, so it is updated when the engine is. Last reviewed 29 October 2026.

Check your own robots.txt.

Your rules read against all 12 names, with a plain answer for each one.

Check my crawler access