Free tool · under a minute
Can ChatGPT read your website? Check in five seconds
We read your robots.txt and test it against 12 named crawlers from 6 AI engines, then tell you which ones you are turning away.
The 12 crawlers we check
Allowing or blocking each one is a real decision with a real consequence. This is the consequence, stated once per crawler.
| User agent | Engine | What it does | If you block it |
|---|---|---|---|
| GPTBot | ChatGPT | Collects pages to train OpenAI models. | Keeps your content out of training. Does not remove you from ChatGPT search. |
| OAI-SearchBot | ChatGPT | Builds the index behind ChatGPT search. | Removes you from ChatGPT search results. Usually the one you did not mean to block. |
| ChatGPT-User | ChatGPT | Fetches a page live when a user asks ChatGPT to open a link. | A user who pastes your link gets nothing back. |
| ClaudeBot | Claude | Collects pages for Anthropic. | Keeps your content out of training data. |
| Claude-SearchBot | Claude | Reads the web for searches made in Claude. | Your pages are left out of what Claude finds when it searches. |
| Claude-User | Claude | Fetches a page when somebody asks Claude about it. | Claude cannot open your pages when asked to. |
| Google-Extended | Google AI | Controls whether your pages train Gemini and appear in AI answers. | Does not affect ordinary Google Search ranking. It is a separate switch. |
| PerplexityBot | Perplexity | Indexes pages so Perplexity can cite them. | Perplexity stops citing you, and it cites heavily. |
| Perplexity-User | Perplexity | Fetches a page live for a specific question. | Live lookups of your site fail. |
| Applebot | Apple Intelligence (minor) | Apple’s long-standing crawler, behind Siri and Spotlight. | You lose Siri and Spotlight results as well as the AI features. |
| Applebot-Extended | Apple Intelligence (minor) | Controls training use only, not search. | Keeps you out of Apple model training while keeping Siri results. |
| CCBot | Common Crawl (minor) | Builds the open dataset that many models are trained on. | Removes you from a dataset used far beyond any one company. |
The four marked minor change little about whether an engine can answer questions about you today. The rest do.
How to allow every AI crawler
Have no rule that names them. A robots.txt that does not mention GPTBot allows GPTBot, because the file works by exception.
How to block training but keep search
This is what most publishers who think about it actually want. Block the training agents, allow the search ones:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: / That leaves OAI-SearchBot, PerplexityBot and the live fetchers free to find you, so you can still be cited in answers without your pages feeding the next model.
The mistake we see most
A blanket User-agent: * with Disallow: / left over from a staging site, or a security plugin blocking anything it does not recognise. Neither was a decision anyone made about AI, and both are why sites that want to be found are not.
Questions about AI crawlers
Can ChatGPT read my website?
If your robots.txt allows GPTBot and OAI-SearchBot, and your page does not need JavaScript to show its text, then yes. This check answers it for your site in a few seconds.
What is the robots.txt line for GPTBot?
To allow it, the simplest answer is to have no rule naming it at all. To block it: User-agent: GPTBot on one line and Disallow: / on the next.
Does blocking GPTBot remove me from ChatGPT?
No, and this is the most common misunderstanding. GPTBot is about training. ChatGPT search uses OAI-SearchBot, which is a separate name and a separate decision.
Does Google-Extended affect my Google ranking?
No. It controls whether your pages feed Gemini and Google’s AI answers. Ordinary Search ranking is governed by Googlebot and is untouched by it.
I allow everything. Why is my content still not used?
Usually one of three things: bot protection challenging the crawler before robots.txt matters, text that only exists after JavaScript runs, or a page that never states what the business is. The full check tests all three.
Crawler access is one check of five.
The full check also covers structure, understanding, speed and security, and shows what each is costing you.