Free tool · under a minute

Can ChatGPT read your website? Check in five seconds

We read your robots.txt and test it against 12 named crawlers from 6 AI engines, then tell you which ones you are turning away.

Free, instant, and nothing is asked of you at the end.

The 12 crawlers we check

Allowing or blocking each one is a real decision with a real consequence. This is the consequence, stated once per crawler.

User agent Engine What it does If you block it
GPTBot ChatGPT Collects pages to train OpenAI models. Keeps your content out of training. Does not remove you from ChatGPT search.
OAI-SearchBot ChatGPT Builds the index behind ChatGPT search. Removes you from ChatGPT search results. Usually the one you did not mean to block.
ChatGPT-User ChatGPT Fetches a page live when a user asks ChatGPT to open a link. A user who pastes your link gets nothing back.
ClaudeBot Claude Collects pages for Anthropic. Keeps your content out of training data.
Claude-SearchBot Claude Reads the web for searches made in Claude. Your pages are left out of what Claude finds when it searches.
Claude-User Claude Fetches a page when somebody asks Claude about it. Claude cannot open your pages when asked to.
Google-Extended Google AI Controls whether your pages train Gemini and appear in AI answers. Does not affect ordinary Google Search ranking. It is a separate switch.
PerplexityBot Perplexity Indexes pages so Perplexity can cite them. Perplexity stops citing you, and it cites heavily.
Perplexity-User Perplexity Fetches a page live for a specific question. Live lookups of your site fail.
Applebot Apple Intelligence (minor) Apple’s long-standing crawler, behind Siri and Spotlight. You lose Siri and Spotlight results as well as the AI features.
Applebot-Extended Apple Intelligence (minor) Controls training use only, not search. Keeps you out of Apple model training while keeping Siri results.
CCBot Common Crawl (minor) Builds the open dataset that many models are trained on. Removes you from a dataset used far beyond any one company.

The four marked minor change little about whether an engine can answer questions about you today. The rest do.

How to allow every AI crawler

Have no rule that names them. A robots.txt that does not mention GPTBot allows GPTBot, because the file works by exception.

How to block training but keep search

This is what most publishers who think about it actually want. Block the training agents, allow the search ones:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

That leaves OAI-SearchBot, PerplexityBot and the live fetchers free to find you, so you can still be cited in answers without your pages feeding the next model.

The mistake we see most

A blanket User-agent: * with Disallow: / left over from a staging site, or a security plugin blocking anything it does not recognise. Neither was a decision anyone made about AI, and both are why sites that want to be found are not.

Questions about AI crawlers

Can ChatGPT read my website?

If your robots.txt allows GPTBot and OAI-SearchBot, and your page does not need JavaScript to show its text, then yes. This check answers it for your site in a few seconds.

What is the robots.txt line for GPTBot?

To allow it, the simplest answer is to have no rule naming it at all. To block it: User-agent: GPTBot on one line and Disallow: / on the next.

Does blocking GPTBot remove me from ChatGPT?

No, and this is the most common misunderstanding. GPTBot is about training. ChatGPT search uses OAI-SearchBot, which is a separate name and a separate decision.

Does Google-Extended affect my Google ranking?

No. It controls whether your pages feed Gemini and Google’s AI answers. Ordinary Search ranking is governed by Googlebot and is untouched by it.

I allow everything. Why is my content still not used?

Usually one of three things: bot protection challenging the crawler before robots.txt matters, text that only exists after JavaScript runs, or a page that never states what the business is. The full check tests all three.

Crawler access is one check of five.

The full check also covers structure, understanding, speed and security, and shows what each is costing you.

Run the free check