Free tool · instant
robots.txt tester What your rules actually say
We read every group in your file, say what each rule does, flag the lines crawlers ignore, and show you the lines yours is missing.
What we check
-
The groups
Every User-agent group, and which rules belong to it.
-
Orphan rules
Rules sitting above the first group, which no crawler applies.
-
Invented directives
Directives that are not real, which crawlers ignore without complaint.
-
An empty Disallow
An empty Disallow:, which allows rather than blocks.
-
Noindex
Noindex:, which has done nothing since 2019.
-
The sitemap line
Whether a Sitemap: line is there at all.
And the lines yours is missing
- When something is wrong
- We hand you the lines to add. If your * group has rules that would catch AI crawlers, we give you the User-agent lines that let the named ones through without touching anything else you have written.
- When nothing is wrong
- We say nothing at all. If your file is already fine, we say nothing. A tool that hands you twelve lines of advice for a file with nothing wrong with it is padding, not help.
Questions about robots.txt
What does a robots.txt tester do?
Reads your file and tells you what it actually says, rather than what you meant. Most robots.txt problems are not typos; they are rules that are valid, silently do nothing, or do the opposite of what they look like.
Why is my robots.txt not working?
The three usual reasons. A rule sits above the first User-agent line, so no crawler claims it. A directive is misspelled, and anything a crawler does not recognise it ignores without complaint. Or an empty Disallow: is read as allowing everything, which is correct and almost never what the author intended.
Does Disallow with nothing after it block or allow?
Allow. Disallow: with an empty value permits the whole site; Disallow: / blocks it. One character between the two, and they are opposites. This check flags the empty form wherever it appears.
Can I use noindex in robots.txt?
No. Google stopped supporting it in 2019 and it never worked reliably elsewhere. A Noindex: line in robots.txt does nothing at all; the meta robots tag on the page is what works. We flag it if you have one.
How do I let AI crawlers through but keep other rules?
Name them. A User-agent group for a specific crawler overrides the * group for that crawler only, so you can keep whatever you have and still let the 12 named AI agents across 6 engines in. This tool gives you those lines, filled in from what your file is missing.
Is this the same as the AI crawler checker?
No, and the difference is worth the two pages. That one answers which AI engines can reach you and says nothing about your file. This one answers whether the file itself is correct, for every crawler and not just the AI ones.
Correct file, still blocked?
The crawler checker reads your rules against every named AI agent and tells you which ones get in.