FREE TOOL · TRAINING VS CITATION
Your robots.txt against AI crawlers
Eleven crawlers checked, and the one distinction that matters: those collecting training corpus, and those fetching your page while answering a user.
Your robots.txt against AI crawlers
You will find it at yourdomain.com/robots.txt. Copy and paste its entire content.
Diagnosis
Training and citation are not the same thing
This is the most expensive confusion around right now. GPTBot collects corpus to train future models. OAI-SearchBot and ChatGPT-User fetch your page live, while the assistant is writing its answer to a user.
Many companies blocked “AI bots” wholesale, often on an agency's advice, then wondered why they never appear in answers. They had blocked both families. Blocking training is a defensible choice; blocking citation means withdrawing yourself from the fastest-growing search channel.
What the tool checks exactly
It applies the rules the way a crawler does: find the User-agent group matching the exact name, fall back to * if none names it, then apply the longest matching path rule — with Allow winning ties.
The verdict also shows where the rule comes from: explicit, inherited from the wildcard, or absent. That is often where the surprise lies — a site believes it allows a crawler while inheriting a Disallow: / set years earlier.
robots.txt is not a security mechanism
It expresses a request that serious actors honour and others ignore. It protects nothing: a page listed under Disallow stays reachable by anyone who knows its address.
A firewall or a CDN setting can also block a crawler with nothing showing in your robots.txt. If the diagnosis here is clean but you are still not cited, that is the next place to look.
Common questions
Should I allow training crawlers?
It is a trade-off, not a rule. Blocking them protects your content from a use you do not control; allowing them raises the odds that models know your brand. Neither answer is wrong — the mistake is not deciding.
What about citation crawlers?
Blocking them almost never makes sense if you sell something. They do not keep your content: they read it while answering, exactly like a classic search engine, and credit you in the answer.
Why is my crawler missing from the list?
The list covers the eleven most common agents. Others exist and appear regularly. A crawler absent from here follows your User-agent: * rule.
Is my file sent anywhere?
No. The analysis runs entirely in your browser, with no network request.
Access is only the first of six levers.
Allowing the crawlers is not enough: your content must exist in the served HTML, your passages must have boundaries and your markup must be correct. That is what AI Visibility covers.