AI crawler access checker

See which AI crawlers your site lets in

AI assistants can only cite pages their crawlers are allowed to read. Enter your domain and we check your robots.txt (the rules file crawlers obey) against the bots behind ChatGPT, Gemini, Perplexity, and more.

Blocking a crawler by mistake? Here is the fix

Access lives in one plain-text file. Changing it takes minutes and no code.

  1. Open your robots.txt

    Find it at yourcompany.com/robots.txt. Look for "Disallow: /" under the blocked bot or under "User-agent: *".

  2. Allow the crawlers you want

    Remove that Disallow line, or add a named group for the bot with "Disallow:" left empty. A named rule always wins over the wildcard.

  3. Publish and re-check

    Upload the updated file to your site root, then run this check again to confirm the crawlers are in.

Reading the result

Blocking an AI crawler is a business decision, not a default.

The bots in this list do different jobs, and treating them as one group is how sites end up invisible in AI answers by accident. Some fetch a page because a user asked a question right now. Others collect text to train a model months from now. The right call is rarely the same for both.

01

Search and answer bots are how you get cited

GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended fetch pages so an assistant can ground an answer in something real, usually with a link back to you. Blocking these does not protect your content, it removes you from the answer and leaves the citation to a competitor who allowed them.

02

Training crawlers are a genuine choice

CCBot, Bytespider and similar agents collect text that ends up in future model training, with no direct traffic back. Publishers with licensing concerns block them on purpose, and that is a defensible position. Just make the decision deliberately rather than inheriting it from a template robots.txt.

03

robots.txt precedence is not first-match

Crawlers follow RFC 9309: a bot obeys the group that names it specifically, and only falls back to the wildcard group if no named group exists. A permissive User-agent: * block does not override a restrictive named one, which is why a site can look open and still block GPTBot.

04

This breaks silently

A blocked AI crawler produces no error, no warning and no ranking drop. Traffic from AI assistants simply never starts, so there is nothing in your analytics to investigate. It is usually introduced by a plugin default, a staging config promoted to production, or a security rule written before these bots existed.

What to change

  • Unblock the answer bots

    If GPTBot, OAI-SearchBot, PerplexityBot or Google-Extended came back blocked and you want to appear in AI answers, remove those disallow rules. This is the highest-leverage fix on the whole list.

  • Decide on training separately

    Keep or lift the block on pure training crawlers based on how you feel about licensing, not on the same rule that governs the answer bots. They are different trades.

  • Recheck after deploys

    robots.txt changes arrive through plugins, platform updates and infrastructure rules that nobody reviews as content. Run this check again after any migration or major release.

Keep going

AI crawler access:
frequently asked questions

  • What does "blocked" mean in this check?

    It means your robots.txt tells that crawler not to fetch your pages, via a "Disallow: /" rule for its user-agent (its identifying name) or for the "*" wildcard. A blocked crawler cannot read your content, so the assistant it feeds cannot quote or cite you.

  • Should I allow every AI crawler?

    Not necessarily. Allowing the search and answer bots (GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended) is how you show up in AI answers. Some teams still block pure training crawlers like CCBot or Bytespider for content-control reasons. That is a business call, not a technical default.

  • Does blocking a bot remove me from ChatGPT completely?

    It stops that crawler from fetching new content, so fresh pages will not be read or cited. It does not erase what a model already learned in past training. For visibility going forward, open access is what matters.

  • Is this checker free, and do you store the domains I check?

    Free, no account. We fetch only the public robots.txt and llms.txt of the domain you enter and return the result. We do not store the domains you check.

Not sure which crawlers you actually want?

We audit your AI access end to end (robots.txt, rendering, and structured data), then implement the fixes and prove the lift with data.