TR

AI Crawler Checker: can AI read your site?

Can ChatGPT, Perplexity, Claude and Google reach your page, or are they blocked or served something different? Enter a URL and see your robots.txt rules and the response 8 crawlers get, in seconds.

What does this tool check?

It first opens your page like a normal browser, then requests the same page as Googlebot, Bingbot, Applebot, OAI-SearchBot, ChatGPT-User, PerplexityBot, GPTBot and CCBot. Each crawler’s response is compared with the browser’s: status code, redirects, title, H1, canonical, noindex and page text.

It also reads your robots.txt and shows whether the rules allow these crawlers, plus ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended and Applebot-Extended, which do not publish a testable user agent.

How to read the results

  • Same page as a browser: the crawler gets exactly what visitors see. This is what you want.
  • Blocked / Hit a challenge page: your server, firewall or CDN (often Cloudflare) turns this crawler away. This is the most common reason a site is missing from AI search.
  • robots.txt: Disallowed: your rules keep this crawler out. If that is not deliberate, remove the rule.
  • Different content / Thinner page: the crawler gets a different or cut-down page. If your content loads with JavaScript, most AI crawlers cannot see it.
  • noindex for this bot: the page tells this crawler not to index it.

Search crawlers, AI search crawlers and training crawlers

AI search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot decide whether your page can be cited in ChatGPT, Perplexity or Claude answers. ChatGPT-User and Claude-User arrive when a user opens your page inside a chat. GPTBot, CCBot, ClaudeBot and Google-Extended collect data for model training. You can block training crawlers and still allow search crawlers, and many publishers do exactly that.

How to allow AI search crawlers

Name each crawler in robots.txt. For example, to stay visible in AI search while opting out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

If you use Cloudflare, also check the “Block AI bots” setting under Security → Bots; it can turn crawlers away even when robots.txt allows them.

Limitations

Requests are sent from our server using each crawler’s user agent. CDNs verify real crawlers by IP address, so results can differ from what the real crawler gets, in either direction. Treat “blocked” as a strong signal and “OK” as a good start. The tool checks one page and does not run JavaScript.

The checking logic of this tool is adapted from Puneet Singh’s open-source Open SEO Crawler (MIT licence). The crawler list and comparison rules come from that project; thanks for the work.

FAQ

Frequently asked questions

Short answers about AI crawlers and access.

Why can’t ChatGPT access my website?

The three most common reasons are a Disallow rule for OAI-SearchBot or GPTBot in robots.txt, a CDN setting such as Cloudflare’s that blocks AI bots, and content that only loads with JavaScript. This tool shows the first two directly.

If I block GPTBot, will I disappear from ChatGPT search?

No. GPTBot collects training data. To be cited in ChatGPT search you only need to allow OAI-SearchBot; the two are managed separately.

Why is ClaudeBot checked through robots.txt only?

Anthropic’s crawlers and tokens like Google-Extended either do not publish a testable user agent or are not separate crawlers, so only their robots.txt rules are evaluated.

Is it free, and do you store the URL?

It is free with no sign-up. The URL is used only for the check; if the same URL is checked again within 15 minutes the result comes from cache, and nothing is stored permanently.