Free tool · AI crawler check
Which AI assistants may read your site?
Enter your website. We read your robots.txt the way each AI crawler reads it and show, bot by bot, whether it is allowed in and what it is for.
What it checks
The tool fetches yoursite.com/robots.txt and applies its rules to 8 AI user agents: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, Applebot-Extended and CCBot (Common Crawl).
For each one it finds the group of rules that applies: a group written for that bot by name, or the general * group. A bot counts as blocked when that group disallows the whole site (Disallow: /) without allowing it back.
If there is no robots.txt at all, every crawler is allowed by default.
Why it matters
An AI assistant can only recommend what it has read. If OAI-SearchBot or PerplexityBot is blocked, ChatGPT search and Perplexity cannot fetch your pages to cite them.
The bots are not all the same. Some collect pages for model training (GPTBot, ClaudeBot, CCBot), some fetch pages to answer a user right now (ChatGPT-User), and Google-Extended and Applebot-Extended are rules about how collected pages may be used. You may want to allow search and answer bots while blocking training. That is a business choice, and it should be a deliberate one.
Many stores block AI bots by accident: a security plugin, a template robots.txt, or an old rule nobody remembers.
How to fix it
- Open your robots.txt (on Shopify, through a robots.txt.liquid template in your theme code; on most other platforms, in the SEO settings).
- Find any group that names the bot, or the * group, with Disallow: /.
- To allow a bot, remove that line from its group or add a group for it with Allow: /.
- To block training but keep answers, disallow GPTBot, ClaudeBot and CCBot by name and leave OAI-SearchBot, ChatGPT-User and PerplexityBot allowed.
- Run the check again to confirm.
Questions
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended controls use of your pages for Gemini models. Google Search crawling is done by Googlebot, which this rule does not affect.
Do all AI companies obey robots.txt?
robots.txt is a request that crawlers choose to honor. The companies behind the 8 bots checked here say their crawlers follow it.