Free tool · Robots.txt generator
Robots.txt generator with AI crawler controls.
Choose what crawlers may read, decide bot by bot which AI crawlers get in, and copy a robots.txt file that says exactly that.
Platform
Save the file as robots.txt at the root of your site, so it opens at yoursite.com/robots.txt.
Search engines and all other crawlers
AI crawlers
OpenAI. Collects public pages that may be used to train OpenAI models.
OpenAI. Builds the index ChatGPT search uses to find and link sites in answers.
OpenAI. Opens a page when a ChatGPT user asks about it or clicks through.
Anthropic. Collects public pages that may be used to train Claude models.
Perplexity. Indexes pages so Perplexity can find and cite them in answers.
Google. Not a separate crawler: a rule that controls whether Google may use your pages for Gemini models. Google Search is not affected.
Apple. A rule that controls whether Apple may use pages Applebot collects to train its AI models.
Common Crawl. Builds the open Common Crawl dataset, which many AI models are trained on.
robots.txt
User-agent: * Allow: /
After you publish it, test it with the AI crawler check.
Next step: the free Uruora scan reads your whole store and ranks what to fix first.
Get the full scanWhat it does
The generator writes a robots.txt from a short form: allow or block all crawlers, folders to block, your sitemap address, and an allow or block choice for 8 AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot.
Each bot is labeled by what it does: training (collecting pages that may train AI models) or search and answers (finding and citing pages when someone asks). One click sets "block training, allow answers".
You can load your current robots.txt first, and the form fills in from it. Platform presets adjust the output: on Shopify you get a robots.txt.liquid template that keeps Shopify's default rules, on WordPress the standard admin rule, on Wix a note on where to paste it.
Why it matters
robots.txt is the first file a crawler reads. One wrong line, like Disallow: / under User-agent: *, can keep your whole store out of search.
AI crawlers add a business choice. Blocking the search and answer bots means ChatGPT search or Perplexity cannot fetch your pages to cite them. Blocking only the training bots keeps your pages out of model training while staying visible in AI answers. Either way, it should be a deliberate decision, not a leftover from a template.
How to use the result
- Copy or download the file and publish it at the root of your site, so it opens at yoursite.com/robots.txt.
- On Shopify, open your theme code editor, right-click the Templates folder, add a new file named robots.txt.liquid and paste the code.
- On Wix, paste the rules in the robots.txt editor in your SEO settings.
- Test the live file with the AI crawler check.
- Do not use robots.txt to hide a page from Google search results. Use a noindex tag on that page instead.
Questions
Where does robots.txt go?
At the root of your domain: yoursite.com/robots.txt. Crawlers do not look for it anywhere else.
Does robots.txt remove a page from Google?
No. A blocked page can still be indexed if other sites link to it, just without its content. To keep a page out of results, let it be crawled and add a noindex tag.
If I block GPTBot, will ChatGPT stop citing my store?
OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search results. Blocking GPTBot alone does not block OAI-SearchBot.
Can I edit robots.txt on Shopify?
Yes, through a robots.txt.liquid template in your theme. Shopify strongly recommends building on its default rules, and the Shopify preset here keeps them in.
Do all crawlers obey robots.txt?
robots.txt is a request that crawlers choose to honor. Major search engines and the companies behind the 8 AI bots listed here say their crawlers follow it.