Set your crawl rules, tick the AI crawlers you want to block, add your sitemap, and copy or download a correct robots.txt.
How it works
The generator builds the file live from your choices. You get a default rule group for all bots (allow everything, block everything, or your own Disallow and Allow paths), one block per AI crawler you tick, and the Sitemap line. Copy it, or download robots.txt ready to upload to your site root.
Two limits to keep in mind. Robots.txt is a request that well-behaved crawlers follow, not access control. And blocking a page from crawling does not remove it from the index. Use a noindex meta tag for that. Everything runs in your browser.
Frequently asked questions
What does robots.txt do?
It tells crawlers which parts of your site they may fetch. Well-behaved bots, including Google, Bing, and the AI crawlers listed here, read it before crawling and follow the rules. It lives at exactly yoursite.com/robots.txt.
Does robots.txt hide pages from Google?
No, and this matters. Robots.txt blocks crawling, not indexing. A blocked page can still appear in results, without a description, if other sites link to it. To keep a page out of the index, allow crawling and add a noindex robots meta tag instead. Never rely on robots.txt for private content. It is a public request, not access control.
How do I block AI crawlers like GPTBot and ClaudeBot?
Tick the bots you want to block, or use the master toggle. The generator writes a User-agent block with Disallow: / for each one: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, and meta-externalagent. These vendors say they follow robots.txt. Blocking Google-Extended opts out of Gemini training without affecting Google Search.
What is the difference between Disallow and Allow?
Disallow blocks paths that start with the value. Allow carves exceptions back out. For example, disallow /admin/ but allow /admin/public/. An empty Disallow: means everything is allowed, which is the correct explicit default.
Why no crawl-delay option?
Because Google ignores it, it was never standardized, and misuse hurts crawling more than it helps. Set the crawl rate in each search engine's webmaster tools instead.
Should I add my sitemap to robots.txt?
Yes. The Sitemap: line tells every crawler where your XML sitemap lives, so they find new pages faster. Enter the full URL, such as https://yoursite.com/sitemap.xml, and the generator adds the line. You can list more than one sitemap.
Can a robots.txt mistake take my site out of Google?
Yes, and it is a common accident. A single Disallow: / under User-agent: * blocks every crawler from every page, which is right for a staging site and disastrous for a live one. Read the preview before you upload, and run the file through our Robots.txt Tester with a few real URLs.
Where do I put the file?
Download it and upload it to your site root, so it is served at https://yoursite.com/robots.txt. A subdirectory does not work, and each subdomain needs its own file. Then test it with our Robots.txt Tester.