Home / SEO & AEO / Robots.txt Generator
Robots.txt Generator
Build a robots.txt with allow and disallow rules, block AI crawlers in one click and add your sitemap.
Where it goes: upload the file to the root of your domain so it’s reachable at https://yoursite.com/robots.txt. It controls crawling, not indexing — to keep a page out of Google’s results use a noindex meta tag instead, and never rely on robots.txt to hide private content, since the file itself is public.
About the Robots.txt Generator
The Robots.txt Generator builds crawl rules for your website. Set a default for all crawlers, block or allow AI crawlers such as GPTBot and ClaudeBot in one click, keep bots out of admin and search pages with custom rules, and add your sitemap URLs. Copy or download a ready-to-upload robots.txt.
How to use the Robots.txt Generator
- Choose the default for all crawlers.
- Choose how to treat AI crawlers — block training bots, block all or allow all.
- Add custom rules for folders such as /admin/ and your sitemap URLs.
- Copy or download robots.txt and upload it to your site’s root.
Frequently asked questions
Where does robots.txt go?
It must be at the root of your domain, such as https://example.com/robots.txt. Crawlers don’t look for it anywhere else.
Does robots.txt stop a page appearing in Google?
Not reliably. It stops crawling, but a blocked page can still be indexed if other sites link to it. Use a noindex tag to keep a page out of search results.
Should I block AI crawlers?
It depends on whether you want AI tools to use your content. Blocking training bots doesn’t affect your Google rankings.