Robots.txt Generator
Create a complete robots.txt file using preset rules for common crawlers. Control access to specific directories and add custom rules with a user-friendly interface.
robots.txt is a plain-text file at the root of your domain that controls which crawlers can access which pages. It is the first file Googlebot fetches — but important: Disallow only blocks crawling, not indexing. Use a noindex meta tag to prevent a page from appearing in search results.
No disallow rules
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Frequently Asked Questions
What is robots.txt used for?
robots.txt is a file that tells search engine crawlers which pages or sections of your site they can or cannot request. It's the first file Googlebot reads when visiting your site.
Does robots.txt block indexing?
No — robots.txt only blocks crawling, not indexing. To prevent a page from appearing in search results, use a noindex meta tag instead.
How do I block AI crawlers like GPTBot?
Use our 'Block AI bots' preset to automatically add Disallow rules for GPTBot, ChatGPT-User, CCBot, and other known AI training crawlers.
What should I never disallow in robots.txt?
Never disallow /_next/, /static/, or CSS/JS directories — doing so prevents Googlebot from rendering your pages correctly, causing them to appear blank or broken in Google's eyes. Only disallow pages you genuinely don't want crawled.