AI Crawler robots.txt Generator
Generate a cautious robots.txt policy for AI search and model-training crawlers. Pick a policy, add private paths, then copy the result. It runs entirely in your browser.
One path per line. This tool does not modify your live robots.txt file.
Your draft
Before publishing: robots.txt manages respectful crawler access; it is not a security control and does not reliably remove a URL from search results. Use noindex, authentication, or removal for content that must not be discoverable.
Bot roles are based on published provider guidance. Recheck official documentation and your CDN/WAF settings before a production change. Related: AI crawler robots.txt guide · More free tools