Generate your custom robots.txt + llms.txt kit free. Control crawlers and guide AI models to your content — simple, fast.
Learn how it works →
By default, all bots are blocked from crawling your website, except those you explicitly allow below. Certain repositories listed below will be restricted even for approved bots.
Crunching the data… Big sites need a bit more time.
