How robots.txt rules are matched
A crawler reads your file top to bottom, finds the group whose User-agent line matches its name most specifically, and ignores every other group. That detail trips up many sites: if you create a group for Googlebot, Googlebot stops reading the * group entirely, so any shared rules must be repeated. Our generator copies your general rules into the AI-search group for exactly this reason.
Inside the chosen group, the longest matching path wins, and when an Allow and a Disallow are equally long, Allow wins. Two wildcards are supported: * matches any sequence of characters and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks PDFs without blocking /pdf-guide/. The tester on this page implements these same rules, so you can verify a path before you ship.
A sensible default for most businesses
- Allow everything by default; disallow only admin areas, carts, checkout, internal search and endless filter parameters.
- Never block CSS or JavaScript files — Google needs them to render and evaluate your pages.
- Keep AI search bots allowed so ChatGPT, Claude and Perplexity can cite you; decide separately about training bots.
- List every sitemap with absolute URLs.
- Don't use robots.txt to hide sensitive pages. The file is public; protect private areas with authentication.
- On staging sites, use “Block all” plus password protection, and remember to switch it back at launch — a leftover
Disallow: /is one of the most common causes of sudden traffic drops.
After publishing, run the AI Visibility Checker to confirm how each major crawler now sees your site, and the Google Index Checker for individual URLs.
Frequently asked questions
Where do I put robots.txt?
At the root of each host: https://yourdomain.com/robots.txt. A subdomain like shop.yourdomain.com needs its own file. The file must be plain text and return a 200 status.
Does robots.txt stop a page from being indexed?
No — it stops crawling. Google can still index a blocked URL (without a snippet) if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag instead.
Can I block AI training but stay visible in AI search?
Yes. Disallow training crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider, and allow search and user-fetch bots like OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. The presets above do exactly that.
Is Crawl-delay supported?
Bing, Yandex and some smaller crawlers respect Crawl-delay. Google ignores it and adjusts crawl rate automatically based on how your server responds.