Robots.txt generator with verified crawler rules

A robots.txt generator should show the rule that actually wins

A robots.txt file is a list of crawl preferences published at the root of a host. The easy part is printing User-agent, Allow and Disallow. The hard part is proving which rule a crawler is supposed to use after exact product-token groups, wildcard fallback, duplicate groups and overlapping paths have all been resolved. This generator keeps that proof beside the generated file.

Under RFC 9309, the Robots Exclusion Protocol, all groups that match the same product token are combined. They do not override each other from top to bottom. An exact group also does not inherit the rules in User-agent: *; the wildcard group is only the fallback when no specific group matches. Inside the selected rules, the longest matching path wins. If an Allow and a Disallow are equally specific, Allow wins. Those details are why appending a generic “block AI” snippet can silently produce the opposite result.

Import first when a file already exists

Paste the current file or open a local text file before selecting new controls. Comments, Sitemap records and directives this generator does not interpret are preserved. When a selected product token shares a group with another agent, the unselected agent keeps its original rules. The selected token is moved to one managed group so duplicated exact groups cannot leave a hidden equal-length Allow behind.

The diagnostic list names rules that appear before any agent, malformed lines, duplicate exact groups, equal Allow/Disallow patterns and the wildcard-inheritance trap. Add the paths that matter on your site—not only /. A broad Disallow: / may still lose to a longer Allow: /public/, and a suffix pattern ending in $ may stop matching as soon as a query string follows the filename.

AI crawler controls need purpose, not one switch

“AI bot” is not one operational category. OpenAI currently documents OAI-SearchBot for search, GPTBot for potential training use, and ChatGPT-User for user-triggered actions. Anthropic likewise separates ClaudeBot, Claude-SearchBot and Claude-User. Allowing automatic search discovery while disallowing a training crawler is therefore a coherent choice. Treating both as the same toggle hides the consequence.

User-triggered agents need an extra boundary. OpenAI says robots rules may not apply to ChatGPT-User actions. Perplexity says Perplexity-User generally ignores robots.txt because the fetch was requested by a user. The generator still lets you express a preference, but it prints that operator-specific limitation next to the control instead of presenting the output as enforcement.

Google-Extended is a product token, not another Googlebot

Google documents Google-Extended as a standalone robots product token for controlling specified Gemini training and grounding uses. It has no separate HTTP user-agent string: Google performs the crawling with existing Google user agents and reads the product token as a control. Google also states that Google-Extended has no effect on inclusion or ranking in Google Search. Blocking Googlebot is a different decision with a different consequence.

Common Crawl needs different wording too. CCBot is the crawler for the Common Crawl open web corpus. It is not honestly described as an operator-labeled model-training bot. A new CCBot rule can request that Common Crawl stop future fetches, but it cannot pull a page out of earlier corpus releases or control copies already held by downstream users.

Robots preferences are public requests, not security

RFC 9309 says the protocol’s rules are not access authorization. Any client can send a familiar User-Agent string, and a non-compliant crawler can ignore the file. Robots.txt is also public, so writing a sensitive path into it announces that path rather than protecting it. Use authentication, authorization and server-side controls for private material. Use verified IP ranges or reverse-DNS procedures when an operator offers them and you need to authenticate traffic at a firewall.

A Disallow request does not delete a model, revoke an existing training record, remove a page from every search index or prove that an operator saw the update. Some operators state that changes can take time to propagate, and cached robots files are part of the protocol. The registry on this route records the date each official source was checked. Re-open those sources before publishing a high-impact policy because tokens, purposes and operator commitments can change independently of this tool.

Publish the generated file in the right place

Serve the result as UTF-8 plain text at exactly /robots.txt on the host it is meant to govern. Rules for https://www.example.com do not automatically govern another subdomain, and an application mounted below the root cannot publish an authoritative file from its own folder. Keep Sitemap URLs absolute, test a representative public and restricted path for every important crawler token, and retain the generation receipt with the policy review that approved the file.