Which AI Crawlers to Allow or Block in robots.txt

The best practice for how to block AI crawlers in robots.txt is to sort them by job before writing a rule. OpenAI, Anthropic and Google each document separate agents or control tokens for model training, for search indexing and for fetches a user triggers. Perplexity documents the last two, and a Disallow means something different for each. Block the search token when you meant the training one, and the site leaves AI search answers while training collection carries on. Every rule below is tied to the operator's own documentation, read on October 7, 2026.

Three jobs, and a token for each

Training crawlers gather content that may end up in a model. Search crawlers build the index an assistant draws on when it answers with links, the first gate for getting cited by ChatGPT and Perplexity. User-triggered fetchers open one URL because a person asked. The table gives each operator's own account of the job and of what a block changes.

OpenAI keeps training and search on separate switches

OpenAI's crawler documentation for website owners makes OAI-SearchBot the token that decides visibility in ChatGPT search. OpenAI describes each setting as independent of the others, and its own example is a webmaster who allows OAI-SearchBot to appear in search results while disallowing GPTBot. ChatGPT-User sits outside that pair: "Because these actions are initiated by a user, robots.txt rules may not apply." A fourth agent, OAI-AdsBot, visits only pages submitted as ads on ChatGPT.

What OpenAI asks of a site that wants to be found

OpenAI's guidance for a website that wants its content retrieved by the search crawler, OAI-SearchBot, comes down to two allowances: the token in robots.txt, and its published IP ranges at the firewall. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update, and when both bots are allowed it may use one crawl for both use cases, so a user agent missing from your logs is weak evidence of a block.

Anthropic documents three bots

Anthropic's official robots.txt guidance covers ClaudeBot, Claude-SearchBot and a third bot, Claude-User. The Anthropic help center article, which carries the date April 7, 2026, says restricting ClaudeBot signals that a site's "future materials" should be excluded from training datasets, so read the rule as forward-looking. Claude-User is where Anthropic departs from OpenAI and Perplexity: the article says disabling it prevents retrieval of your content for a user query, and we found no caveat there that user-initiated fetches may bypass robots.txt. The same article says Anthropic supports the non-standard Crawl-delay extension, asks for the opt-out on every subdomain, and warns that blocking its IP addresses may fail as an opt-out because it impedes the bots' ability to read robots.txt.

Google-Extended has a narrower job than its name suggests

Google's official documentation on robots.txt and AI crawlers describes Google-Extended as a control token with no crawler of its own. Google writes, "Google-Extended doesn't have a separate HTTP request user agent string." It therefore never shows up in a log file. The token covers training of future Gemini models and grounding in Gemini Apps and on Vertex AI, so a training opt-out and an answer surface share one switch. It has no bearing on Search: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

AI Overviews and AI Mode sit under Googlebot. Google Search Central names robots.txt directives for Googlebot as the control for how a site is crawled for Search, and points to nosnippet, data-nosnippet, max-snippet and noindex for limiting what is shown. We found no robots.txt token in the Google pages cited here that removes a page from AI Overviews while Googlebot keeps crawling it. A separate Google page lists the user-triggered fetchers and marks Google-NotebookLM as a former agent supported until August 2026, a reminder that bot lists age.

Perplexity documents no training token to block

Perplexity documents two agents and says of PerplexityBot, "It is not used to crawl content for AI foundation models." We found no training crawler in its documentation, so the one robots.txt decision there is whether to appear in Perplexity's results. Changes may take up to 24 hours to be reflected.

Two rule sets you can copy

Hypothetical example: a B2B software company at example.com wants to stay in AI search answers, opt out of model training, keep its cart private, and keep a 2024 price list that legacy customers still use out of AI answers.

# All other crawlers, Googlebot included
User-agent: *
Disallow: /cart/

# AI search crawlers: allowed, minus private and legacy paths
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/
Disallow: /pricing/2024/

# User-directed fetches that Anthropic lets a site control
User-agent: Claude-User
Disallow: /cart/
Disallow: /pricing/2024/

# Training crawlers: full opt-out
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

Each named group repeats the cart rule. Under RFC 9309, the IETF standard for robots.txt as published by the RFC Editor, a crawler obeys the group that matches its own token and falls back to the * group only when no group matches, so a group for OAI-SearchBot without the cart rule would free it from that rule. Google-Extended is absent on purpose: this company chose to stay eligible for Gemini grounding, and because the same token also governs Gemini training, its training opt-out covers OpenAI and Anthropic only. The standard also allows several User-agent lines above one set of rules, which keeps a full opt-out short:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /

Googlebot stays off that list unless you also mean to stop Google crawling the site for Search. The ChatGPT-User and Perplexity-User lines state a preference their operators say may go unapplied; where a block has to hold, enforce it at the firewall against the published IP ranges.

What a path rule can do about an old pricing page

Blocking GPTBot from old pricing pages in robots.txt takes one line, and it settles less than the request implies. Disallow: /pricing/2024/ under GPTBot tells OpenAI the path should stay out of training. The other routes by which that page can reach a ChatGPT answer, OAI-SearchBot's search results and a ChatGPT-User fetch, are untouched. A buyer handed an outdated price by an assistant brings that number to the sales call, so the path rule belongs in the search-crawler group too. When an allow and a disallow both match, RFC 9309 says the most specific rule wins, defined as the match with the most octets, so Allow: /pricing/ beside Disallow: /pricing/2024/ keeps current pricing crawlable.

robots.txt is still a request. The RFC's introduction says, "These rules are not a form of access authorization." Three of the four operators document user-triggered fetchers that may ignore the file, so a price list that must never be quoted needs a login, or a redirect to current pricing once nobody needs it. Retiring it also ends a contradiction between two of your own URLs, the subject of our entity consistency audit.

The example leaves the old price list crawlable for Googlebot on purpose. Google's control for keeping a page out of Search results is noindex, and Google Search Central warns: "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it."

Where does llms.txt fit?

For llms.txt, robots.txt and AI crawlers, the best practice in 2026 is to give each file one job. The proposal at llmstxt.org, last modified August 10, 2026, draws the line itself: "robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user." We found no allow or block syntax in its format, and no mention of llms.txt as an access control in the four operators' crawler pages. Publish one if you want to hand agents a curated map (our llms.txt generator is a further resource), and keep every access decision in robots.txt.

How to confirm the hits in your logs are real

A user-agent string proves little alone, and Google's fetcher page carries the caution "The user agent string can be spoofed." You need raw server or CDN logs with user agent, source IP and path.

1. Filter a recent window by the token names in the table. Expect no Google-Extended lines.
2. Verify each source IP. OpenAI and Perplexity publish a file of IP addresses per agent, Anthropic a single list, and Google IP range files plus reverse DNS masks for its common crawlers and user-triggered fetchers. Discard hits that fail.
3. Check the status your firewall returns for robots.txt. Under RFC 9309, a response in the 400 range lets a crawler access any resource, and one in the 500 range means it must assume complete disallow.
4. Wait a day before judging a change. OpenAI cites about 24 hours for search, Perplexity up to 24 hours, and the RFC says crawlers should not use a cached copy for longer.

Who approves the change, and what to watch afterward

A training opt-out is a licensing position, so legal or leadership should own it. Search-crawler access is a visibility decision for marketing, the path list belongs to whoever owns pricing pages, and engineering deploys the file and confirms the firewall agrees with it. Record the owner, date and source URL as comments beside each group. OpenAI offers an RSS feed for its crawler page and Anthropic a change-notification form, so subscribe to both.

Afterward, keep three weekly counts: verified hits per token on the paths you care about, verified hits on disallowed paths (expect zero from crawlers that honor the file), and whether assistant answers still quote the legacy page. The last needs prompt checks, with the caveats in our guide to prompt-based visibility tracking, and it feeds your AI share of voice. Our AI Visibility Checker is a further resource, and our AI search optimization service is where this work sits if you would rather hand it off. A robots.txt file with its sources written into it is one the next person can defend and correct when a token changes.

Sources

• OpenAI: Overview of OpenAI Crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (April 7, 2026)
• Google: Google's common crawlers (last updated July 14, 2026)
• Google: Google User-Triggered Fetchers (last updated August 19, 2026)
• Google Search Central: AI Features and Your Website (last updated December 10, 2025)
• Google Search Central: Block Search Indexing with noindex (last updated December 10, 2025)
• Perplexity: Perplexity Crawlers
• RFC Editor: RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
• llmstxt.org: The /llms.txt file, v2 (modified August 10, 2026)