The AI Crawler Allowlist: Twelve Bots and Which Ones You Can't Afford to Block

Every AI company now runs separate bots for training, search indexing, and live user fetches. Block the wrong one and you vanish from AI answers with no error message. The full roster, a copy-paste robots.txt, a five-minute check, and the Cloudflare default that changed on September 15, 2026.

Most robots.txt files were written before AI companies split their crawlers. One rule blocked "the AI bot." That rule now hits three different systems with three different consequences. The person who wrote it has usually left the company.

Three kinds of bot

Training crawlers collect pages in bulk for the next model. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended. Blocked, they leave your future content out of the weights; today's answers stay the same.

Search crawlers build the index a live answer is retrieved from. OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Bingbot for the Bing index ChatGPT leans on. Blocked, they drop you from the candidate pool. There's no error, no dashboard notice. You stop being named.

User-fetch agents pull one page when a person asks about it in a chat. ChatGPT-User, Claude-User, Perplexity-User. Blocked, they can't read your pricing page when a buyer pastes the link. Anthropic says all three of its bots honor robots.txt, per its February 2026 documentation (coverage). OpenAI and Perplexity say rules may not bind their user-fetch agents, since a human asked for the page. (Reasonable, and a reminder that robots.txt is a request rather than a lock.)

The roster

• GPTBot (OpenAI). Training data for future GPT models. No effect on ChatGPT citations.
• OAI-SearchBot (OpenAI). Builds the index behind ChatGPT search. Blocking it takes you out of ChatGPT's live answers.
• ChatGPT-User (OpenAI). Fetches a page when a ChatGPT user asks about it or pastes the URL. OpenAI says robots.txt may not apply.
• ClaudeBot (Anthropic). Training. Anthropic publishes no IP ranges, so verify by user agent.
• Claude-SearchBot (Anthropic). Indexes for Claude's web search. Anthropic's docs say blocking it reduces visibility in Claude's answers.
• Claude-User (Anthropic). Live fetch when a Claude user asks. Honors robots.txt.
• PerplexityBot (Perplexity). Indexes for Perplexity answers.
• Perplexity-User (Perplexity). Live fetch. Perplexity says robots.txt generally doesn't apply.
• Google-Extended (Google). Opt-out token for Gemini training and grounding. It does nothing to Google Search or AI Overviews; Googlebot handles those.
• Applebot-Extended (Apple). Opt-out for Apple Intelligence training.
• Meta-ExternalAgent (Meta). Meta AI and Llama.
• Bingbot (Microsoft). Bing's index, which grounds ChatGPT search and Copilot. If you blocked Bingbot years ago to save server load, you blocked ChatGPT too.

The copy-paste block

User-agent: GPTBot

Allow: /

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: ClaudeBot

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: Claude-User

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: Perplexity-User

Allow: /

User-agent: Google-Extended

Allow: /

User-agent: Applebot-Extended

Allow: /

User-agent: Meta-ExternalAgent

Allow: /

User-agent: Bingbot

Allow: /

Sitemap: https://yoursite.com/sitemap.xml



Write each name with that exact casing; some implementations are case-sensitive. Under RFC 9309, a specific User-agent group overrides your wildcard rules, so these lines win even where User-agent: * blocks a folder.

The two mistakes we see most

The edge disagrees with the file. robots.txt says allow. A WAF rule or rate limiter written by the security team returns 403 or 429 to anything with "bot" in the user agent. The crawler reads that as a block. We see it in roughly [yours: share of Beacon audits] of the sites Beacon audits. The fix is a monthly look at server logs for verified hits per agent. OpenAI publishes IP lists for each of its crawlers (gptbot.json, searchbot.json, chatgpt-user.json), so a request with the right user agent from the wrong IP is an impostor you can block without touching your real visibility.

Cloudflare's defaults. Cloudflare fronts about 20% of web domains. On July 1, 2025 it started blocking all known AI crawlers by default on new domains. On September 15, 2026 it replaced the single switch with three categories (Search, Agent, Training) and set new defaults: Training and Agent blocked on pages that show ads, for new domains and for Free-tier accounts that never changed the setting; Search allowed (Cloudflare's post). The legacy "Block AI bots" toggle is deprecated. If you run ads and haven't touched it, ChatGPT-User and Claude-User can't reach you. If you onboarded a domain between July 2025 and last week and never opened the panel, nothing could.

Allow the training bots?

For most B2B and product brands, yes. Cloudflare's own 2025 figure had ClaudeBot making about 71,000 requests for every referral click it sent back (source). As a traffic deal that's terrible. As a brand decision it's backwards. The training crawl is how the next model learns your company exists, what it does, what it costs, and who it's for. A publisher selling pageviews has a real case to block. A company selling a product wants to be in the weights when the next model ships. [yours: has a model ever described Space wrong, and what did that cost]

Check it in five minutes

Run curl -A "OAI-SearchBot" -I https://yoursite.com. You want 200. Repeat with ClaudeBot and PerplexityBot. Then pull a month of logs and count verified hits per agent. Zero hits from OAI-SearchBot on a site that ranks in Google means the edge is blocking something the file allows.

Beacon checks all twelve as part of its 86-point audit, free, back within 24 hours. The curl line above takes less time than reading this post did.