AI Crawlers List 2026: Every Bot Behind ChatGPT, Google AI, Copilot, Claude and More | Hailder House
AI crawler guide

Every AI crawler that decides if AI can see you.

By Chris Moran, founder of Hailder House. Updated November 4, 2026.

When someone asks ChatGPT, Google, Copilot or Siri who to hire, the answer comes from what a crawler read on your site, or didn't. Here's every one that matters in the US, what it feeds, and the exact name to use in robots.txt. I checked each one against the company's own documentation, not somebody's blog.

Scan my site against all of them Jump to the full list
The AI engines we optimize for

Every AI your buyers ask, and what each one reads.

They don't all work the same way. Some send their own crawler. Some answer from another company's index. Tap one to see where its answers come from.

ChatGPT

OpenAI

Where its answers come from

ChatGPT search uses OpenAI's own search crawler plus third-party search providers. Its live fetcher grabs a page when someone asks about it.

What reads your site
OAI-SearchBotanswers & searchChatGPT-Userlive fetchGPTBotAI training
The trap most sites miss

Blocking GPTBot (training) is a separate choice from OAI-SearchBot. Plenty of sites block both by accident and drop out of ChatGPT search.

What I check

That OAI-SearchBot can reach you, that your pages answer buyer questions plainly, and that your facts are easy to quote.

Checked against each company's own published crawler documentation, November 2026. Hailder House isn't affiliated with any of these companies. Names are their owners' trademarks.

Read this first

There are three kinds of AI crawler, and they don't mean the same thing.

Most people block or allow "AI bots" as one thing. That's how good businesses vanish from AI answers without knowing it.

Search crawlers

These build the index an AI pulls answers from. OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, meta-webindexer for Meta AI, Applebot, Amzn-SearchBot, Googlebot and bingbot. Block one of these and you're out of that tool's answers. This is the group you almost always want to allow.

Live fetchers

These grab a page in the moment, because a person asked the AI about it. ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User, DuckAssistBot. Some of them say plainly that they may not follow robots.txt, since a human asked for the page.

Training crawlers and control tokens

These collect pages to train future models. GPTBot, ClaudeBot, meta-externalagent, Amazonbot, MistralAI-Training and CCBot. Then there are control tokens like Google-Extended and Applebot-Extended. They don't crawl anything themselves. They just tell the company whether it can use what its regular crawler already collected. Whether you allow training is your call. Blocking it doesn't take you out of search answers.

The full list

Every crawler, what it feeds, and the robots.txt name.

Grouped by company. "Search" means it feeds AI answers. "Live" means it fetches when a user asks. "Training" means model training. "Control" means a token with no crawler of its own.

Companyrobots.txt nameWhat it feedsKindFollows robots.txt?
OpenAIOAI-SearchBotChatGPT searchSearchYes. Takes about 24 hours to apply
OpenAIChatGPT-UserChatGPT and custom GPTs, when a user asksLiveOpenAI says robots.txt rules may not apply
OpenAIGPTBotTraining OpenAI's modelsTrainingYes
GoogleGooglebotGoogle Search, including AI Overviews and AI ModeSearchYes. Google calls it the control for AI features in Search
GoogleGoogle-ExtendedGemini training, Gemini Apps and groundingControlYes. Doesn't affect Google Search
MicrosoftbingbotBing, Copilot, and much of DuckDuckGo and EcosiaSearchYes
AnthropicClaude-SearchBotClaude's searchSearchYes
AnthropicClaude-UserClaude, when a user asksLiveAnthropic says site owners can control it
AnthropicClaudeBotTraining ClaudeTrainingYes
PerplexityPerplexityBotPerplexity's search results. Perplexity says it isn't used to train AI modelsSearchYes
PerplexityPerplexity-UserPerplexity, when a user asksLivePerplexity says it generally ignores robots.txt
Metameta-webindexerMeta AI search results and citationsSearchYes
Metameta-externalfetcherMeta AI, when a user shares or asks for a linkLiveMeta says it may bypass robots.txt
Metameta-externalagentTraining Meta's AI models, or indexingTrainingYes
AppleApplebotSiri, Spotlight and Safari searchSearchYes. Follows your Googlebot rules if you don't name Applebot
AppleApplebot-ExtendedWhether Apple can use Applebot's data to train its AIControlYes. Blocking it doesn't remove you from Apple search
AmazonAmzn-SearchBotSearch in Amazon products, including Alexa. Amazon says not for trainingSearchYes. Follows your other search bot rules if not named
AmazonAmzn-UserAlexa, when a user asksLiveAmazon says it may not follow every rule
AmazonAmazonbotAmazon products and services, may be used for trainingTrainingYes
DuckDuckGoDuckAssistBotDuckAssist AI answers. DuckDuckGo says it isn't used to train AILiveYes. Takes about 72 hours
MistralMistralAI-UserMistral's assistant, when a user asksLiveMistral says the name controls which sites it can reach
MistralMistralAI-TrainingTraining Mistral's modelsTrainingYes
You.comYouBotYou.com's search indexSearchYes, including crawl delay
Common CrawlCCBotA free public web archive that many AI companies have trained onTrainingCommon Crawl shows how to block it
BravenoneBrave SearchSearchNo name of its own. Won't crawl what Googlebot can't

Sources: each company's own crawler documentation, read November 4, 2026. A few AI companies, including xAI (Grok), don't publish an official crawler page yet, so I've left them off rather than guess.

Who borrows whose index

Fix two crawlers and you've covered most of the list.

A lot of AI answers don't come from the AI company's own crawler. They come from Google's or Bing's index.

What runs on Google

AI Overviews and AI Mode are part of Google Search, so Googlebot decides them. Brave's crawler won't go anywhere Googlebot can't. And if your robots.txt doesn't name Applebot, Apple follows your Googlebot rules too, which takes in Siri and Spotlight.

What runs on Bing

Copilot answers from Bing's index. DuckDuckGo says its regular links come largely from Bing. Ecosia shows results from Bing, Google and its own European index. If Bing can't see you, a big chunk of the AI world can't either.

What runs on its own crawler

ChatGPT search, Perplexity, Claude, Meta AI and Alexa each run their own search crawler. ChatGPT search also uses third-party search providers, which OpenAI doesn't name.

A starting point

A robots.txt that lets AI answers find you.

This allows every search crawler and live fetcher on the list. It leaves the training question to you, because that's a business decision, not a technical one.

# Let AI answers find you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: Googlebot
User-agent: bingbot
Allow: /

# Optional: opt out of AI training only.
# Remove this block if you're fine with training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

Sitemap: https://yourcompany.com/sitemap.xml

Change the sitemap line to your own. If you already block anything, check it against the list above before you paste this in. And robots.txt is only half of it: a firewall or a Cloudflare bot setting can block these crawlers even when robots.txt allows them. The free AI check tests for both.

What's changing

Three changes coming in 2027.

Bing is replacing its AI controls

Bing announced on October 30, 2026 that its old NOARCHIVE and NOCACHE settings are being retired for AI use. New page-level tags control whether Bing can use a page for AI training and for grounding AI answers like Copilot's. Bing says it starts honoring them February 1, 2027. If you used the old tags to keep content out of Copilot, you'll need to switch by March 1, 2027. None of it changes whether you show up in regular Bing search.

A shared "no AI training" signal

There's a new standard way to say "don't train on this" that works across companies, instead of naming every training crawler one by one. Bing and Anthropic have announced support. Google says it plans to follow. Until it's everywhere, naming the training crawlers yourself is still the safe move.

More live fetchers

Every major AI now has a crawler that fetches a page when a user asks about it. Some of them say they may not follow robots.txt. So the content on your pages matters more every month, and the rules file matters a little less.

Questions

AI crawler questions people ask me.

If I block GPTBot, am I out of ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which is a separate name. OpenAI says the two settings are independent, so you can block training and still show up in ChatGPT search. Plenty of sites block both by accident.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. It also doesn't control AI Overviews. Googlebot and your snippet settings do.

How do I keep a page out of Google's AI Overviews?

Google says AI Overviews follow the same controls as regular snippets: nosnippet, data-nosnippet, max-snippet and noindex. Most businesses want the opposite, though. The goal is to be the source the overview cites. My AI Overviews guide covers how.

Why does a crawler show up in my logs when robots.txt blocks it?

A few reasons. It might be a live fetcher that says it may not follow robots.txt, like Perplexity-User or meta-externalfetcher. It might take a while for the company to pick up your change. OpenAI says about 24 hours, DuckDuckGo about 72. Or it might be a fake. Anyone can put "GPTBot" in a user agent, which is why OpenAI, Anthropic and Common Crawl publish the IP addresses their real crawlers use.

Is allowing every crawler enough to get recommended?

It's the floor, not the goal. Access means the AI can read you. Whether it recommends you comes down to whether your pages answer real questions with specific facts, whether you sound like a real expert, and what the rest of the internet says about you. That's the part I actually rebuild.

Your free AI check

Find out if your company is in the answer today.

Start with the free scan. It scores your site out of 100 in about 15 seconds. Then send me three questions your buyers would ask, and I'll run them through ChatGPT, Claude, Perplexity, Gemini and Google and email you the screenshots: who gets named, and whether you do.

Get my free AI checkOr book a 30-minute call

Your website and questions are used only to run your check, and I don't sell them. Please don't include client names, case details or anything confidential.