CRAWLER DIRECTORY
AI crawlers, search bots and training agents.
A compact reference of the crawler identities we check: what they are for, how they identify themselves and where their verification sources live.
30 crawlers found
AI answers
Fetch a page because a person asked an AI assistant something. This is usually the traffic that precedes a citation or a click.
OpenAI
ChatGPT-User
Visits a page when a ChatGPT or custom GPT user’s request needs it.
Anthropic
Claude-User
Retrieves a page when a Claude user asks a question that needs it.
Perplexity
Perplexity-User
Visits a page to answer a Perplexity user’s question and link to it.
Google-Agent
Used by agents hosted on Google infrastructure to navigate the web and act on a user’s request.
Google-GeminiNotebook
Fetches individual URLs that Gemini Notebook users added as sources.
Amazon
Amzn-User
Supports user actions, such as Alexa questions that need up-to-date information.
Meta
meta-externalfetcher
Fetches individual links at a user’s request to support Meta AI product features.
DuckDuckGo
DuckAssistBot
DuckDuckGo’s crawler for AI-assisted answers.
Mistral
MistralAI-User
Fetches pages when a Mistral assistant user’s request needs them.
Search indexes
Crawl pages so search and answer engines can discover, refresh and index them.
OpenAI
OAI-SearchBot
Surfaces sites in ChatGPT search results.
Anthropic
Claude-SearchBot
Improves the quality of Claude’s search results by analysing online content.
Perplexity
PerplexityBot
Surfaces and links websites in Perplexity search results.
Googlebot
Google’s main crawler for Search.
Google-InspectionTool
Used by Search testing tools such as URL inspection and the Rich Results Test.
Storebot-Google
Google’s crawler for Shopping and merchant product pages.
Microsoft
bingbot
Microsoft’s crawler for Bing search.
Amazon
Amzn-SearchBot
Improves search experiences in Amazon products and services.
Apple
Applebot
Apple’s crawler for search in Spotlight, Siri and Safari.
DuckDuckGo
DuckDuckBot
DuckDuckGo’s web crawler.
Meta
meta-webindexer
Navigates the web to improve Meta AI search result quality.
Training and data
Collect public content that may be used to train or improve models, or the robots.txt tokens that control that use.
OpenAI
GPTBot
Collects public web content that may be used to train OpenAI’s generative models.
Anthropic
ClaudeBot
Collects web content that could contribute to training Anthropic’s models.
Google-Extended
A robots.txt control token for how Google may use crawled content for Gemini.
Apple
Applebot-Extended
A robots.txt control for whether Apple may use crawled content to train its generative models.
Amazon
Amazonbot
Used to improve Amazon products and services.
Meta
meta-externalagent
Meta’s crawler for AI-related use cases such as training and indexing.
Common Crawl
CCBot
Builds the open Common Crawl web archive that many AI datasets use.
Other bots
Link previews, ad checks and general-purpose crawlers run by AI and platform companies.
Descriptions summarise each provider’s own documentation as of October 2026 and may change. Product names belong to their owners and are used only to describe compatibility. Crawlers can be spoofed, so only a match on both user agent and published IP range is shown as verified.
Discover when bots and AI assistants crawl your website.
Add a small server-side snippet to see which AI assistants, search engines and training crawlers request your pages, including pages that return 404.
- ChatGPT, Claude, Perplexity and Gemini traffic in one view
- Verified against each provider’s published IP ranges
- No visitor cookies and no crawler IP addresses stored