# AI crawlers, search bots and training agents. Source: https://analytics.5h3ll.site/crawlers A compact reference of the crawler identities we check: what they are for, how they identify themselves and where their verification sources live. 30 crawlers found ## AI answers Fetch a page because a person asked an AI assistant something. This is usually the traffic that precedes a citation or a click. User-triggered · 9 OpenAI ChatGPT-User Visits a page when a ChatGPT or custom GPT user’s request needs it. Anthropic Claude-User Retrieves a page when a Claude user asks a question that needs it. Perplexity Perplexity-User Visits a page to answer a Perplexity user’s question and link to it. Google Google-Agent Used by agents hosted on Google infrastructure to navigate the web and act on a user’s request. Google Google-GeminiNotebook Fetches individual URLs that Gemini Notebook users added as sources. A Amazon Amzn-User Supports user actions, such as Alexa questions that need up-to-date information. Meta meta-externalfetcher Fetches individual links at a user’s request to support Meta AI product features. DuckDuckGo DuckAssistBot DuckDuckGo’s crawler for AI-assisted answers. Mistral MistralAI-User Fetches pages when a Mistral assistant user’s request needs them. ## Search indexes Crawl pages so search and answer engines can discover, refresh and index them. Discovery · 11 OpenAI OAI-SearchBot Surfaces sites in ChatGPT search results. Anthropic Claude-SearchBot Improves the quality of Claude’s search results by analysing online content. Perplexity PerplexityBot Surfaces and links websites in Perplexity search results. Google Googlebot Google’s main crawler for Search. Google Google-InspectionTool Used by Search testing tools such as URL inspection and the Rich Results Test. Google Storebot-Google Google’s crawler for Shopping and merchant product pages. Microsoft bingbot Microsoft’s crawler for Bing search. A Amazon Amzn-SearchBot Improves search experiences in Amazon products and services. A Apple Applebot Apple’s crawler for search in Spotlight, Siri and Safari. DuckDuckGo DuckDuckBot DuckDuckGo’s web crawler. Meta meta-webindexer Navigates the web to improve Meta AI search result quality. ## Training and data Collect public content that may be used to train or improve models, or the robots.txt tokens that control that use. Model data · 7 OpenAI GPTBot Collects public web content that may be used to train OpenAI’s generative models. Anthropic ClaudeBot Collects web content that could contribute to training Anthropic’s models. Google Google-Extended A robots.txt control token for how Google may use crawled content for Gemini. A Apple Applebot-Extended A robots.txt control for whether Apple may use crawled content to train its generative models. A Amazon Amazonbot Used to improve Amazon products and services. Meta meta-externalagent Meta’s crawler for AI-related use cases such as training and indexing. C Common Crawl CCBot Builds the open Common Crawl web archive that many AI datasets use. ## Other bots Link previews, ad checks and general-purpose crawlers run by AI and platform companies. Supporting · 3 OpenAI OAI-AdsBot Checks that pages submitted as ads on ChatGPT meet policy. Meta facebookexternalhit Fetches pages shared on Facebook, Instagram or Messenger to build link previews. Google GoogleOther A generic Google crawler used by product teams, for example for research and one-off crawls. Descriptions summarise each provider’s own documentation as of October 2026 and may change. Product names belong to their owners and are used only to describe compatibility. Crawlers can be spoofed, so only a match on both user agent and published IP range is shown as verified. ## Discover when bots and AI assistants crawl your website. Add a small server-side snippet to see which AI assistants, search engines and training crawlers request your pages, including pages that return 404. Get started Read the docs - ChatGPT, Claude, Perplexity and Gemini traffic in one view - Verified against each provider’s published IP ranges - No visitor cookies and no crawler IP addresses stored