Crawler directory
Training and data
robots.txt token only
Apple
Applebot-Extended
About this crawler
A robots.txt control for whether Apple may use crawled content to train its generative models.
Usually means: This token has no separate fetch. Disallowing it opts your content out of Apple model training while Applebot can still index you for search.
Not detectable in request logs.
User agent
Applebot-ExtendedWe match these as user-agent tokens, not full browser strings. Providers may add surrounding version or product text.
IP verification
This is a robots.txt control token. It does not make requests of its own, so it never appears in server logs and cannot be tracked or verified.
Discover when bots and AI assistants crawl your website.
Add a small server-side snippet to see which AI assistants, search engines and training crawlers request your pages, including pages that return 404.
- ChatGPT, Claude, Perplexity and Gemini traffic in one view
- Verified against each provider’s published IP ranges
- No visitor cookies and no crawler IP addresses stored