Crawler directory
Google
Training and data
robots.txt token only
Google-Extended
About this crawler
A robots.txt control token for how Google may use crawled content for Gemini.
Usually means: This token has no user agent of its own. You will not see it in server logs; it only controls how Google uses content it already crawled.
Not detectable in request logs.
User agent
Google-ExtendedWe match these as user-agent tokens, not full browser strings. Providers may add surrounding version or product text.
IP verification
This is a robots.txt control token. It does not make requests of its own, so it never appears in server logs and cannot be tracked or verified.
Discover when bots and AI assistants crawl your website.
Add a small server-side snippet to see which AI assistants, search engines and training crawlers request your pages, including pages that return 404.
- ChatGPT, Claude, Perplexity and Gemini traffic in one view
- Verified against each provider’s published IP ranges
- No visitor cookies and no crawler IP addresses stored