# Common Crawl

Source: https://analytics.5h3ll.site/crawlers/ccbot

CCBot

## ABOUT THIS CRAWLER

Builds the open Common Crawl web archive that many AI datasets use.

**Usually means:** Common Crawl is archiving public pages. Its archive is widely reused for research and model training. Common Crawl warns that other crawlers sometimes pretend to be CCBot.

Its address list could not be retrieved when we checked, so only the user agent is matched.

## USER AGENT

CCBot

We match these as user-agent tokens, not full browser strings. Providers may add surrounding version or product text.

## IP VERIFICATION

This crawler is matched on its user agent only, because no machine-readable list of its IP addresses is available to us. Because user agents can be spoofed, it is shown as “claimed”, not verified.

## Discover when bots and AI assistants crawl your website.

Add a small server-side snippet to see which AI assistants, search engines and training crawlers request your pages, including pages that return 404.

Get started

Read the docs

- ChatGPT, Claude, Perplexity and Gemini traffic in one view
- Verified against each provider’s published IP ranges
- No visitor cookies and no crawler IP addresses stored