# AI crawler tracking

Source: https://analytics.5h3ll.site/docs/crawlers

See which AI and search crawlers read your site, verified against the operators own IP ranges.

Normal analytics cannot see crawlers, because they do not run JavaScript. To see them, a small Cloudflare Worker in front of your site forwards each crawler visit to your dashboard. Visitors are untouched: the Worker only reports requests whose user agent looks like a known crawler.

| What you get | Detail |
| --- | --- |
| Per crawler visits | 28 tracked crawlers from OpenAI, Anthropic, Google, Perplexity, Meta, Apple and more. |
| Verification | Each visit is checked against the IP ranges the operator publishes, so spoofed user agents are marked. |
| Pages and status | Which paths are read and what status they returned, including 404s. |

## Set up

1. Open **Crawlers** in your website and copy your API key from **Settings**.
2. Create a Worker in Cloudflare and paste the snippet below.
3. Add a secret named `ANALYTICS_KEY` with your API key.
4. Route the Worker to your domain, for example `example.com/*`.

```
// Cloudflare Worker: forwards AI and search crawler visits to Analytics by 5H3LL.
// 1. Add a Worker secret named ANALYTICS_KEY containing your website's API key.
// 2. Route this Worker to your domain, for example example.com/*.
const CRAWLER_TOKENS = [
  'ChatGPT-User', 'Claude-User', 'Perplexity-User', 'Google-Agent',
  'Google-GeminiNotebook', 'Amzn-User', 'meta-externalfetcher', 'DuckAssistBot',
  'MistralAI-User', 'OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot',
  'Googlebot', 'Google-InspectionTool', 'Storebot-Google', 'bingbot',
  'Amzn-SearchBot', 'Applebot', 'DuckDuckBot', 'meta-webindexer',
  'GPTBot', 'ClaudeBot', 'Amazonbot', 'meta-externalagent',
  'CCBot', 'OAI-AdsBot', 'facebookexternalhit', 'GoogleOther',
];
const CRAWLERS = new RegExp(CRAWLER_TOKENS.join('|'), 'i');

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const userAgent = request.headers.get('user-agent') || '';
    if (CRAWLERS.test(userAgent)) {
      ctx.waitUntil(
        fetch('https://analytics.5h3ll.site/api/v1/crawls', {
          method: 'POST',
          headers: { authorization: 'Bearer ' + env.ANALYTICS_KEY, 'content-type': 'application/json' },
          body: JSON.stringify({ hits: [{ user_agent: userAgent, ip: request.headers.get('cf-connecting-ip'), url: request.url, status: response.status }] }),
        }).catch(() => {}),
      );
    }
    return response;
  },
};
```

```
npx wrangler secret put ANALYTICS_KEY
```

The snippet is generated from the same list as the [crawler directory](https://analytics.5h3ll.site/crawlers), so it always matches what we can classify.

## Verification states

| State | Meaning |
| --- | --- |
| `verified` | The user agent matched and the request came from an IP range the operator publishes. |
| `claimed` | The user agent matched, but the IP is outside the published ranges. Treat as possibly spoofed. |
| `unverifiable` | The operator publishes no IP ranges, so only the user agent can be checked. |

## Allowance

Crawler visits have their own monthly allowance on every plan, separate from page views. They never use your page view quota.

| Plan | Crawler visits per month |
| --- | --- |
| Free | 2,000 |
| Starter | 10,000 |
| Indie | 50,000 |
| Growth | 250,000 |
| Founder Lifetime | 250,000 |

## Privacy

Crawler IP addresses are used to verify a visit and are not stored. Only daily counts per crawler, path and status are kept, for 400 days.

Previous

Dodo Payments

Next

API reference

Something missing or wrong? [Open a support ticket](https://analytics.5h3ll.site/support).