Inspect AI crawler requests and verification

The question to answer

Browser analytics cannot observe most crawler requests because bots often do not run the tracking script. Crawler reporting uses a separate server-side integration. It tells you which recognised agents requested pages and how your server responded; it does not measure search rankings, model training outcomes or recommendations.

Before you start

Access to your website’s server or Cloudflare Worker request handler.

Your website ID and a server-side API credential from settings.

A decision to enable the optional crawler integration and protect its credential.

1. Follow the server-side setup

Open the crawler documentation and use the integration that fits your hosting. This is a backend request handler, not an additional browser script. Keep the API credential in a server secret and out of public repositories, URLs and browser code.

2. Forward the intended request details

The integration sends recognised crawler requests with their user agent, page path, status and requesting IP for verification. The service uses the IP to check published ranges and does not store that IP in crawler reports.

3. Check ingestion and verification

Open Crawlers for the website. Review whether requests appear as verified, unverified or otherwise classified by the available evidence. A crawler-like user agent alone is not proof that the operator sent the request.

4. Investigate paths and errors

Compare crawlers, requested pages and response statuses. A repeated 404 may point to an obsolete link; a 403 may be an intentional access rule. Verify the underlying server behaviour before changing a block rule.

5. Review privacy and allowances

The crawler pool is separate from the normal visitor-event allowance. Review your plan and the integration’s data flow. Server-side collection does not make all crawler activity safe or remove the need for an accurate privacy notice.

Worked example

Verification is an evidence check against available published ranges. Some operators do not publish suitable ranges, and cached range data may need refreshing. Request success does not promise inclusion in an AI response.

Check your setup

The credential exists only in the backend environment.

Requests appear for the intended website with sensible paths and statuses.

Human visitor totals remain separate from crawler counts.

How to interpret the result

A browser-only tracker cannot capture all server-side bots.

Crawler counts are requests, not unique visitors or sales.

robots.txt and access-control decisions remain your website’s responsibility.

Your next action

Start with the most requested failing path and check the real server response before changing access rules.