Anyone can send a request that claims to be GPTBot. So on the AI crawlers page each visit is checked against the IP addresses the bot's vendor publishes (OpenAI, Anthropic, Perplexity, Apple and Mistral) or by a reverse DNS lookup (Google, Apple and Amazon). Other bots are not checked.
| Label | Means |
|---|---|
| Verified | The visit came from an address or DNS name its operator publishes |
| Unverified | It claimed a bot but came from somewhere else |
| Not checked | The bot's operator publishes nothing to check against |
Include unverified traffic
Unverified visits are left out of the figures. Tick Include unverified traffic to count them: they appear as a separate line on the trend chart. Beside the box you see how many visits were unverified and how many verified.
On the Overview tab, each bot in the list carries its label. Hover it to see how many of its visits were verified.
Discard unverified traffic
If you can upload logs, the page has a Settings tab with one option: Discard unverified traffic. With it on, visits claiming a bot from an IP address its operator does not publish are discarded when you upload a log, and the upload message says how many were.
Log set-up
The Log set-up card has three steps:
- Find your provider documentation. We read your website's headers to detect your host, and link to its own docs on logs.
- Upload log file.
- Confirm your data is flowing. It shows Data received, with the latest day, or No data received. Check now looks again.
Once data arrives the card closes, and its heading still shows whether data is flowing.
Why lines were dropped
After an upload, a line shows how many lines were dropped, out of how many, with a count for each reason. Hover a reason for its meaning.
| Reason | What it means |
|---|---|
| Not an AI bot | The user agent doesn't match a known AI crawler. This is the most common reason: regular browser traffic is filtered by design. |
| Duplicate | The same request appeared twice, so it is counted once. |
| No user agent | The record has no user agent, so it can't be classified. |
| Unreadable | The line is not in a log format we read (nginx, Apache or Cloudflare), or its time or address could not be read. |
| Outside the time window | The visit is older than the days we keep from one upload, or too old or too far ahead to be a live visit. |
| Not a project's website | The event hostname doesn't match the website of any project in this workspace. |
When your count may be low
A yellow line warns you when bot visits are likely missing:
- AI bot visits fell to near zero. A cache or CDN in front of your server may be answering bots before they reach it, so its logs miss them. A robots.txt rule or a firewall block can do the same.
- AI bot visits were answered from Cloudflare's cache. Your own server's logs never see these. Cloudflare's logs and the Cloudflare connector count them.
Your CDN's own logs count visits before the cache. See Connect Cloudflare.