AI crawlers shows every AI bot that visits your site and which pages each one reaches. It reads a server access log that you upload, so it needs no live connection. Choose Upload log file and pick your log. The file is read in your browser, and only AI crawler and agent traffic is kept, with IP addresses and query strings removed before anything is stored.
The four figures
| Figure | What it means |
|---|---|
| Total bot visits | Total requests from AI bots in the selected period. |
| Active bots | Number of distinct bots active in this view. |
| Pages visited | The number of distinct URLs that received at least one bot visit. |
| Failure rate | The proportion of requests that returned an error (a 4xx or 5xx status code). |
The rest of the page
- Crawl activity over time is a line chart of visits by platform.
- Bots lists each bot with when it was Last seen.
- Top pages shows the most visited pages. See all opens the Pages tab with Page, Bot visits, Platforms and Status codes.
- Pages returning errors to bots lists pages that answered a bot with an error.
Bots are named by their user agent. When bot checking is on, visits claiming a bot from an address its operator does not publish appear as a separate line.
What to do next
Fix pages that return errors to bots first, since a bot that gets an error may not return. If a bot you expect never appears, check that your robots.txt allows it with Site check. To remove uploaded data, use the delete option at the foot of the page.