The AI crawlers page has five tabs. They read the same bot visits. The page itself is explained in AI crawlers.
| Tab | What it shows |
|---|---|
| Overview | The figures, how bot visits change over time by engine, platform or bot, the top pages and the pages returning errors |
| Pages | Which pages AI bots reach, and which fail or are never crawled |
| Bots | Every known AI crawler, who runs it, and how to allow or block it |
| Logs | Each bot request, newest first |
| Settings | The log set-up, uploads, the unverified traffic options and the delete |
Overview
One line leads: the bot with the most visits and its share, then the requests and pages in the period. Four figures follow, each with its change against the same number of days just before: Total bot visits, Active bots, Pages visited and Failure rate.
- Bot visits over time is a line chart. Group it by Engine, Platform or Bot. The chart's menu has Show as share, which draws each line as its share of visits instead of a count.
- The list beside it has one row per group: a bar, the visits with their change, and the failure rate at the right. Hover a row to see its share and the bots in it.
Engine is the platform a bot belongs to, whatever it does: OpenAI's training bot counts under ChatGPT, as its search and user bots do. Other is named AI bots that belong to none of these engines, not unknown traffic.
Bots
One line sits on top: how many AI bots are known, from how many platforms, how many were seen on your site in the last 90 days, and how many follow robots.txt. Under it, the AI crawlers directory shows who runs each AI crawler, what it does with your content, and how to allow or block it.
Filter by type: All, Search, User Query, Training or Other. Each row has:
| Column | What it means |
|---|---|
| Robots.txt | Whether the bot follows robots.txt, by its operator's account: Yes, Not always, Token only or Not stated |
| Advice | Allow, for search crawlers and fetchers a person asked for, so you can be found and cited. Your choice, for training crawlers |
| Visits (90 days) | Its visits on your site |
Tap a row to open the bot's user agent, its operator's page, how its visits are verified, a robots.txt rule to allow or block it (with Copy) and firewall advice.
Pages
Four figures: Pages discovered, Cited in answers, Blocked and Errors.
- HTTP status counts each page once, by the status code AI bots got most on the last day it was visited: Success, Redirect, Blocked, Client error or Server error.
- Sitemap coverage says how many of the sitemap pages the latest site check read were visited by at least one AI bot, and lists the bots your robots.txt blocks. Run a Site check to fill it.
- Pages lists each page as Crawled, Blocked, Error or Not crawled, with its status code, bot visits, platforms, citations and last visit. Filter by status or by status code.
Blocked means the page refused AI bots at their last visit (401, 403, 407, 429 or 451).
Logs
Recent bot requests lists each request an AI bot made, newest first, kept for 30 days: Time, Bot, Path and Status. Above it are Bot requests, Active bots and Failure rate, and a chart of bot requests per day.
Filter by bot, status code, path or dates. Clear filters shows everything again, and Show more loads older requests.
Requests here come from a live log connection, such as Connect Cloudflare. Uploaded log files show on the Overview and Pages tabs instead.
Settings
The log set-up card, Upload log file, Include unverified traffic (visits claiming a bot from an address its operator does not publish, drawn as a separate line on the chart), Discard unverified traffic (drops them at upload) and, under Last upload, Delete all log file data.