The URL tester card on Site check answers one question: can AI bots read this page? It works for your own pages and for any other public website.
How to use it
- Open Site check.
- In the URL tester card, type an address into Enter URL to check. You can leave off
https://. - Press Check.
You get two counts, Allowed and Blocked, and a table with one row per AI bot: Bot, Purpose, Operator and Status. Hover a status to see the rule that decided it, such as the Disallow: line.
How it is worked out
The tester reads the robots.txt file of the address's own site, once, and decides each bot for the path you typed. It does not open the page itself.
| What the site's robots.txt does | What you see |
|---|---|
Has rules for the bot, or a rule for everyone (*) | Allowed or Blocked for that path, with the rule on hover |
| Has no rule that matches | Allowed |
| Is missing (a "not found" answer) or is a web page | Allowed, shown as No robots.txt |
| Cannot be read, or the site answers with a server error | Blocked, shown as robots.txt error |
A bot marked Unverified is one we could not match to a current page from its operator. It is listed but not counted in Allowed or Blocked.
Limits
- The address must be a public website. Private and internal addresses are refused.
- A workspace can test 30 addresses a day.
- Only
robots.txtis read. A site can also block bots in other ways, such as at the server, which this does not see.
To check your whole site and get fixes, use the site check itself: Site check. For bots visiting your site, see AI crawlers.