Send AI crawler visits from Fastly

If your site is served through Fastly, log streaming can send your request logs to your project by itself. Nothing is installed on the site. The visits show on the AI crawlers page.

Only AI crawler requests are kept. Requests from people and from other bots are dropped when they arrive. No IP address, no user agent, no query string and no referrer is stored.

What you need

• A Fastly service that serves your site.
• A workspace Admin to make the token in Settings.

Fastly may bill for log streaming. Check your plan.

Set it up

1. Open Settings, find Fastly log connection and choose Generate token. Copy the token, the URL and the log format. The token is shown once.
2. In Fastly, open your service, then its logging settings, and create an HTTPS logging endpoint.
3. Fill in:

• URL: the address from Settings.
• Content type: application/json.
• Custom header name: X-KlientBrain-Fastly-Token. Custom header value: your token. If you prefer, name the header Authorization and use Bearer followed by the token.
• JSON log entry format: either works (Array of JSON, or Newline delimited).
• Log format: paste the line below, exactly as it is.

4. Save the endpoint and activate the new version of the service. Fastly first checks that you control the address by asking it a question at /.well-known/fastly/logging/challenge, and we answer it for you.
5. Visits appear on the AI crawlers page within a few minutes. Last batch received in Settings shows the newest.

The log format, one line per request:

{"epoch_seconds":%{time.start.sec}V,"host":"%{json.escape(req.http.Host)}V","path":"%{json.escape(req.url.path)}V","status":%>s,"user_agent":"%{json.escape(req.http.User-Agent)}V"}

It names five fields: the time, the host, the path, the status and the user agent. We never ask for the client address, the query string or the referrer.

If nothing shows

• Fastly says it cannot verify the address: the Fastly log connection is not switched on for your workspace yet.
• A 401 in Fastly's delivery log: the token is wrong or was replaced. Generate a new one in Settings and update the header value.
• No visits at all: check the version with the endpoint is active, that the endpoint is on the service that serves your site, and that the log format is pasted exactly.
• Visits missing for one host: only requests for your project's own website are kept. Check the website in Settings matches the host in your logs.
• Counts seem low: only visits from the AI crawlers we list are counted. Rows left out so far in Settings says how many were not.