Send AI crawler visits from Akamai

If your site is served through Akamai, DataStream 2 can send your request logs to your project by itself. Nothing is installed on the site. The visits show on the AI crawlers page.

Only AI crawler requests are kept. Requests from people and from other bots are dropped when they arrive. No IP address, no user agent, no query string and no referrer is stored.

What you need

• An Akamai property with DataStream 2 turned on.
• A workspace Admin to make the token in Settings.

DataStream 2 may be billed by Akamai. Check your contract.

Set it up

1. Open Settings, find Akamai log connection and choose Generate token. Copy the token and the Endpoint URL. The token is shown once.
2. In Akamai Control Center, create a stream for your property. When asked for the data set, include at least these fields: Request time (reqTimeSec), Request host (reqHost), Request path (reqPath), User-Agent (UA) and HTTP status code (statusCode). You can leave out the client IP, the query string and the referrer. We never read them.
3. On the Log format tab choose JSON, not Structured.
4. For the destination choose Custom HTTPS, then fill in:

• Endpoint URL: the address from Settings.
• Authentication: Basic. The user name is klientbrain and the password is your token.
• Content type: application/json.
• Send compressed data: either way works.

5. Choose Validate & Save. Akamai sends one test message, and we answer it once the token is right.
6. Activate the stream. Visits appear on the AI crawlers page within a few minutes. Last batch received in Settings shows the newest.

Instead of Basic authentication you can choose None and add a custom header named X-KlientBrain-Akamai-Token whose value is the token. Akamai does not allow a custom header called Authorization, which is why Basic is the first choice.

If nothing shows

• Validate & Save fails: the token is wrong or was replaced, or the Akamai log connection is not switched on for your workspace yet. Generate a new token in Settings.
• A 401 in Akamai's delivery log: the same. Update the password.
• No visits at all: check the stream is active, that it covers the property that serves your site, and that the four fields in step 2 are included.
• Visits missing for one host: only requests for your project's own website are kept. Check the website in Settings matches the host in your logs.
• Counts seem low: only visits from the AI crawlers we list are counted. Rows left out so far in Settings says how many were not.