Custom log integration

If your site is not on Cloudflare or WordPress, your own server can send AI crawler visits to your project. Count the visits from AI crawlers in your logs, then send the counts in batches to your project's address, signed with a secret. They show on the AI crawlers page exactly as an uploaded log does.

Set it up

1. Open Settings, then Connected Apps, then Custom log integration. Only a workspace Admin can see this tab.
2. Choose Generate signing secret and copy it. It is shown once. Keep it on your server, never in a web page.
3. Choose Copy the address. It is the address for this project only.
4. On your server, send a batch with one of the examples below. Last batch received on the tab shows it within a few seconds. Choose Check now to look again.

What a batch looks like

A batch counts visits per day, crawler, page and status code, with the first and last visit of each day. Send at most 1 MB in one request.

{
  "stats": { "files": 1, "bytes": 0, "lines": 6, "aiVisits": 6, "unreadLines": 0, "noAgentLines": 0, "formats": { "combined": 0, "common": 0, "cloudflare": 0 } },
  "days": [ { "day": "2026-10-20", "firstAt": "2026-10-20T09:00:00Z", "lastAt": "2026-10-20T10:30:00Z" } ],
  "rows": [
    { "day": "2026-10-20", "bot": "GPTBot", "path": "/pricing", "status": 200, "visits": 3 },
    { "day": "2026-10-20", "bot": "ClaudeBot", "path": "/", "status": 200, "visits": 3 }
  ]
}

• bot is the crawler's name as the AI crawlers page lists it, for example GPTBot, ClaudeBot or PerplexityBot. If even one row names any other bot, the whole batch is refused with a 400 and nothing in it is stored. Leave out rows for other bots before you send.
• path is the page without anything after a question mark. Anything after one is removed.
• A batch whose day and times overlap one already sent replaces it, so send each stretch of time once.
• unreadLines and noAgentLines are the lines your script could not read and the lines with no user agent. Where your workspace shows why lines were left out, they appear under Dropped on the tab.

Sign each request

Send the header X-KlientBrain-Signature: t=<time>,v1=<signature>, where <time> is the time in Unix seconds and <signature> is the HMAC-SHA256 of the time, a full stop and the exact body, made with your signing secret and written in hex. A request signed more than 5 minutes away from our clock is refused, so sign each request when you send it.

With curl:

SECRET="your signing secret"
URL="the address from the tab"
BODY="$(cat batch.json)"
TS="$(date +%s)"
SIG="$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* //')"
curl -sS -X POST "$URL" \
  -H "Content-Type: application/json" \
  -H "X-KlientBrain-Signature: t=$TS,v1=$SIG" \
  --data-binary "$BODY"

With Node.js 18 or later:

import { createHmac } from "node:crypto";
import { readFileSync } from "node:fs";

const secret = process.env.KB_SIGNING_SECRET;
const url = process.env.KB_INGEST_URL;
const body = readFileSync("batch.json", "utf8");
const t = Math.floor(Date.now() / 1000);
const v1 = createHmac("sha256", secret).update(`${t}.${body}`).digest("hex");

const res = await fetch(url, {
  method: "POST",
  headers: { "Content-Type": "application/json", "X-KlientBrain-Signature": `t=${t},v1=${v1}` },
  body,
});
console.log(res.status, await res.json());

Sign the body exactly as you send it. If your code changes the JSON after signing it (a different spacing or key order), the signature no longer matches.

Ready-made examples

You do not have to write the counting and signing yourself. Four small examples do both. Each is one file with no packages to install, and each counts after the response and sends from a timer or a background thread, so a visitor never waits on it.

Each needs two settings on your server: the address you copied from the tab as KB_INGEST_URL, and your signing secret as KB_SIGNING_SECRET. Add the example to your site, deploy, then look at Last batch received on the tab. While there is something to send, a batch goes out every five minutes (a project takes 288 a day).

• They send only the crawlers on the AI crawlers page, so the 400 above for another bot does not happen.
• Each stretch of time is sent once. Reading the same log twice counts it once, but two different logs that cover the same hours replace each other.
• The Next.js example counts in the memory of one running copy of your site and sees the status your middleware returns, not the status the page then answers. On a host that starts many short lived copies, use the relay on your logs instead.
• The relay reads the "combined" log format, the one with the user agent. Read files oldest first.

The examples are in the log-ingest-examples folder of KlientBrain's integrations. They are not on a public download page yet, so ask us for the folder.

What the answers mean

• 201: stored. The answer gives the visits and days stored, and why any lines were left out.
• 200 with "duplicate": true: the same batch arrived again, so it was counted once.
• 400: the batch is not in the shape above, or a row names a bot that is not on the list. Nothing in the batch is stored. The message says what is wrong.
• 401: the signature is missing, does not match, or is too old, or the project has no signing secret.
• 404: the address is wrong.
• 413: the body is over 1 MB. Send fewer rows at a time.
• 429: too many requests. A project can take 30 batches a minute and 288 a day; one sending address can make 120 requests a minute; and after 10 requests in a minute with a signature that does not match, the project's address refuses everything until the minute ends. For the per-minute limits, wait for the time in the Retry-After header. For the daily limit, try again the next day (UTC).

Good to know

• No IP addresses, cookies or user agents are sent, only counts. We keep counts per day, crawler, page and status, as for an uploaded log.
• Replace secret makes a new secret, and batches signed with the old one are refused from then on. Turn off removes the secret; the visits already received stay.