Send AI crawler visits from Adobe Experience Manager

If your site runs on Adobe Experience Manager (AEM) as a Cloud Service, its CDN can send AI crawler visits to your project by itself, using Adobe's own HTTPS log forwarding. Nothing is installed on the site. The visits show on the AI crawlers page.

Only AI crawler requests are kept. Requests from people and from other bots are dropped when they arrive. No IP address, no user agent and nothing after a question mark in a page address is stored.

What you need

• An AEM as a Cloud Service program with self-serve log forwarding.
• Cloud Manager access to commit to the program's Git repository and run a config pipeline.
• A workspace Admin to make the token in Settings.

Log forwarding bandwidth counts towards your program's Network I/O, as Adobe's page says.

Set it up

1. Open Settings, find AEM log forwarding and choose Generate token. Copy the token and the address. The token is shown once.
2. In Cloud Manager, open the environment's Configuration, then Environment Variables, and add a variable named KLIENTBRAIN_AEM_TOKEN. Set its type to Secret, apply it to All services, and paste the token as the value.
3. In your program's Git repository, add config/logForwarding.yaml. Use your own address from Settings in place of the one below. Replace SECRET-REFERENCE with the variable name KLIENTBRAIN_AEM_TOKEN written the way Adobe writes secret references: a dollar sign, then the name inside double curly brackets, with nothing else. Adobe's log forwarding page shows the exact form. Setting aem.enabled to false sends only CDN logs, not application or Dispatcher logs.

kind: "LogForwarding"
version: "1"
metadata:
  envTypes: ["prod"]
data:
  https:
    default:
      enabled: true
      url: "https://YOUR-ADDRESS-FROM-SETTINGS"
      authHeaderName: "X-KlientBrain-AEM-Token"
      authHeaderValue: "SECRET-REFERENCE"
    aem:
      enabled: false

4. Deploy it with a Cloud Manager config pipeline (Targeted deployment, include Config). Full stack and web tier pipelines do not deploy this file.
5. Adobe's CDN checks the address once before it sends anything, and we answer that check for you. Visits appear on the AI crawlers page within a few minutes. Last batch received in Settings shows the newest.

Optional: add the referrer

AEM's CDN logs have no referrer by default. If you want one, Adobe lets you copy the Referer header into a log property with a CDN request transformation. Add this to config/cdn.yaml, then deploy it the same way. The property name must be lowercase referer.

kind: "CDN"
version: "1"
data:
  requestTransformations:
    rules:
      - name: klientbrain-log-referer
        when: "*"
        actions:
          - type: set
            logProperty: referer
            value:
              reqHeader: referer

Today the AI crawlers page does not use the referrer, so you can leave this out.

If nothing shows

• No visits at all: check the pipeline that deployed was a config pipeline, and that the environment variable is a secret applied to All services.
• A 401 in Cloud Manager's log: the token is wrong or was replaced. Generate a new one in Settings and update the variable.
• Visits missing for one host: only requests for your project's own website are kept. Check the website in Settings matches the host in your AEM logs.
• Counts seem low: only visits from the AI crawlers we list are counted. Rows left out so far in Settings says how many were not.
• Sandbox programs: we have not confirmed whether Adobe lets sandbox programs forward logs.