Send AI crawler visits from Amazon CloudFront

If your site is served through Amazon CloudFront, your distribution can send its request logs to your project by itself. Nothing is installed on the site. The visits show on the AI crawlers page.

Only AI crawler requests are kept. Requests from people and from other bots are dropped when they arrive. No IP address, no user agent, no query string and no referrer is stored.

If you would rather upload a log file by hand, see the CDN log files article instead.

What you need

• A CloudFront distribution.
• Permission in AWS to create an Amazon Data Firehose stream and to edit the distribution's logging.
• A workspace Admin to make the token in Settings.

AWS may bill you for Firehose and for log delivery. CloudFront does not charge for turning on standard logging.

Set it up

1. Open Settings, find Amazon CloudFront and choose Generate token. Copy the token and the HTTP endpoint URL. The token is shown once.
2. In the AWS console, open Amazon Data Firehose in the US East (N. Virginia) region, which is where CloudFront standard logging looks for a stream. Choose Create Firehose stream.
3. Set Source to Direct PUT and Destination to HTTP Endpoint.
4. Fill in the destination:

• HTTP endpoint URL: the address from Settings.
• Access key: the token. Firehose sends it in the header X-Amz-Firehose-Access-Key.
• Content encoding: GZIP.
• Buffer hints: 1 MiB and 60 seconds work well. A large buffer makes each delivery bigger than we accept.

5. Firehose needs an S3 bucket for records it cannot deliver. Choose or create one, and set the backup to failed data only.
6. In the CloudFront console open your distribution, then the Logging tab, then Add and choose Amazon Data Firehose. Select the stream. Set Output format to JSON.
7. Under Field selection include at least: timestamp(ms) (or date and time), x-host-header, cs-uri-stem, cs(User-Agent) and sc-status. You can leave out the client IP, the query string and the referrer. We never read them.
8. Save. Visits appear on the AI crawlers page within a few minutes of the first AI crawler request. Last batch received in Settings shows the newest.

If nothing shows

• Firehose reports a 401: the token is wrong or was replaced, or the CloudFront connection is not switched on for your workspace yet. Generate a new token in Settings and paste it into the stream's Access key.
• Firehose reports a 413: the buffer is too large. Set it to 1 MiB.
• Firehose reports a 400: the output format is not JSON, or the fields in step 7 are not all selected.
• Waiting for first event in Settings: the stream is active but no AI crawler has fetched your site yet, or nothing has reached us. Check the stream's monitoring in AWS, and that the distribution's logging points at the stream.
• Visits missing for one host: only requests for your project's own website are kept. Check the website in Settings matches the host in your logs.
• Counts seem low: only visits from the AI crawlers we list are counted. Rows left out so far in Settings says how many were not.