Upload logs from CloudFront, Fastly, Vercel or Netlify

If your site is served by Amazon CloudFront, Fastly, Vercel or Netlify, export its request logs from there and choose Upload log file on the AI crawlers page. You can pick several files at once, and files ending in .gz are fine. The file is read in your browser. Only counts of AI crawler visits are sent to your project: no IP addresses, no user agents, and nothing after a question mark in a page address.

Amazon CloudFront

1. Turn on standard logging for your distribution, and keep the default fields, or at least these: date, time, cs-uri-stem, sc-status and cs(User-Agent).
2. Download the log files CloudFront has written. It delivers several files an hour, so choose all the files for the days you want.
3. Upload them. Each file starts with a #Fields: line, which tells us where each value is, so the order of the fields does not matter. A file without that line cannot be read.

CloudFront's description of the fields: Standard logging reference.

Fastly

1. Add a log streaming endpoint for your service and choose where Fastly should write the logs, such as your own file storage.
2. Set the log format to the combined log format, which includes the user agent:

%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"

3. Download the files and upload them. The first field of each line is the visitor's address. It is skipped and never stored.

Fastly lists this format with its others: Useful log formats.

Vercel

1. In your Vercel team, create a Log Drain with a custom endpoint and choose NDJSON as the format. Include the static source, and the lambda and edge sources if your pages are served by functions.
2. Save what your endpoint receives to a file, one entry per line.
3. Upload the file. Entries that are not requests, such as build output, are skipped. When one request wrote several entries, it is counted once.

Vercel's description of the fields: Log Drains reference.

Netlify

1. In your site's settings, create a Log Drain with a General HTTP endpoint and include traffic logs.
2. Leave Exclude personally identifiable information (PII) unticked. With it ticked Netlify leaves out the user agent, and no crawler can be recognised.
3. Save what your endpoint receives to a file, one entry per line, and upload it.

Netlify's description of the fields: Log drains.

If nothing is found

• Check the file has the page request lines, not only build or function output.
• Check the lines include the user agent. Lines without one are counted, but no crawler can be named from them.
• Uploading the same days again does not count them twice.