AI answers change from one ask to the next.
Repeats and ranges
Each prompt is asked up to three times on each engine, depending on your plan and settings. Every figure is also worked out separately for each repeat. The lowest and highest of those form its range, which you see on hover. A wide range means the answers vary, and you should read the figure loosely.
What counts as a real change
The Overview always shows how a figure moved, and marks it Within normal range when this run's range overlaps the previous run's range. What changed lists a change only when this run's range sits wholly above or below the previous run's range. Anything smaller is treated as ordinary variation. For a single prompt on a single engine, a gain or loss counts only when it moves from none of the repeats to at least half of them, or the reverse.
That is why What changed can be short even when a figure moved by a point or two.
Answers that are not counted
Failed calls, and searches where Google showed no AI Overview or AI Mode answer, are stored but left out of every figure. The count is shown.
Quick checks only feed their own panel
Watch-list checks cover a handful of prompts, so they never feed the headline figures or trends. How the watch list works
When the model behind an engine changes
Vendors change models without notice. The app has three safeguards:
- Model breaks. Every answer records the model that answered. When it differs from the previous full run, the chart shows a dashed line, the engine shows "New model" instead of a change, and the report says what changed.
- Overlap test. In Settings, you can ask the current model and a candidate the same prompts side by side, to measure the step caused by the model alone. It shows the cost first and runs only after you confirm. Overlap runs never feed your figures.
- Health checks after every run. Warnings appear on the Overview for a high error rate, answers suddenly empty, far fewer sources per answer, Google's AI answers suddenly absent, or a changed model.
The method note
Every report ends with a short "how we measured this" note listing the engines, models, locations and repeats used. Settings, under How we measure, shows the same for the live project.