Why AI answers change every time: measuring visibility when results are volatile

Ask a generative engine the same buying question twice and you may receive two different shortlists: a brand named in the morning is missing by the afternoon, a citation moves, the order reshuffles. For anyone responsible for AI visibility measurement this is the central methodological problem, because a single screenshot cannot separate a real shift in how an engine treats your brand from the ordinary randomness of the instrument. Answer volatility is the run-to-run variation in what an engine says, names and cites for the same prompt under nominally identical conditions; once it is acknowledged, visibility stops being a property of one answer and becomes a rate estimated from repeated, comparable runs.

What follows separates the documented sources of that variation, reviews what empirical studies have measured, and sets out a method for defining the rate, sizing the sample, reading a Wilson interval and telling noise from change. Choosing prompts and competitors is covered in why you should start measuring AI visibility today, and choosing metrics in the GEO dashboard and its KPIs; the question here comes before both: how far can any of those numbers be trusted when the instrument itself is noisy?

Why does the same prompt produce a different answer?

A language model writes one token at a time: at each step it computes a probability distribution over possible next tokens and selects one, a process called decoding. OpenAI's API reference describes temperature as the sampling temperature, with higher values such as 0.8 making output more random and lower values such as 0.2 making it more focused and deterministic. Because a different token early on changes everything after it, two runs can diverge from the first sentence onward, and with them the list of brands an answer names.

Lowering the temperature does not cure this. OpenAI states that Chat Completions are non-deterministic by default and that, even with its seed parameter, determinism is not guaranteed. In September 2025 Horace He and colleagues at Thinking Machines Lab reported 80 distinct texts among 1,000 temperature-zero completions of one prompt from an open-weights Qwen3 model, and traced the divergence mainly to inference kernels whose results depend on batch size, and therefore on server load. In consumer interfaces such as ChatGPT or Google AI Overviews, moreover, these settings are chosen by the provider, not by the person measuring them.

How does retrieval add a second layer of variation?

Engines that answer with citations, such as ChatGPT search and Google's AI features, retrieve web results before they answer, and retrieval is itself variable. OpenAI's Help Center explains that ChatGPT search typically rewrites a question into one or more targeted queries sent to search providers, and Google's documentation says that AI Overviews and AI Mode may use a "query fan-out" technique, issuing multiple related searches across subtopics and data sources. When the rewritten queries differ between runs, so do the candidate pages and the brands an answer can mention. The index moves too: Google notes that crawling can take from a few days to a few weeks.

Even the presence of an AI answer varies. Google states that AI Overviews appear only when its systems judge them additive to classic Search and therefore often do not trigger. How this layer reshapes the results page is discussed in Google AI Overviews and zero-click search; for measurement, "no AI answer was shown" is an outcome in its own right and must not be merged with "the answer did not name us".

Does it matter who is asking, from where and with what history?

It does, and the documentation names the channels. ChatGPT search may use an approximate location derived from the IP address to provide local results and, when memory is enabled, relevant saved memories when rewriting a search query; Google announced in January 2026 an opt-in Personal Intelligence feature that lets AI Mode draw on Gmail and Google Photos, first for paying subscribers in the US, and widened its US availability in March 2026. The answer a marketing manager sees on a personal account is therefore drawn from a personalized distribution, not the one a typical buyer meets. Language and market act the same way at a larger scale, so each tracked prompt needs a fixed market and language, and each combination is its own series; the reasons languages diverge are examined in multilingual GEO.

When does a change come from the engine rather than from you?

Sampling and retrieval scatter results around a stable average; product updates move the average itself. On 27 January 2026 Google announced that Gemini 3 was becoming the default model for AI Overviews globally, and OpenAI publishes Model Release Notes that log model changes in ChatGPT. A series crossing such a date contains two regimes and needs a dated log of external events beside it. Differences between engines form a separate axis, examined in why each AI engine sees your brand differently.

Source of variation	What it changes	Timescale	How to control or record it
Sampling and decoding	Wording, order, brands named first	Every run	Repeat runs and estimate a rate
Prompt wording	How the need is interpreted	When the text changes	Freeze the text; track rewordings separately
Query rewriting, fan-out	Documents retrieved and cited	Every run	Repeat runs; record cited sources
Index and crawl state	Pages the engine can find	Days to weeks	Date content changes; allow for crawl lag
Location, account, memory	Personal context of the answer	Per user and session	Fixed market and language; no personal history
Answer triggering	Whether an AI answer appears	Per query, over time	Record "no answer" as its own state
Model and product updates	The whole response distribution	Irregular steps	Annotate the date; split the series
Measurement failure	Whether a result exists	Unpredictable	Mark "not measured"; never count as zero

What do empirical studies show about how much answers vary?

SparkToro has published an industry measurement of this variability. In research released in January 2026, Rand Fishkin, working with Patrick O'Donnell of Gumshoe.ai, had about 600 volunteers run 12 prompts through ChatGPT, Claude and Google's AI results, and normalized 2,961 responses into ordered brand lists. For ChatGPT and Google's AI they report a less than one in a hundred chance of the same list appearing twice and roughly one in a thousand of it appearing in the same order; they dismiss any "ranking position in AI" as close to useless but consider a visibility percentage across dozens to hundreds of prompts, each run several times, a reasonable metric. The authors state that they are not professional researchers, the study appeared on SparkToro's blog without peer review, and participants kept their default settings, with location not controlled.

A 2026 arXiv preprint by Ronald Sielinski sampled citations from Perplexity, OpenAI's SearchGPT and Google Gemini on three consumer-product topics over nine days. It finds that citation counts vary considerably between samples, that many apparent differences between domains fall inside bootstrap confidence intervals and that rankings are unstable, and it argues that visibility metrics are sample estimates of an underlying distribution. Sclar and colleagues (ICLR 2024) add that meaning-preserving formatting changes moved one open model's accuracy by up to 76 points. Two design rules follow: measure presence rather than rank, and freeze prompt wording, because a reworded prompt is a different instrument.

What exactly is a visibility rate?

A rate is meaningful only when its definition is complete: for a given prompt, market, language, engine and window, the share of measured answers in which the brand is mentioned. A citation rate is built the same way, with answers citing the brand's domain in the numerator, and the two stay separate because they reward different work. Each element is a control: the prompt text is frozen (choosing it is the subject of prompt research), market and language are fixed, cadence is regular, and each answer counts once however often it repeats the name.

A pooled category figure is legitimate if the report states what was pooled, and because generative engine optimization ultimately aims to move such rates, their definition belongs in writing before the first report. The competitive version, with brands in the denominator, is developed in share of voice in AI search.

How many runs does a rate need?

Precision grows with the square root of the number of runs. For a true rate near 50 percent, the half-width of a 95 percent Wilson interval is about 26 percentage points with 10 measured answers, 17 with 30, 10 with 100 and 5 with 400, so halving the uncertainty costs four times the measurements. A weekly figure for one prompt can therefore reveal only large movements, while a figure pooled across thirty prompts can detect much smaller ones.

These formulas assume independent draws. Runs taken minutes apart may share retrieval results and carry less information than runs spread across days. Pooling raises a related issue: if a pooled figure is meant to speak for buyer questions in general rather than for the fixed prompt set, runs cluster within prompts whose true rates differ, and the effective sample size falls below the raw count of answers. Spreading runs across the window and reporting per-prompt and pooled figures together keep these distortions visible. The sample size should also be fixed in advance: the Sielinski preprint observed that an interval can narrow and then widen again as samples accumulate, so stopping when it first looks narrow enough understates the uncertainty.

What does a Wilson interval actually tell you?

A 95 percent confidence interval comes from a procedure that, over many repeated samples, would cover the true rate about 95 percent of the time; it describes the method, not a promise about one interval. The textbook Wald interval, the observed share plus or minus 1.96 standard errors, fails where GEO measurement lives: with zero mentions in ten answers it collapses to zero width, implying a certainty ten runs cannot give.

The Wilson score interval, introduced by Edwin B. Wilson in 1927, inverts the score test instead; its center is pulled slightly toward one half and it never leaves the range from zero to one. Reviewing the alternatives in Statistical Science (2001), Brown, Cai and DasGupta found Wald coverage erratic and recommended Wilson or Jeffreys intervals for small samples. In practice, zero mentions in ten answers gives roughly 0 to 28 percent, which reads as "rarely, possibly never" rather than "never"; 12 in 20 gives about 39 to 78 percent; 60 in 100 gives about 50 to 69 percent. The same observed 60 percent thus carries very different weight depending on how many runs stand behind it.

A single AI answer is an observation; visibility is a rate, and a rate is not reportable until its sample size, its interval and its unmeasured runs are reported with it.

How do you tell noise from a real change?

Consider a hypothetical prompt cluster with mentions in 8 of 20 measured answers one week and 12 of 20 the next. The move from 40 to 60 percent looks decisive, yet a two-proportion test gives a z statistic of about 1.3, short of the conventional 1.96; with the same shares over 60 answers per period, z reaches about 2.2. Checking whether intervals overlap is far stricter than a 5 percent test: Knol, Pestman and Grobbee (European Journal of Epidemiology, 2011) noted that, for independent estimates with equal variances, non-overlapping 95 percent intervals arise under the null with a probability of only about 0.006, so overlap can still hide a real difference.

Three habits complete the picture. Persistence: a difference seen in one run is an observation, and only one that holds across consecutive windows is sustained. Multiplicity: with fifty prompts tested weekly at the 5 percent level, two or three are expected to cross the threshold by chance alone in a week when nothing has changed. Context: check the event log before interpreting any move. Showing that your own content caused a change requires a control group, and the designs are set out in GEO experiments.

Why must "not measured" stay separate from zero?

Every scheduled run ends as a measured answer, a measured absence of any AI answer, or no measurement because a provider failed or a request limit was reached. Only the first belongs in a mention rate's denominator; the second is a finding about the engine, the third a gap in the instrument, and counting it as zero manufactures a decline. Likewise, when the prompt set, aliases or parsing rules change, the series needs a new methodology version. Distinguishing a broken number from broken data is treated in report anomaly alerts.

Rocketly's GEO Visibility module records these distinctions explicitly. Each run is stored as measured, no answer or not measured, and "not measured" is never counted as zero; with at least two measured runs a 95 percent Wilson interval is shown; weekly changes are labeled distinguishable or not distinguishable; answer differences are marked observed for a single run and sustained when they persist; and the versioned methodology is documented on a "How measurement works" page. Google AI Overview is measured weekly as standard, ChatGPT and Perplexity are in a controlled rollout, and Gemini, Claude and Grok are not measured.

If you would rather report visibility as a rate with its uncertainty and its gaps than as a screenshot, you can create a Rocketly account and evaluate Rocketly GEO Suite, a separate add-on, against your own prompt set.

Frequently asked questions

Can AI answers be made stable by setting the temperature to zero?

Not in the products buyers use, because there the decoding settings are chosen by the provider, not by the user. Even through an API, OpenAI states that determinism is not guaranteed, and published experiments have found divergent outputs at temperature zero. Retrieval, personalization and model updates would remain in any case.

How often should AI visibility be measured?

Consistency matters more than frequency: a fixed cadence, such as weekly, keeps reporting windows comparable. The number of measured answers per reporting window sets the width of the interval, and runs spread across the window are more informative than the same number taken in one burst.

What should every visibility report state?

The engine, market, language, prompt set and window; the number of measured answers and the confidence interval; the number of runs that could not be measured; and the methodology version in force. Dated external events, such as a model update, belong beside the figures, because without them two reports cannot be compared even when they show the same number.