GEO experiments: proving that a content change improved AI visibility

A team rewrites its comparison page, and three weeks later its brand appears in more AI answers to the questions that page addresses. The tempting conclusion is that the rewrite worked, yet the same weeks may also have held a model update, a competitor's launch, a seasonal shift and an index refresh, any of which could draw the same chart. A GEO experiment is a planned comparison designed to estimate how much of a change in AI visibility can be attributed to a specific intervention rather than to what would have happened anyway; that unobserved "what would have happened" is the counterfactual, and each design below is a different way of approximating it.

The word "proving" therefore needs a precise meaning. In generative engine optimization the randomized user split familiar from landing pages is rarely available, so certainty is not on offer; what is achievable is a graded level of evidence, from bare association to a replicated, controlled effect, and an honest report says which level it reached. The statistics of comparing two versions are covered in what A/B testing is, and the unreliability of a single AI answer in why AI answers change every time; both are assumed here.

What does it mean to prove an effect you cannot observe directly?

In the potential-outcomes framework, the effect of a change on a prompt's mention rate is the difference between the rate observed after the change and the rate the same prompt would have shown, over the same period, without it. Only one of the two is ever observed, so every causal claim rests on a stand-in for the missing one, such as the same prompts before the change or other prompts that were not affected, and is exactly as credible as that stand-in. Association means that visibility and the change moved together in time; causation means that visibility moved because of the change, and only design, not confident wording, closes the gap between them.

The framing also disciplines the outcome. An experiment needs one primary metric chosen in advance, usually a mention rate or a citation rate on a defined prompt set, engine, market and language. The two respond to different work, as explained in mention versus citation in AI answers, and choosing between them after seeing the data turns a test into a search for a flattering number.

Why is a simple before-and-after comparison so weak?

A before-and-after comparison uses the treated prompts' own past as the counterfactual, assuming that nothing else relevant changed, and in GEO that assumption fails for identifiable reasons. Engines change models: Google announced on 27 January 2026 that Gemini 3 was becoming the default model for AI Overviews globally, so a series spanning that date compares two systems. Competitors publish, buyer questions follow seasons, and the index refreshes on its own schedule; Google notes that crawling can take from a few days to a few weeks, so even the moment an engine first sees your change is uncertain. The third-party pages engines cite, the subject of source gap analysis, can gain or lose weight without any action on your side.

A subtler confounder is regression to the mean. Pages are usually chosen for a content refresh because their visibility was unusually poor, and a measurement selected for being extreme tends to be followed by one closer to the average even if nothing is done. Barnett, van der Pols and Dobson describe in the International Journal of Epidemiology (2005) how this effect makes natural variation look like real change, especially when only a subgroup selected on its baseline value is followed up. On its own, then, a before-and-after rise is evidence of association and nothing more.

How does SEO split testing establish causality, and why does it transfer only partly?

SEO testing met a similar problem with page-level experiments. As SearchPilot describes its method, a search crawler is effectively a single "user", so the test splits pages built on one template, rather than visitors, into control and variant groups that are statistically similar in traffic, variability and seasonality; the change goes live only on the variants, and a model combining the pre-test relationship between the groups with the control group's live performance forecasts what the variants would have done without it. Because both groups run simultaneously, an algorithm update or a seasonal swing hits them equally. SearchPilot reports that such tests generally need at least hundreds of pages on one template and typically take two to four weeks to reach significance, while Google's website-testing guidance adds the hygiene rules: never show Googlebot different URLs from people, prefer rel="canonical" to noindex, and use temporary 302 rather than permanent 301 redirects.

The logic transfers; the conditions mostly do not. The outcome in GEO is a prompt's answer rather than a page's traffic, and the mapping is many-to-many: one answer can draw on several of your pages and on third-party ones, while one page can influence many prompts. Causal inference calls this interference; it violates the no-interference component of what Rubin named the stable unit treatment value assumption (SUTVA), under which one unit's outcomes must not depend on the treatments other units receive. A brand's prompt set is also usually far smaller than the page sets such tests require, and its outcome is sampled through repeated runs rather than counted from complete traffic logs. A randomized page-level split is therefore realistic only for sites with many near-identical template pages whose prompts are cleanly separable, and even then the result holds only for the engine, market and period measured.

What do control prompts add?

For teams that cannot randomize pages, the most practical upgrade is a set of control prompts: questions similar in intent, language and market to the treated ones but unrelated to the content that changes. The comparison then follows the logic of difference-in-differences, a quasi-experimental design that compares the change over time in a treated group with the change in an untreated one. Consider a hypothetical in which treated prompts move from a 30 to a 50 percent mention rate while control prompts move from 32 to 42 percent over the same weeks. The naive effect is 20 points; the difference-in-differences estimate is 10, because half of the rise also occurred where nothing was changed.

The estimate is only as good as its assumptions. As Columbia University's guide to population-health methods sets out, the design requires parallel trends, meaning that without the intervention the gap between the groups would have stayed constant, as well as no spillover and an allocation not driven by the baseline outcome. In GEO terms: plot several pre-period weeks to check that the groups moved together; choose controls the changed page is unlikely to affect, since a stronger page can lift neighboring questions; and fix both groups before the change. The difference also carries sampling uncertainty, so report it with an interval.

Can staggered rollouts do better?

When a change can be applied to several comparable page groups, staggering it is often more informative than changing everything at once. If group A changes in week one and group B in week five, the design predicts that A's prompts move after week one while B's stay flat, and that B's move after week five. An effect that appears twice, each time after its own rollout, is far harder to blame on a model update or a season, which would hit both groups on the same date. A related idea, estimating the counterfactual from untreated series observed over the same period, underlies synthetic-control methods: CausalImpact, the Bayesian structural time-series approach that Brodersen and colleagues at Google published in the Annals of Applied Statistics in 2015, builds such a synthetic control from related series that the intervention did not affect.

Design	What it can support	Requirements	Typical pitfall
Single before/after	Association in time	A stable baseline of several weeks	Model update, season or index refresh in the same window
Interrupted time series	A break in level or trend at the change date	Many weekly points on both sides; dated event log	Another event on or near the same date
Control prompts (difference-in-differences)	An effect beyond shocks shared with controls	Parallel pre-trends; no spillover; groups fixed in advance	Controls influenced by the same page or domain
Staggered rollout	A replicated effect following each rollout	Several comparable page groups; spacing between waves	Too few groups, or waves too close to separate
Randomized page-level split	A causal estimate for that engine, market and period	Hundreds of template pages; separable prompts; enough runs	Interference between pages and answers; low power
Wording variants of a prompt	Differences between phrasings of one question	Same engine and window; enough runs per variant	Read as the effect of content, which it does not measure

What must be written down before the change goes live?

Pre-registration means specifying the hypothesis and analysis plan before the outcome is observed. Nosek and colleagues argued in PNAS (2018) that it separates genuine predictions from postdictions, explanations built after seeing the data that hindsight bias makes easy to mistake for tested results. A GEO pre-registration can be a one-page note: the change and the mechanism by which it should work; treated and control prompts; engine, market and language; the primary metric; pre and post windows, including a lag for recrawling; the smallest effect worth acting on; and the decision rule. The anatomy of a usable hypothesis is discussed in turning analyses into A/B test hypotheses, and it applies here with prompts in place of creatives.

A GEO result counts as evidence only when the hypothesis, the comparison group and the success metric were fixed before anyone looked at the numbers.

How long does a GEO experiment need to run?

Duration follows from three constraints. The pre-period needs enough weekly points to show the trend both groups were already on, and the post-period must exclude a lag window for the engine to see the new content. And each period needs enough measured answers to detect the effect you care about: by the standard two-proportion formula, at 80 percent power and a 5 percent two-sided threshold, detecting a rise from 40 to 60 percent needs about 97 measured answers per period, and a rise from 40 to 50 percent about 388. With 25 treated prompts measured once a week, the larger effect needs roughly four weeks on each side of the change and the smaller one about sixteen. These figures cover a simple before-and-after comparison of the treated prompts; a difference-in-differences estimate also carries the control group's sampling noise and needs roughly twice as many answers in each group, and correlation between runs of the same prompt pushes all of these figures higher still.

Once the duration is set, keep it. Johari, Pekelis and Walsh showed that conventional p-values and confidence intervals become unreliable when people monitor a test continuously and stop as soon as it looks significant, and Sielinski's 2026 preprint on AI-visibility measurement likewise recommends committing to a fixed sample size in advance. Looking at the data during the test is acceptable; moving the end date because of what you saw is not.

How do you keep track of everything else that changed?

An experiment is interpretable only against a dated record of other events that could explain its result: content, PR, product and campaign changes, competitor launches and engine updates. Google, for instance, recommends waiting at least a full week after a core update completes before analyzing its effect on a site; the same caution applies when an answer engine changes its model mid-experiment. With a log, an overlapping event becomes a documented limitation instead of a hidden one. The baseline must also exist before the change, one more reason to begin early, as argued in why you should start measuring AI visibility today.

Rocketly's GEO Visibility module supports this observational discipline and labels its limits. Notes record content, PR, product or campaign changes; the Observational comparison panel shows mention metrics before and after a note in a window of 3 to 28 days and labels the result as association, not cause and effect; and Question variants track an alternative wording of a prompt separately and compare rates on the same engine over a 30-day window, which the product itself states is not a causal experiment. Rocketly has no A/B testing or holdout feature, so control groups, staggered rollouts and the causal judgment remain part of your own design.

What can you claim at the end, and in which words?

The claim should match the design. A before-and-after rise supports "visibility on these prompts rose after the change". A difference-in-differences estimate whose interval excludes zero, with plausible parallel pre-trends, supports "the rise exceeded what similar untreated prompts showed, which is consistent with an effect of the change". Only a replicated effect across staggered rollouts, or a randomized page-level split, supports "the change caused an estimated increase of so many points on this engine, in this market, during this period". None of these licenses claims about other engines, languages or future model versions, and a null result means "no effect large enough to detect with this sample", not proof that the change does nothing.

Reported this way, results stay comparable across experiments and can feed the views described in the GEO dashboard and its KPIs and share of voice in AI search. To apply this discipline to measured data, you can open a Rocketly account and try Rocketly GEO Suite, a separate add-on that measures Google AI Overview weekly as standard, with ChatGPT and Perplexity in a controlled rollout, and labels its before-and-after and variant comparisons as observational rather than causal.

Frequently asked questions

Can I run a true A/B test on AI answers?

Not in the user-split sense, because you cannot show different versions of a page to different engine "users", and showing Googlebot a different set of URLs from the one people see counts as cloaking under Google's testing guidance. A randomized split of pages is possible only with many comparable template pages and separable prompts, so for most teams control prompts and staggered rollouts are the realistic options.

How many control prompts do I need?

Roughly as many as treated prompts, matched in intent, language and market, because the precision of a difference is limited by its noisier side. Too few controls make the control rate unstable, and the estimate inherits that instability.

What if the effect appears on only one engine?

Report it for that engine alone. Engines differ in models, retrieval and interfaces, so an effect on one is a legitimate finding that does not generalize automatically; the others are separate hypotheses for a later test.