Backtesting an ad rule before you turn it on

Tuesday, 9:15 a.m. A performance marketer finally writes the rule she has been mulling over for three weeks: "If ROAS drops below 1.0 over the last 7 days, pause the campaign." Next to the save button sits a second one: what would this rule have done in the last 28 days? She clicks it. The result on screen looks nothing like the picture in her head: the rule would have fired 46 times, touched 9 of 12 campaigns at least once, and among the things it would have paused is the retargeting campaign that drove the most revenue all month. The reason is mundane: that campaign's ROAS reliably dips to 0.8 on Monday mornings and climbs back to 2.4 by Wednesday afternoon. The numbers here are illustrative. What is not illustrative is the outcome: had the rule been live, the account's best campaign would have shut down on a Monday morning while nobody was watching the dashboard.

What follows treats backtesting not as a confidence ritual but as an instrument that measures a rule's reach and its blind spots. We start by separating what a backtest proves from what it cannot prove. Then we look at where the threshold number actually comes from, what a 28-day window hides, and the feedback problem — the fact that a rule rewrites the very history it is tested against. After that: where simulation departs from backtesting, how guardrails trim the trigger count, how to read that count as a decision table, what preview-only mode catches that a backtest never will, and which evidence you collect once the rule is live. The last section covers when to retire a rule.

What does a backtest actually prove?

A backtest answers one question: applied to the history I already have, how many times and on which campaigns would this rule have fired? The "Backtest before you turn it on" screen in Ads Automation reports exactly that — the last 28 days, the trigger count, the names of the matching campaigns, without touching anything. It sounds like a small piece of information, but it catches two expensive mistakes in the first minute: a rule so narrow it will never run, and a rule so broad it would sweep half the account.

What it cannot prove is outcome. A backtest will never tell you "ROAS would have been higher with this rule on," because if the rule had paused that campaign on 12 March, the data from 13–28 March would never have existed. The history you hold is a history in which the rule did not intervene. Drawing that line early stops you from loading the backtest with weight it cannot carry. The same discipline shows up in software: running an automation against past inputs before it goes live is the cheapest way to see how it behaves, which is why testing an AI agent before launch follows the same logic.

The practical upshot: a backtest is a scope measurement, not a performance forecast. It answers "how many campaigns, how often, which ones." Read that way, the 46 on screen is a warning, not a promise.

Where does the threshold number really come from?

Most rules fail not because they were written badly but because they were written against the wrong number. Where did the 1.0 in "pause if ROAS is under 1.0" come from? Usually from a round figure someone said in a meeting, with no relationship to the account's actual distribution. If your median ROAS is 0.9, that threshold pauses half your campaigns. If the median is 3.2, the same threshold never fires all year and the rule sits there as decoration.

So pick thresholds from the distribution, not from intuition. The "Pick the threshold from your data" view shows how a metric is distributed across your campaigns, lets you turn the median into a threshold in one click, and tells you how many campaigns that threshold catches. The order of reasoning inverts: you decide how many campaigns you are willing to touch, then take the threshold that produces that number. A lower-quartile threshold gives you a selective rule; the median gives you a broader sweep.

Factor in how noisy the metric is. Daily spend is relatively stable, ROAS jumps around at low conversion volume, and CPL can double on a single form fill. A tight threshold on a noisy metric means the rule catches a different campaign every week. Sales teams meet the same calibration problem from the other side; calibrating a lead scoring model also begins by asking whether the cut-off reflects reality.

What does the 28-day window hide?

The backtest window is fixed at the last 28 days. That is enough to capture day-to-day variance and the weekday/weekend rhythm; it is not enough to capture seasonality. If a product launch, a promotional period or a stockout falls inside the window, the backtest learns the rules of that period and shows you nothing about an ordinary month. The reverse is just as risky: if the window landed on a quiet stretch, the rule gets approved without ever demonstrating what it does under load.

The second limit is data quality. Before a rule can be saved, the data health gate verifies sync freshness, 28 days of data coverage and the account currency; stale or partial data does not produce rules. That check protects the base the backtest reads from, but it cannot fix attribution: if conversions are credited to the wrong campaign, the backtest flags the wrong campaign too. Getting UTM parameters and campaign tracking right before you write rules is what makes every row the backtest reads meaningful.

Third, the learning phase. While a platform is still learning a new campaign, its metrics are unstable, and a rule firing in those days kills a campaign before it matures. The guardrail option that skips objectives in the learning phase exists for exactly this, and re-running the backtest with that option on shows you how many of the triggers were learning noise all along.

A rule rewrites history: the feedback problem

This is the subtle part. The moment an action is applied, the following days' data changes shape. A paused campaign spends nothing and produces no data, so there is no second opportunity for the rule to pause it. The backtest, reading history as it stands, counts that campaign as paused five more times. This systematically inflates the trigger count.

With budget rules the bias runs the other way. Had "increase budget when ROAS beats target" been live, the extra budget would likely have pushed CPM up and pulled ROAS partway back down; the backtest cannot see that response, because the data it holds was produced at the lower budget. Backtests are therefore optimistic on scaling rules and overstated on pausing rules.

Read the number as a ceiling, not an estimate. "At worst, this rule could touch this many campaigns" is the most honest sentence a backtest can produce. And spend less time on how often it fired than on which campaigns appear in the list: if you recognise the names and they should not be there, the problem is the rule itself.

When is simulation more useful than a backtest?

A backtest asks "would this have happened"; a simulation asks "what happens at these values." In the "Simulate with your own numbers" view you type sample metrics by hand: ROAS 0.7, spend $1,400, frequency 4.2. Does the condition match, how does the guardrail clip the budget change, what value would the action have applied — all visible, nothing executed.

That distinction matters in three situations. First, a brand-new campaign has no history; the backtest comes back empty while the simulation still exercises the logic. Second, you want to probe the edges by hand: what happens exactly at the threshold, does the condition break when spend is zero. Third, in low-volume accounts 28 days is statistically thin to begin with; in LinkedIn Ads for B2B, where monthly conversions barely reach double digits, the pattern a backtest surfaces can easily be coincidence.

A workable order: describe the rule in a plain sentence, review the condition → action draft it produces, validate the logic in simulation, then measure the reach with a backtest. Run it the other way round and you will spend an afternoon debating the trigger count of a condition that was written wrong.

How do guardrails trim the trigger count?

No rule runs on its own. Minimum data window, budget-change limits, cooldown between actions, a daily budget floor and learning-phase protection all apply before a rule does any work. Running the backtest with those settings on drops the number substantially: the cooldown alone, which stops the engine from touching the same campaign twice in a row, eliminates most repeat triggers.

That makes it worth reading the backtest twice. The number without guardrails gives you the rule's raw reach; the number with them gives you its real behaviour. If the gap is large — 46 falling to 11, say — your rule is effectively being governed by the guardrails, and loosening the threshold slightly will make it far easier to reason about.

Guardrails have a sibling on the media-buying side: exclusion lists. Both protect budget by saying "do not do this." Negative keywords stop the budget burn at the point of entry; guardrails cap the automation on the way out. In an account with neither, building those two layers pays back more than any backtest will.

What trigger count is healthy?

There is no single correct number, but the range you land in points clearly to the next step. The table below assumes the 28-day window and a mid-sized account with 10–30 active campaigns; scale it to your own campaign count.

Backtest result	Likely cause	Next step
0 triggers	Threshold sits outside the distribution	Go back to the distribution, take the median
1–5 triggers	Narrow, selective rule	Move straight to approval mode
6–20 triggers	Reasonable reach	One week preview-only, then approval
20+ triggers, few campaigns	Oscillation on the same campaigns	Extend the cooldown between actions
Most campaigns on the list	Threshold too loose or wrong metric	Split the rule, narrow the scope

That last row is the most common one, and it usually means the rule is trying to do too much. When "pause low ROAS" and "cut spend with no leads" get compressed into a single sentence, one threshold is being asked to solve two different problems, and it solves neither cleanly.

What does preview-only mode catch that a backtest misses?

A backtest looks backward; preview-only mode looks forward. Every rule runs at one of three autonomy levels: preview only, with my approval, and automatic. At the first level the rule evaluates your live campaigns and reports how many were assessed, how many matched and what would have changed — without writing a single value to your account. The principle that preview never writes covers this mode end to end.

The two tools catch different classes of error. The backtest exposes flaws in the logic; preview-only surfaces problems in data flow and timing — sync lag, metrics that only settle at end of day, the realisation that the rule should run on selected days and hours rather than every hour. A week's worth of accumulated decisions in preview mode is more persuasive than the 28-day table, because every line in it maps to a campaign you can open right now.

Approval mode is the next rung: matches land in an approval queue, you see the reason and the exact change, and nothing touches the campaign until you say so. If you buy across several platforms, that queue also becomes a shared language, carrying the same logic that makes managing Meta, Google, TikTok and LinkedIn in one dashboard workable into your automation decisions.

What evidence do you collect after the rule is live?

Once a rule flips to automatic, the backtest's job ends and measurement begins. Every applied action gets its before and after compared over equal windows, and that comparison is observational: it makes no causal claim. It will not say "ROAS improved because of the rule"; it says "the 7 days before this action looked like this, and the 7 days after looked like that." Preserving the distinction matters, because a new asset going live in the same week or a seasonal swing can explain the whole delta on its own — which is why running creative tests in their own system also cleans up your automation measurement.

The second way to shrink risk is a staged rollout: run the rule on a small slice of campaigns first, compare the effect, then scale it to the rest. If it goes wrong, you can reverse a single decision or an entire run within 24 hours and the budget returns to its previous value. That rollback window puts a one-day price on everything the backtest could not show you.

What really decides the quality of the evidence is which revenue you measure. Real ROAS, computed from won opportunity revenue in the CRM rather than the platform's conversion value, prevents a loss verdict before cohorts have matured. Building that link means matching ad spend to CRM revenue; without it, the rule ends up rewarding cheap conversions that never become customers.

The same pipeline runs in the other direction too. Once you know which segments actually close, pushing those CRM lists back to the ad platforms improves the audience your rules operate on, and a threshold set against a good audience outlives one set against a poor one.

When should you retire a rule?

Rules accumulate. Six months in, an account holds rules nobody remembers writing, some of them cancelling each other out — one raising a budget while another cuts it. Conflicting rule pairs in the same scope get flagged and resolved on one screen, and you set the order rules run in, so precedence is never left to chance. Silent rules are their own overhead: if a rule produces no result run after run, a notification goes to the marketing owners, so broken automation does not sit there quietly.

For retirement decisions, a simple cadence works: re-backtest every active rule each quarter. The account's distribution will have shifted, and the old threshold is now either never firing or catching half the account. A rule can be correct the day it is written and wrong three months later; only repeating the measurement keeps it honest. Whenever the account structure changes — campaigns consolidated, a new account opened, or an ad account suspended and rebuilt — the scope of your old rules quietly drifts and has to be measured again.

Sitting above all of it is the master switch: automation ships off by default, nothing changes in your ads until you deliberately turn it on, and switching it off never deletes your rules. If you want to walk a rule through backtest, simulation and preview-only in that order, you can open a Rocketly account and test your first rule without touching a single campaign.

FAQ

Can a backtest damage the campaigns in my account?

No. A backtest only reads historical data and reports what the rule would have done; no value is written to the connected account. The same holds for preview-only mode and for simulation.

The backtest returned 0 triggers. Should I still turn the rule on?

You can, but check the threshold first. A zero result usually means the threshold sits outside the distribution, and only occasionally that the rule genuinely covers a rare event. In the second case turning it on is harmless, since it will fire rarely by design.

Can I backtest over a window longer than 28 days?

The backtest window is the last 28 days. If you need a longer view, the practical route is to run the rule in preview-only mode for a few weeks and read the decisions it accumulates; that gives you a window that widens going forward.

Why does the real result differ from the backtest?

Because applying a rule changes the data that follows. A paused campaign produces no data; an increased budget moves costs. A backtest cannot model that feedback, which is why the trigger count should be read as a ceiling rather than a prediction.