Statistical Significance in Paid Media: Stop Reacting to Noise
Most marketers call winners too early and read random fluctuation as a real effect. At small sample sizes, that's not optimization — it's an expensive coin flip.
Here's an uncomfortable truth about paid media: most of the "wins" teams celebrate are noise. A variant pulls ahead, someone declares victory, budget shifts — and the "lift" evaporates the next week because it was never real. At the conversion volumes most campaigns actually run, distinguishing signal from randomness is hard, and the industry is mostly bad at it.
The peeking problem
The most common error is stopping a test the moment it looks like a winner. Statisticians call it "peeking," and it quietly wrecks your error rate. Checking results before the planned sample size and stopping early inflates the false-positive rate from the intended 5% to 20–30% or higher — the more often you peek, the worse it gets. You think you're taking a 1-in-20 risk of a false result; you're actually taking a 1-in-4 risk.
Most tests never actually reach significance
The scale of the problem is documented. In an analysis of 28,304 experiments, only 20% reached a 95% statistical significance level. The convention in conversion work — 95% confidence and 80% power, with sample size fixed up front — exists precisely because results below it are unreliable. The right stop signal isn't "we hit 95% today"; it's "we reached the sample size we planned for."
Significance isn't the same as profit
Even a genuinely significant result isn't automatically worth acting on. Significance tells you the effect is real. It doesn't tell you the effect is large enough to matter, or that acting on it will outperform doing nothing. The full question is: is this real, is it meaningful, and does the change it requires make sense given what else is happening in the account? That's a judgment call significance alone can't make.