Vector 1

Demo · SEO split testing

Did the SEO change work, or is it noise?

You can't A/B test one page in search, but you can test a template change across many similar pages: change half, leave half alone, and compare. Organic traffic is noisy, so most tests are too small or too short to see a real lift. This simulator runs the test on synthetic traffic where the true effect is known, reads it out the way a careful analyst would, and shows how many pages and weeks you need.

The test

Volatility is how much each page's traffic wanders on its own from the 4-week baseline to the test period (rankings shift, competitors move). It is the main source of noise, more than daily randomness.

The readout

How long and how big does the test need to be?

The chance this test design detects the true lift (statistical power), from 200 simulated repeats of each test length. Aim for 80% or more before you start; a test that can't detect the lift you care about will mostly come back "inconclusive" even when the change works.

How the readout works

Each page's clicks in the test period are compared with its own 4-week baseline, and the average change in the changed group is compared with the average change in the untouched group (a difference-in-differences on log clicks). The untouched group absorbs seasonality, algorithm updates and anything else that hits every page at once. The 95% interval comes from the spread of page-level changes, so more pages and less volatile pages give tighter intervals.

All traffic is simulated: each page has its own baseline (log-normal), its own drift between periods (the volatility setting), shared weekly seasonality and a sitewide shock in the test period, and Poisson daily clicks. Real sites add complications this ignores, such as pages that cannibalize each other and uneven indexing of the change.