🧪 Split Test Lab
🧪 Split Test Lab
The two numbers every experiment platform charges you to see: is this difference real, and how long must the test run? Frequentist z-test with confidence intervals, a proper sample-size and duration planner, and a Bayesian win-probability estimate — all computed here, with nothing sent to a server.
The maths, and where these tests lie to you
The p-value is a two-proportion z-test on the pooled rate, with the normal CDF from the Abramowitz &
Stegun 7.1.26 error-function approximation (accurate to about 1.5×10-7). Sample size uses
the standard two-sided formula
n = (z1−α/2√(2p̄(1−p̄)) + zpower√(p₁(1−p₁)+p₂(1−p₂)))² ÷ (p₂−p₁)².
Three things will still fool you: peeking at the results daily and stopping when it looks good inflates the
false-positive rate well past 5%; a real win on a metric nobody acts on is still worthless; and if you ran
ten experiments, one of them is "significant" by luck alone. The Bayesian tab answers a friendlier question
— "how likely is B better?" — but it does not fix a biased split.