Experimentation & A/B Testing

Experimentation is the engine of data-driven decision making. Without experiments, every decision is a guess — an educated guess perhaps, but a guess nonetheless. A/B testing (and its more sophisticated relatives) lets you causally determine what works, what does not, and why. It is the only method that definitively answers "did this change cause that outcome?"

This topic takes you from the theory of hypothesis testing through the mechanics of designing, running, and interpreting A/B tests. You will learn sample size calculation, statistical significance, p-values, confidence intervals, and the common pitfalls that ruin experiments (peeking, multiple testing, novelty effects). You will also learn when multi-armed bandits are a better choice than traditional A/B testing. By the end, you will be able to design and evaluate experiments that produce trustworthy, actionable results.

Start with DDDM Fundamentals and KPI Design & Metric Frameworks for the basics of data-driven decisions and metrics. Some comfort with basic statistics (means, percentages) helps — you do not need a statistics course yet.

Every experiment starts with two competing hypotheses:

• Null hypothesis (H₀): The change has no effect. "Redesigning the button color does not change click rate."
• Alternative hypothesis (H₁): The change has an effect. "Redesigning the button color changes click rate."

The experiment does not prove H₁ is true — it asks: if H₀ were true, how likely is the data we observed? If the probability is very low (typically below 5%), we reject H₀ and accept H₁. This is not certainty — it is probabilistic reasoning.

Key terms:

• Control group: the unchanged version (status quo)
• Treatment/variant group: the version with the change
• Primary metric: the one number that determines success or failure
• Guardrail metrics: metrics you monitor to ensure the change does not cause unintended harm

A well-designed A/B test follows these steps:

1. Define the hypothesis — "Changing X will improve Y by Z% because [reasoning]."
2. Choose the primary metric — one number that defines success (e.g., conversion rate, click-through rate, revenue per user).
3. Select guardrail metrics — things that must not get worse (e.g., page load time, error rate, bounce rate).
4. Calculate sample size — use a power calculator (e.g., Evan Miller's calculator) with your baseline rate, minimum detectable effect (MDE), statistical significance threshold (usually 95%), and power (usually 80%).
5. Determine test duration — run for at least one full business cycle (usually 1–2 weeks) to account for day-of-week effects.
6. Randomize correctly — assign users to control/treatment randomly at the user level (not session level, to avoid contamination).
7. Run the test — do not peek at results and stop early unless you have a pre-specified stopping rule.
8. Analyze results — check statistical significance, effect size, and segment-level results.

Sample size is the most critical design decision. Too small and you waste time running a test that cannot detect real effects (underpowered). Too large and you waste resources detecting trivial effects (overpowered).

The four inputs to sample size calculation:

Rule of thumb: if your baseline conversion is 5% and you want to detect a 10% relative improvement (to 5.5%), you need roughly 30,000–40,000 users per variant. For most web experiments, plan for at least 1–2 weeks of traffic.

P-value: The probability of observing your results (or more extreme) if the null hypothesis is true. A p-value of 0.03 means there is a 3% chance the result is due to random noise. Convention: p < 0.05 is "statistically significant."

Confidence interval (CI): A range of values within which the true effect likely falls. A 95% CI of [0.2%, 1.8%] means you are 95% confident the true improvement is between 0.2% and 1.8%. If the CI includes zero, the result is not significant.

Effect size: How big the actual change is. A statistically significant p-value of 0.01 on a 0.01% improvement is meaningless in practice. Always report: p-value + effect size + confidence interval.

Even well-designed tests fail due to these frequent mistakes:

Traditional A/B testing splits traffic 50/50 for the entire test duration. Multi-armed bandits (MAB) dynamically shift traffic toward the better-performing variant as the test runs.

When to use MAB:

• The cost of showing the losing variant is high (lost revenue during the test)
• You need results quickly and cannot wait for full statistical power
• You are testing many variants simultaneously
• The environment is non-stationary (user preferences change over time)

When to stick with traditional A/B:

• You need a definitive, statistically rigorous answer for a high-stakes decision
• You need to understand why something works (MAB optimizes faster but gives less explanatory insight)
• You are making a one-time decision, not ongoing optimization

Algorithms: ε-greedy (explore randomly with probability ε), Thompson Sampling (Bayesian approach — probabilistically favors better variants), Upper Confidence Bound (UCB — balances exploration and exploitation).

Tools and statistics are not enough — a true experimentation culture requires organizational commitment:

1. Pre-registration: Every experiment is documented before it runs: hypothesis, metrics, sample size, and decision criteria. This prevents p-hacking and post-hoc rationalization.
2. Decision log: Record what was decided, why, and what the data showed. Build institutional memory.
3. Experiment reviews: Regularly review failed experiments — they often teach more than successful ones.
4. Statistical literacy: Train product managers and designers to interpret results, not just hand off to data teams.
5. Velocity: Aim for one experiment per team per sprint. Speed of learning matters more than individual test outcomes.

Practice designing and interpreting experiments with these exercises.

1. Design an A/B test for a hypothetical feature: write the hypothesis, choose primary and guardrail metrics, and calculate the required sample size using an online calculator.
2. Given a set of experiment results (p-value, confidence interval, sample size), determine whether to ship, kill, or iterate.
3. Identify the flaw in each of these experiments: (a) a test that ran for 2 days, (b) a test with 50 variants and one "significant" result, (c) a test where the winning variant showed 3% improvement but the CI included 0.
4. Compare A/B testing and multi-armed bandits for a news website testing headline formats. Which approach is better and why?
5. Write a pre-registration document for an experiment you would run at your workplace or a hypothetical company.

Google runs over 10,000 experiments per year across Search, Ads, Maps, and every product. One famous experiment: Google tested 41 shades of blue for a link color. The winning shade generated an estimated $200 million in additional annual revenue. While the specific number is debated, the principle is sound — small, well-tested changes compound into massive impact at scale. Netflix tests everything from thumbnail images to autoplay behavior to subtitle fonts. Their experimentation platform runs hundreds of concurrent tests, and they pre-register every experiment to maintain rigor. The lesson: at scale, experimentation becomes the primary driver of product improvement, and the compound effect of many small wins is enormous.

Business problem: A SaaS company offering a free trial wants to test whether reducing the trial from 14 days to 7 days increases paid conversion. The hypothesis is that urgency will drive faster decisions and reduce "forgot about it" drop-offs.

Current metrics: Trial-to-paid conversion rate = 12%, average time to convert = 8 days, 15,000 new trial sign-ups per month.