Course Preview

A/B Testing and Experimentation

A/B testing is a randomized comparison of two product experiences. It assigns comparable users to a control or treatment, measures a predefined outcome, and uses the difference to estimate whether the change caused an effect.

itArtificial intelligence and machine learning | OpenSkills.info

Course pathWalk it in order

Look it upDip in anytime

Go furtherLeaves this page

Don't Panic — A/B Testing and Experimentation

Ship the change on Tuesday, read the numbers on Friday, declare victory. The flaw is that Tuesday also brought a campaign, a payday, a heatwave and a competitor's outage. Time and the change moved together, so nothing could pull them apart.

The fix is structural rather than clever. Run both versions at once, on people picked at random, and let the same week hit both. The control is the existing experience and the treatment is the changed one; the gap between them estimates what the change caused rather than what the week did.

Everything after that is bookkeeping, which is precisely where experiments die.

Eligible, assigned and exposed are three different things. Somebody can qualify for the test, get sorted into a variant, and still never encounter what changed — and only the last of those says anything about the change itself.

The randomization unit is whatever gets independently sorted into a variant: one person, one company account, one session, or a whole cluster of connected users. It has to be large enough to contain the ways those units affect each other, and the larger it gets the fewer independent observations remain.

Every metric has exactly one job. The primary one decides. Guardrail metrics cap the harm the change may do, such as errors or refunds. Diagnostics explain where behaviour shifted, and data-quality metrics say whether to believe any of the rest.

The arithmetic is less forgiving than it looks. The number of people needed grows with the square of the effect being looked for, so halving that effect quadruples the traffic. That sum is a power calculation, it belongs before anything gets built, and it should be expected to rule ideas out.

Then the part that reframes the whole enterprise. Roughly two-thirds of carefully argued ideas that teams actually built failed to move the metric they were designed to move. That is the point rather than a scandal: what an experiment buys is not confirmation but the right to be wrong cheaply.

Which means a team whose experiments mostly win is not unusually gifted. It has a measurement fault, a habit of stopping the moment results look good, or it is testing changes nobody doubted. A healthy programme looks faintly disappointing on a dashboard.

Statistics cannot repair bad input, so sample ratio mismatch — the two groups ending up further apart in size than chance explains — gets diagnosed before anyone reads an outcome at all.

And "not significant" does not mean "the same". A result is one of four things: ship, do not ship, inconclusive, or invalid. The last two are honest answers rather than failures to produce one.

If you read only one more tab, make it Field Notes — it has what all this costs once real traffic is involved. The Intro works through the causal machinery slowly, and the Cheatsheet holds the design card.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources