Don't Panic: Experimental Design

Experimental design is the arrangement of a study so that a difference in results can be tied to a cause, rather than to luck, timing, or some uninvited variable wearing a false moustache. It exists because observational data is eager to tell a story and almost never comes with a receipt. Users of a feature may retain better, but that does not show the feature caused retention. Loyal users may merely have arrived first.

The central device is randomization, which assigns experimental units to conditions by chance. An experimental unit is the thing that receives one treatment, such as a user, a batch, or a server. Random assignment distributes known and unknown confounders between control and treatment on average. It does not make a bad metric good, but it prevents pre-existing group differences from pretending to be an effect.

Next come replication and blocking. Replication gives each condition enough independent units to show variability rather than a single theatrical coincidence. Blocking groups units with a known source of variation, then randomizes within each group. If mobile and desktop users differ, blocking on device type reduces that noise without letting device type choose the treatment. The surprise is that the design work happens before the data becomes exciting. A p-value is not a repair kit for a study that assigned the wrong unit or chose its metric after seeing the result.

Power analysis sets the size of the experiment before it runs. It combines the minimum effect worth detecting, alpha, power, and response variance to estimate how many units each group needs. Small effects and noisy responses need more observations. That can feel inconvenient because mathematics has declined to negotiate with the traffic forecast. It is still better than running an experiment that cannot detect the change the team cares about.

When results arrive, compare control and treatment with an appropriate test, then report an effect size and confidence interval alongside the p-value. A statistically significant effect can be too small to justify the cost of acting. Also check for sample ratio mismatch, attrition, spillover, novelty effects, multiple comparisons, and peeking. Repeatedly checking a dashboard and stopping at the first attractive result changes the false-positive risk, which is a poor foundation for confidence.

Read the Intro for the vocabulary and common designs. Use Slides to see how randomization, power, and validity threats connect. Keep the Cheatsheet nearby when selecting a design or interpreting an error type. The Practice Reference turns the process into a repeatable plan, and the Quiz checks whether the causal chain still holds together when the labels are removed.