Course Preview

A/B Testing and Experimentation

A/B testing is a randomized comparison of two product experiences. It assigns comparable users to a control or treatment, measures a predefined outcome, and uses the difference to estimate whether the change caused an effect.

itArtificial intelligence and machine learning | OpenSkills.info

Course pathWalk it in order

Look it upDip in anytime

Go furtherLeaves this page

A/B Testing and Experimentation

A/B testing is an online controlled experiment. Eligible units are randomly assigned to a control, A, or a treatment, B. Both groups run at the same time. The difference in their outcomes estimates the causal effect of the treatment under the experiment's assumptions.

This design separates a treatment effect from changes that happen at the same time, such as seasonality, marketing campaigns, and shifts in the user population. A before-and-after comparison cannot provide the same protection because time and treatment change together.

The experiment system

An online experiment connects five parts:

Hypothesis and decision rule. A falsifiable statement names the treatment, target population, expected direction, primary metric, minimum useful effect, guardrails, and action for each result.
Assignment. A stable randomization key maps each eligible unit to a variant. The randomization unit might be a user, account, device, session, or cluster.
Delivery and exposure. The product serves the assigned variant. An exposure event records that the unit encountered the treatment, not merely that it was eligible.
Measurement. Outcome events become a primary metric, guardrail metrics, diagnostic metrics, and data-quality checks.
Analysis and decision. The analysis estimates effect size and uncertainty, checks validity, and applies the rule chosen before results were visible.

These parts form one causal measurement system. Correct statistics cannot repair biased assignment, missing exposures, or a metric that does not represent the decision.

From population to estimate

The target population is the group to which the decision should apply. Eligibility rules select an experiment population from it. Random assignment then creates groups that are comparable in expectation.

Continue the course

This section is part of the paid course.

See pricing to subscribe, or log in if you already have access.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Data Scientistsupporting

Sources