Use when
- A live product has enough traffic and instrumentation to compare alternatives.
- The team needs evidence about which option performs better on a measurable outcome.
Do not use when
- Traffic is too low or instrumentation is not trustworthy.
- The team still does not understand the user problem or task well enough.
Best for
- Optimization with live traffic
- Comparing alternatives
- Rollout validation
How to design it
- Define one decision and one success metric clearly.
- Control audience assignment and duration intentionally.
- Avoid running experiments when the underlying experience is still fundamentally unclear.
Experiment TemplateUse this when you need a practical starting template.
View template
- What is the hypothesis?
- Which audience sees each variant?
- What metric decides the winner and what would count as inconclusive?
Output
- Experiment plan
- Variant performance summary
- Decision and follow-up action