Overview

An A/B test compares two versions of a page, email or ad to see whether one converts better. This checks whether the difference you saw is likely to be real or just chance.

Enter conversions and visitors for each variant, and your significance level. 5% is the usual choice.

Worked examples

InputsResult
Variant a Conversions420Variant a Visitors10,000Variant B Conversions495Variant B Visitors10,000Alpha (%)5Variant A CR: 4.2%Variant B CR: 4.95%Lift: 17.86%z: 2.54p-value: 0.01Significant result at alpha=5%
Show 2 more examples
Variant a Conversions100Variant a Visitors2,000Variant B Conversions120Variant B Visitors2,000Alpha (%)5Variant A CR: 5%Variant B CR: 6%Lift: 20%z: 1.39p-value: 0.17Not significant at alpha=5%
Variant a Conversions50Variant a Visitors1,000Variant B Conversions52Variant B Visitors1,000Alpha (%)5Variant A CR: 5%Variant B CR: 5.2%Lift: 4%z: 0.2p-value: 0.84Not significant at alpha=5%

Variant A CR by variant a visitors

With the calculator's starting values, the variant a CR is 4.2%. The table shows how it changes with the variant a visitors.

Variant a VisitorsVariant A CR
6,0007%
8,0005.25%
10,000starting value4.2%
12,0003.5%
14,0003%

Other inputs: Variant a Conversions: 420 · Variant B Conversions: 495 · Variant B Visitors: 10,000 · Alpha: 5%. Change them in the calculator above for your own numbers.

How it works

It runs a two-proportion z-test. The two conversion rates are compared using a pooled rate to estimate the standard error, giving a z-score and a two-sided p-value. If the p-value is below alpha, the result is called significant. Lift is B's change relative to A.

The formula

z = (p_B − p_A) ÷ √(p(1 − p)(1/n_A + 1/n_B))
p_A, p_BConversion rate of each variant
pPooled rate across both
n_A, n_BVisitors in each variant

Practical tips

  • 420 of 10,000 against 495 of 10,000 is a 17.86% lift with p ≈ 0.01, which is significant at 5%.
  • Decide the sample size before starting, and don't stop the test the moment it turns significant. Peeking raises false positives.
  • Run tests for whole weeks, so weekday and weekend visitors are both included.

Frequently asked questions

What does p = 0.05 mean?

If there were truly no difference, a gap this large would show up about 5% of the time by chance.

How many visitors do I need?

It depends on your base rate and the lift you want to detect. Small lifts on low conversion rates need tens of thousands of visitors per variant.

Is statistical significance enough?

No. Also check whether the lift is big enough to matter for the business.