What a Klaviyo A/B test has to show before it changes your next send

Before a subject-line winner goes into the brand’s playbook, check what Klaviyo tested to award it. Its help centre states the rule: a campaign test is statistically significant when at least 50 people have received each variation and the leading variation has a win probability of 90% or more, judged on the metric chosen at setup, open rate, click rate or placed order rate. The results view shows the winning variation, the status, the win probability, the lift over the runner-up and each variation’s rate.

Fifty recipients per variation is the minimum Klaviyo needs before it labels a result significant. With 50 recipients, one open is two percentage points of open rate: variation A at 20 opens and B at 23 reads as 40% against 46%.

A win probability is Klaviyo’s estimate of how likely it is that the leading variation is really the better one. Taken at face value, 90% means a one-in-ten chance that it is not. A brand that runs a test on every send and adopts every winner at the 90% bar should expect up to one adopted learning in ten to be wrong.

The winner is also chosen early. In Klaviyo’s setup guide the test goes to a portion of the audience, 20% per variation in the example. The test window, six hours in the example, is adjustable. When it closes, the rest of the audience receives the winner. So the pick is made on the opens, clicks or orders that had arrived within the window. For placed order rate that can be a handful of orders per variation. The full-send figures keep accumulating for days afterwards.

Klaviyo’s benchmarks documentation says Apple Mail Privacy Protection opens inflate open rates. Its A/B testing pages say Klaviyo anticipates its tools “should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance”. How the tools account for it is not documented. For accounts that test regularly and see more than about 45% of opens on Apple Mail, Klaviyo suggests a custom report with the privacy-open property.

The check

Applied to every result before it changes the next send.

1. Recipients per variation. Klaviyo’s floor is 50. Set your own from the metric: for open or click rate, enough recipients that one event moves the rate by no more than 0.1 points, which is 1,000 per variation; for placed order rate, enough that each variation has at least 30 orders inside the window. Write the floors down and do not act below them.
2. Win probability. Klaviyo’s bar is 90%. Set yours at 95% for a result you will carry to the next send, which halves the face-value rate of wrong calls from one in ten to one in twenty, and is in line with Klaviyo’s note that privacy opens may call for a higher threshold.
3. Winning metric matches the question. Klaviyo recommends open rate for subject line, preview text and sender tests, and click or placed order rate for content. If a placed-order window closed with single-digit orders per variation, record it as inconclusive whatever the label says.
4. Read the full-send figures after the attribution window closes. Open the campaign’s A/B results and record each variation’s rate on everyone who received it. The losing variation only ever reached its test share, so the two audiences differ in size and send time, and the comparison is not the same test again. The decision rule is simple: if the winner no longer leads on the full-send figures, the test is inconclusive.
5. Record it, and adopt only on a repeat. A test record has two rows of figures, the window and the full send, and a decision. A winner goes into the playbook after it wins again on a later send of the same metric to the same list or segment, at your bar, within the next three sends.

One test record

An invented example, Northline Outfitters, a subject-line test on open rate.

Sendfire’s Tests are built for the last two lines: one question, the campaigns that joined it, and a conclusion the marketer writes. The Tests page shows one.