A/B test a campaign
Send competing versions to a slice of your list, let the better one win on opens or clicks, then send the winner to everyone else.
On this page›
An A/B test sends two or more versions of a campaign to a small share of your list, measures which one performs better, and then sends the winner to everyone who has not received anything yet. You get the benefit of the better version on the bulk of your audience instead of finding out afterward.
How the test runs#
A test has three phases, and the product moves through them for you.
First the test phase. You choose what percentage of the list to test on. That slice is divided between your variants, and each one receives its version.
Then the wait. You set how many hours to wait before judging. Nothing is sent during this window. It exists because opens and clicks arrive over hours, not minutes, and a winner picked after ten minutes is mostly measuring who happened to be at their desk.
Then the winner phase. When the wait is over, the winning variant is declared and sent to the remaining subscribers.
One test per campaign
The test belongs to the campaign. You build the variants inside the campaign you are about to send, rather than creating separate campaigns and comparing them by hand afterward.
Set up a test#
Build the campaign as usual
Choose the list or segment, set the sender details, and build your first version. This becomes the control, the version you are trying to beat.
Add your variants
Each variant is a complete version of the email with its own name. Because a variant carries its own content, you can change the subject line, the body, or both. Give each one a name you will still understand in a week, like "short subject" rather than "version 2".
Set the split
Decide what share of the list the test runs on, and how the test slice divides between variants. An even split between variants is the usual choice, because an uneven one makes the comparison harder to read.
Choose how the winner is decided
Pick open rate or click rate. Choose the one that matches what you are testing, which is covered below.
Set the wait
Choose how many hours to wait before the winner is declared. See the guidance on timing below.
Send
The test slice goes out immediately. When the wait elapses, the winner is declared and sent to the rest of the list. You can also declare the winner yourself before the wait is over if the result is already obvious.
Choosing the winning metric#
Open rate measures whether people opened at all, so it is the right metric when the thing you changed is what they see before opening: the subject line or the preheader.
Click rate measures whether people acted, so it is the right metric when you changed something inside the email: the offer, the layout, the call to action.
Open rate has a real weakness worth knowing. Some mail clients preload images and record an open the recipient never performed, and others block tracking entirely. Open rate is directionally useful but noisier than click rate. When you are testing something inside the email, prefer clicks.
Test one thing at a time#
If a variant changes the subject line, the header image, and the button color, and it wins, you have learned that this particular bundle beat the other bundle. You have not learned which change did it, so you cannot apply the lesson to the next campaign.
Change one thing. The point of a test is a transferable lesson, not a single better send.
Size the test honestly#
This is where most A/B tests go wrong. A difference between two variants only means something if it is bigger than normal random variation, and on small numbers that variation is large.
As a rough guide, a test slice needs to be in the low thousands of recipients per variant before a few percentage points of difference means anything. If your whole list is 2,000 people and you test on 20 percent, each variant reaches 200. A result of 22 percent against 19 percent on those numbers is noise, and acting on it teaches you the wrong lesson.
If your list is small
Below roughly 5,000 subscribers, most single-campaign tests cannot separate a real effect from chance. You will learn more by testing the same idea across several campaigns and looking for a consistent direction than by declaring a winner from one send.
If your list is large enough, a bigger test slice gives a more trustworthy result but leaves fewer people to receive the winner. Testing on 20 to 30 percent is a common balance.
How long to wait#
Long enough for the result to settle, short enough that the winner still arrives at a sensible time for the rest of your list.
Most opens land within the first few hours, but the tail runs much longer, and a morning send behaves differently from an evening one. Four hours is a reasonable floor. Overnight is often better for a campaign that goes out in the morning, because it captures people across time zones and working patterns rather than only the ones who read email immediately.
If you set the wait too short, you will systematically favor whichever variant appeals to fast readers, which is rarely the thing you meant to measure.
Reading the result#
Before you accept a winner, ask two questions.
Is the gap large enough to be real, given how many people were in each variant? A one point difference on a few hundred recipients is not a finding.
Does the result make sense as an explanation? If the winning subject line has no plausible reason to be better, treat it as unproven and test the idea again on a future campaign rather than rewriting your style guide around it.
A test that produces no clear winner is not a failed test. It tells you that the thing you changed does not matter much to this audience, which is worth knowing before you spend more time on it.
What to test first#
Start with the changes that move the most and are easiest to interpret.
| Test | Metric | Why it is worth doing |
|---|---|---|
| Short subject against long | Open rate | Usually the largest single effect, and easy to act on |
| Question against statement | Open rate | Tells you how this audience prefers to be addressed |
| One call to action against several | Click rate | Often improves clicks by removing choices |
| Plain text style against designed | Click rate | Frequently surprising, and affects deliverability too |
| Offer framing, discount against benefit | Click rate | Directly commercial, and the lesson transfers |
Last updated September 10, 2026