An email A/B test splits your audience, sends two versions, and compares results. It is the most-recommended practice in email marketing and among the most commonly done wrong, because the sample sizes involved are usually far too small for the differences being claimed.
If you take one thing from this: a 2-point difference on a 1,000-person split is noise, and acting on it is worse than not testing at all, because you have now built a belief on nothing.
Test one thing, and make it a big thing
Change one variable per test. Two changes give you a result you cannot attribute.
More importantly, test things that differ substantially. Two phrasings of the same subject line will produce a difference too small to detect at any list size you have. Two genuinely different angles — a benefit versus a question, a number versus a promise — might produce a difference large enough to see.
Small tests need enormous samples. Big differences need modest ones. So test big differences.
Worth testing: subject line angle, sending from a person versus a brand, offer framing, a single call to action versus several, plain versus designed, short versus long.
Rarely worth testing: button colour, one word in a subject line, image placement, send time as a one-off. The effects are real and far too small to measure at typical list sizes.
Pick the right success metric
Open rate is the default and the worst choice available.
It is inflated by Apple Mail Privacy Protection proxy loading, which affects an unpredictable share of your list and does not distribute evenly between your two arms. A subject line test measured on opens is measuring a metric partly composed of automated fetches.
Use clicks where the change could plausibly affect clicks, and conversions or revenue per recipient where you have them. Those are harder to move, which means you need larger samples — but the answer is real.
If you must judge a subject line test, judge it on clicks. A subject line's actual job is getting someone to the content, and a line that wins opens while losing clicks attracted the wrong attention.
Sample size, briefly
The arithmetic depends on your baseline rate and the size of difference you want to detect. As orientation: detecting a change from a 2% to a 2.4% click rate with any confidence needs tens of thousands of recipients per arm. Detecting a change from 2% to 3% needs a few thousand.
The practical implication for a list of 5,000 is that you can detect large differences and nothing else. That is not a reason to stop testing — it is a reason to test large differences and to accumulate results across many campaigns rather than declaring a winner from one. Email test sample size has the working.
The three mistakes that produce false wins
Peeking and stopping early. Watching results and stopping when one arm is ahead guarantees you find winners that are not real, because at any moment one arm is ahead by chance. Decide the sample and the duration before you start, and look when it is over.
Not waiting long enough. Opens and clicks accumulate for days. A test called after two hours measures which arm was sent to more early risers.
Running one test and generalising. A result from one campaign tells you about that campaign. Patterns emerge across ten. Keep a log of every test — variable, arms, sample, result — and look for what repeats.
The winner-takes-all send
Many platforms offer to send both versions to a small slice, then send the winner to everyone. This is convenient and it inherits every problem above: the deciding slice is usually far too small, and the decision is usually made on opens after a short window.
If you use it, make the test slice as large as you can tolerate, choose clicks as the deciding metric if the option exists, and give it hours rather than minutes.
What to do instead of testing
For most senders under about 20,000 subscribers, A/B testing is not where the improvement is. The larger levers, in order:
- Inbox placement — no subject line beats not arriving.
- Segmentation — sending the right email to the right people beats optimising the wrong one.
- Automated flows — welcome, abandoned cart, post-purchase — which outperform any broadcast optimisation.
- Frequency, decided deliberately rather than by accident.
Test once those are in place. Then the tests are measuring a system that works, rather than searching for a subject line to rescue one that does not.
Frequently asked questions
How large does an email A/B test need to be?
It depends on your baseline rate and the size of the effect. Detecting a 20% relative change in click rate typically needs a few thousand recipients per arm; detecting a 5% change needs tens of thousands. If your list is small, only test large differences.
Should I test subject lines on open rate?
Preferably not. Open rate is contaminated by privacy-protection proxy loading in proportions that vary between test arms. Judge on clicks where possible — a subject line that wins opens but loses clicks attracted the wrong readers.
How long should I run an email A/B test?
At least 24 hours, and ideally 48, because opens and clicks accumulate over days. Deciding after two hours measures time-zone distribution rather than message quality. Set the duration before you start and do not stop early.
What should I test first?
Whether the email arrives at all, then who receives it. Once inbox placement and segmentation are sound, test big differences in framing and structure rather than small variations in wording.
