The uncomfortable arithmetic of email A/B testing is that the sample sizes needed to detect realistic differences are much larger than most lists. A test that reports a winner on 1,000 recipients per arm has almost certainly reported noise.
This is worth understanding not to discourage testing but to redirect it, because there is plenty worth doing at small scale — just not detecting a 5% relative improvement in click rate.
Why sample size depends on your baseline
Detecting a difference gets harder as the events get rarer. Open rates run around 30%, so opens are relatively common events and differences show up sooner. Click rates run around 2%, so clicks are rare and the same relative difference needs far more people. Conversion rates run around 1%, and are harder still.
Which produces an awkward result: the metric that is easiest to test on is the one worth trusting least, because open rate is contaminated by privacy-protection proxy loading.
Rough numbers to orient by
For a 95% confidence level and 80% power — the conventional settings — here is the approximate scale of what you need per arm:
| Baseline | Detect a 50% relative lift | Detect a 20% lift | Detect a 10% lift |
|---|---|---|---|
| 30% open rate | ~150 | ~800 | ~3,000 |
| 2% click rate | ~2,500 | ~15,000 | ~60,000 |
| 1% conversion | ~5,000 | ~30,000 | ~120,000 |
Treat these as orientation rather than precision; a proper calculator with your actual baseline will give a better number. The pattern is what matters. Detecting a small improvement in click rate requires more recipients than most senders have.
Note also that these are per arm. A test on a 10,000-person list gives you 5,000 each way, which lands you in "can detect a large difference in click rate, cannot detect anything subtle" territory.
What this means for your list
Under 5,000 subscribers. Formal A/B testing will rarely produce a statistically meaningful answer. Test large structural differences — plain versus designed, one call to action versus three, a completely different angle — and judge on patterns across many sends rather than on single results.
5,000 to 50,000. You can detect large differences in open rate and substantial ones in click rate. Test framing, structure and offer, not wording.
Over 50,000. Most of the standard testing programme becomes viable, including conversion-based tests on your larger campaigns.
Automated flows, any list size. This is the exception worth knowing. A welcome email or abandoned cart flow runs continuously, so a 50/50 split accumulates sample over weeks. A flow sending 200 a week reaches thousands per arm in a few months, which makes flows the best place to test on a small list — and they are also the highest-value emails you send.
The three ways tests lie
Stopping when you are ahead. At any moment during a test, one arm is ahead by chance. Watching and stopping at that moment guarantees a "winner" that is not one. Decide the sample and the duration in advance, then look once.
Calling it too early. Opens and clicks accumulate over days. A test evaluated after two hours has measured time-zone distribution.
Running many tests and reporting the wins. Test twenty things at 95% confidence and roughly one will look significant by chance. If you test a lot, expect false positives and require repetition before you believe anything.
What to do instead on a small list
Accumulate. Log every test — variable, arms, sample, result — and look for what repeats across ten campaigns. A pattern that holds seven times out of ten is more actionable than one significant result.
Test in flows, not broadcasts. For the reason above.
Test big differences. Two versions of the same idea will never separate. Two different ideas might.
Spend the effort elsewhere. For a list under 20,000, segmentation, inbox placement and building the automated flows will each move your numbers more than any subject line test could, because they change the system rather than optimising one email inside it.
The general approach to running the tests themselves is in email A/B testing.
Frequently asked questions
How many subscribers do I need to A/B test emails?
It depends on the metric and the size of the effect. Detecting a large difference in open rate can work with a few hundred per arm; detecting a modest difference in click rate needs tens of thousands. Below about 5,000 subscribers, only large structural differences are detectable.
Can I A/B test with a small email list?
Yes, in automated flows, which accumulate sample over weeks rather than in a single send. For broadcasts on a small list, test large differences and judge on patterns across many campaigns rather than single results.
What confidence level should I use?
95% confidence with 80% power is the convention and a reasonable default. Lower thresholds increase false positives, which is a particular risk in email because most senders run many small tests and remember the ones that won.
Why did my A/B test winner not repeat?
Almost always because the original result was noise — an underpowered test found a difference that was not there. It can also be genuine variation between campaigns, since a subject line that suits one email may not suit another.
