A publisher tested 12 headlines. Did any beat the first?
Updated November 2026
Five of 11 headlines clicked less than the first one by enough to look different. After correcting for running 11 tests, none cleared the bar.
Upworthy, a publisher, showed 12 headlines for one story to readers and counted who clicked. We compared each of the 11 later headlines with the first one written. At the usual 0.05 bar, 5 of the 11 looked different. By chance alone, 11 tests at that bar would throw up about 0.55 false alarms: 11 tests, each with a 5% chance of a false alarm, is 11 times 0.05. A p-value is a number from 0 to 1 that says how surprising a result would be if the headlines did equally well. In exact words, it is the chance of seeing a gap at least this big if the headlines really performed the same. A small one, like 0.01, means very surprising. Headline 4 had a p-value of 0.014, so it would be quite surprising if it were no different from the first. A common bar is 0.05: below it, we call a result different. Once the bar is made harder to clear, to account for all 11 tests, no headline clears it. Upworthy's file marked Headline 3 as its pick: 4.11% clicked against 3.89% for the first, with a p-value of 0.78, which is not surprising at all.
Run the free test on your own p-values.
Also on this topic: The Multiple Comparisons Problem, the longer read.
Our position: when you compare many variants, set the bar for the whole batch, not for each result. One test at the 0.05 bar gives a false alarm about 1 time in 20 when nothing is going on. Eleven tests give far more chances.
The answer first: a real test of 12 headlines
The five at the top of this table are the five that looked different at the 0.05 bar. After the Holm correction method, which makes the bar harder to clear, each p-value is raised to an adjusted p-value. The smallest adjusted p-value is 0.154, and nothing falls under 0.05. The other five headlines (5, 9, 10, 11, 12) were not different even before correcting.
Holm controls the chance of any false alarm. The Benjamini-Hochberg procedure instead controls the share of false alarms among your findings.
| Click rate | Raw p-value | After Holm | |
|---|---|---|---|
| First headline (the reference) | 3.89% | n/a | n/a |
| Headline 4 | 2.49% | 0.014 | 0.154 |
| Headline 8 | 2.50% | 0.016 | 0.162 |
| Headline 6 | 2.53% | 0.019 | 0.174 |
| Headline 2 | 2.54% | 0.021 | 0.174 |
| Headline 7 | 2.72% | 0.049 | 0.340 |
| Headline 3 | 4.11% | 0.783 | 1.000 |
*Data: J. Nathan Matias, Kevin Munger, Marianne Aubin Le Quere and Charles Ebersole (2021), The Upworthy Research Archive, osf.io/jd64p, CC BY 4.0. Changes: charts by MCP Analytics, computed from this data. One test of 12 headlines, about 1,930 to 2,081 readers each. The first-created headline had 79 clicks from 2,032 impressions. The file marks no original, so we used the first created, a rule we wrote down before running anything.
What this means for your decisions
- Do not ship the headline that "won" a batch. Pick the best-looking of 12 and you may have picked the one that luck favoured most.
- Read the direction. All five that looked different were lower than the first headline, 2.5% to 2.7% against 3.89%. Only one headline of 11 beat the first at all (Headline 3, 4.11%, p-value 0.78: nothing there).
- A thin reference can fool you. Each comparison here leans on the first headline, which had 79 clicks. If that count ran high by luck, each other headline looks worse. These data cannot tell us whether it did. That is the case for a repeat before acting.
What the multiple comparisons problem is
Run many tests at once and some will look different from the first by luck alone. That is the multiple comparisons problem. Each test has a small chance of a false alarm, and those chances add up. A correction method makes the bar harder to clear for each result, so the whole batch stays honest. The free tool reports three of them: Bonferroni, Holm and Benjamini-Hochberg, explained below.
The batch is every headline, segment or metric you test in one go. Here it is 11 tests, each headline against the first.
One test vs many tests: why 0.05 stops working
For one test, a 5% chance of a false alarm is a fair bet. For 11 tests the chance that at least one is a false alarm is much higher. Which of the three correction methods to use depends on what a wrong call costs you.
- Bonferroni is the strictest: divide the 0.05 bar by the number of tests. Here the bar becomes 0.00455 (0.05 divided by 11) and no headline reaches it.
- Holm does the same job a little less harshly, so it can keep more results than Bonferroni. It kept none of the 5.
- Benjamini-Hochberg aims to keep false leads to about 5% of the results it keeps. It is the one to reach for when you screen many options and can live with a few wrong ones. It kept none either: its smallest adjusted value was 0.057, just over the line.
How to check your own batch of tests
- Count the batch before you look. Each headline, segment or metric you compare in one go is part of it.
- Name the comparison headline first. Say which one the others are compared with, and write it down before the data comes in. We did.
- Run the corrected comparison. Raw p-values in, adjusted p-values out. Act on the adjusted ones.
- Treat what survives as a lead, not a verdict. A result that clears the bar after correction is still worth a repeat on fresh readers before it moves a budget.
What you need and what you get back
What your file needs:
- One row per test.
- The raw p-value of each test, before any correction.
- Optional: a label for each test, such as the headline.
What you get back (from the free multiple comparisons test):
- Bonferroni, Holm and Benjamini-Hochberg side by side.
- Which results survive each correction.
Try it: the free multiple comparisons test runs this on your own p-values.
Questions business owners ask
Is 0.05 the wrong bar?
It is a fair bar for one test. The trouble starts when you use it eleven times. Here 11 tests at 0.05 would be expected to give about 0.55 false alarms on luck alone.
Why did correcting remove all five?
The correction asks each result to clear a higher bar. The best raw p-value was 0.014. Holm turned it into 0.154. A gap that looks decisive on its own can look ordinary once you admit you also looked at ten others.
Does "not significant" mean the headlines are the same?
No. It means these readers did not give us enough to tell them apart from luck. Larger tests may.
Doesn't Upworthy's own testing tool pick a winner?
It does, and its file flags one: Headline 3, the only headline above the first. We did not use that flag. Against the first headline, Headline 3 is 4.11% to 3.89% with a p-value of 0.78.
How many comparisons is too many?
There is no fixed count. The question is whether you would have acted on any one of them. If yes, they are one family.
Should I use Holm or Benjamini-Hochberg?
Use Holm when one wrong call is expensive. Use Benjamini-Hochberg when you are screening many options and a few false leads are cheap to drop.
The method in full
One Upworthy test (clickability_test_id 5308c3271e030e5930000003) with 12 headlines. Each of the 11 later headlines is compared with the first one created, on the share who clicked (clicks out of impressions). Test: each later headline is compared with the first on the share who clicked, by asking how often luck alone would give a gap that big between two click rates, at the 0.05 bar. The three correction methods are Bonferroni, Holm and Benjamini-Hochberg. At 0.05, 5 of 11 look different before correction and 0 of 11 after any of the three. Bonferroni bar 0.00455. Smallest p-value 0.01398 (Headline 4), 0.1538 after Holm, 0.0574 after Benjamini-Hochberg. Upworthy's own flags are not used. The rules were written before the analysis was run. For more, read The Multiple Comparisons Problem.
How we checked it
Three things to know about this test.
One headline to compare against, named first
Each of the 11 later headlines is compared with the first one created, because the file marks no original. We wrote that rule down before running anything.
A test that matches the data
Each headline is compared with the first on the share who clicked, using a standard test for comparing two shares.
Three corrections, one answer
All three correction methods keep none of the five. After the bar is made harder to clear, the smallest p-value is 0.154 with Holm and 0.057 with Benjamini-Hochberg, and both are above 0.05.