Multiple comparisons, repeated peeking, and selective reporting change what a striking result means.
By the end: Calculate a family false-positive probability and identify selection that invalidates a nominal error rate.
The selected minimum is not a single planned test
If twenty independent tests each have false-positive probability 0.05 under their nulls, the chance of no false positives is 0.95 to the twentieth power. The chance of at least one is about 64.15%. Looking only at the most exciting test hides the many opportunities to produce it.
The formula assumes independent tests and exactly 5% false-positive probability per test. Correlated tests require their joint behavior; discrete conservative tests may have lower individual rates. The broader problem remains: selecting among analyses is part of the procedure whose error rate must be evaluated.
Count opportunities as part of the experiment
Consider a laboratory searching for a change across twenty measured outcomes. Suppose every null is true and each planned test rejects with probability exactly 0.05. If the rejection events are independent, the probability that all twenty avoid rejection is 0.95 multiplied twenty times, about 0.358486. The complementary event, at least one false rejection, has probability about 0.641514. The selected smallest p-value comes from this collection of opportunities. Treating it as the result of one prechosen question changes the experiment being described.
Expected number and chance of any are different summaries. Each test contributes an indicator that is one for a false rejection and zero otherwise. Adding their expectations gives twenty times 0.05, or one expected false rejection. This calculation does not require independence. It also does not promise exactly one false rejection in every collection. Under independent tests, some collections have none and others have several. Under dependence, the same expected count can arise from a very different pattern of co-occurrence.
For a smaller outcome grid, imagine two independent tests each with false-rejection probability 0.10. Neither rejects with probability 0.9 times 0.9, or 0.81. Only the first rejects with probability 0.1 times 0.9, or 0.09. Only the second also has probability 0.09, and both reject with probability 0.01. The family probability is 0.09 plus 0.09 plus 0.01, or 0.19. Simply adding 0.10 and 0.10 counts the joint rejection twice.
The family event is the union, including overlap once
The square has probability area one. Horizontal and vertical lengths encode independent test outcomes, each with a 10% false-rejection chance. Strip dimensions give 0.09, 0.09, and 0.01, so their union has mass 0.19, not 0.20.
Compare perfect dependence. If both test reports are guaranteed copies of the same underlying decision, either both reject or neither rejects. The family probability is then 0.10, not 0.19. Both arrangements still have expected rejection count 0.20. Collected twice does not establish independence: shared participants, overlapping outcomes, and common measurement errors can couple results. Conditional independence under a specified generating model might justify multiplication, but two filenames or two reporting dates do not provide that justification.
A family need not consist of twenty named outcome variables. Trying several exclusions, transformations, subgroups, or model specifications can create opportunities too. The relevant procedure includes the choices that could have led to the reported claim. If only one analysis is reported, readers may not know how many alternatives were examined. The problem is not that exploration is forbidden. It is that a probability calculation for one fixed analysis cannot automatically describe a search that selected that analysis because it looked unusual.
Before evaluating a reported result, reconstruct the selection pathway. Ask which outcome was designated first, which alternatives were examined, and what would have been reported if another result had been smaller. When the search is recorded, its behavior can be assessed or new data can test the resulting hypothesis. When it is hidden, the nominal threshold no longer identifies the actual family error. The independent formula is a worked model, while the broader requirement is to analyze the procedure that actually generated the claim.
The same expected count can conceal different joint behavior
Two null tests, 10% each
Neither rejects
Exactly one
Both
Any
Independent
81%
18%
1%
19%
Guaranteed copies
90%
0%
10%
10%
Expected count in either case
0 × P(neither)
1 × P(exactly one)
2 × P(both)
0.20
In the independent row, expectation is 0.18 + 2 × 0.01 = 0.20. For copies it is 2 × 0.10 = 0.20. The family event and count expectation measure different features of the distribution.
Control a stated family of decisions
Bonferroni testing assigns each of m tests a threshold alpha divided by m. The union bound ensures that the chance of one or more false rejections is at most alpha if each true-null p-value is valid, without requiring independence. With twenty tests and family target 0.05, the per-test threshold is 0.0025.
This can be conservative and reduce power. Other procedures target different quantities. False discovery rate concerns the expected proportion of false rejections among all rejections, with the proportion defined as zero when nothing is rejected. It is not the same as the chance of any false rejection. The appropriate target depends on how results will be used.
Allocate error probability across a stated family
A valid null p-value has a specific operating property. Under its null model, the probability that it falls at or below a threshold is no greater than that threshold. Discrete tests may fall below it less often. Bonferroni uses this property by assigning small thresholds to individual tests. The guarantee requires valid p-values for the stated procedures, including their sampling and analysis assumptions. Dividing arbitrary scores by a family size does not turn them into probabilities with this operating property.
Suppose a family contains four planned hypotheses and the total permitted chance of any false rejection is 0.05. Give each test a threshold of 0.0125. For each true null, its rejection event has probability at most 0.0125. The probability of their union is at most the sum of their probabilities. Even if all four nulls are true, that sum is at most 0.05. If some nulls are false, only the remaining true-null events count as false rejections, so the same upper bound still holds.
The union bound does not require independence. Picture overlapping rejection events within the space of possible datasets. Adding their separate weights can count a dataset more than once, but it cannot omit a dataset belonging to their union. Therefore the sum is an upper bound, even when the overlap is large. With four independent tests that each actually reject at rate 0.0125, the family error is one minus 0.9875 to the fourth power, about 0.04907. It is slightly below the bound.
An error budget bounds a union without independence
Planned allocation
Individual thresholds
Sum
Family error bound
Four equal tests
0.0125 each
0.05
At most 5%
One primary, two secondary
0.03, 0.01, 0.01
0.05
At most 5%
Four independent tests, exact rates
0.0125 each
0.05
Actual 4.9070%
Every true-null p-value must be valid for its actual analysis procedure. Summing event probabilities can overcount overlapping datasets, so it bounds the union from above. Independent tests are needed only for the last row's exact product calculation.
Equal allocation is convenient, not compulsory. A prespecified primary question could receive threshold 0.03 while two secondary questions receive 0.01 each. The total remains 0.05, so the same bound applies to any false rejection in the family. This allocation expresses which questions deserve more detection opportunity before seeing their results. Giving the largest budget afterward to whichever test had the smallest p-value would create a different procedure. Its error guarantee would require a separate argument that includes that selection.
Familywise control answers whether the procedure makes any false rejection. False discovery rate asks about a different quantity. In each repetition, divide false rejections by all rejections, using zero if there are none. Then average that fraction across repetitions. For example, imagine half of repeated datasets produce no rejections, while half produce ten rejections including two false ones. The familywise error is 50%, but the false discovery rate is half of 0% plus half of 20%, or 10%.
Neither error target is a posterior probability that a particular reported hypothesis is false. The example's ten rejected claims may have different evidence and different prior plausibility. The average error fraction does not assign identical probabilities to them. Choose the target according to how the discoveries will be used, then use a procedure with appropriate assumptions. A single false rejection can be costly in a confirmatory decision, whereas a screening program may accept some false discoveries for later checking. Those consequences motivate the target but are not its mathematical definition.
Average the false fraction within each repeated dataset
Repeated-dataset outcome
Chance
Rejections
False rejections
False fraction
Nothing rejected
50%
0
0
0% by convention
Ten rejected, two false
50%
10
2
20%
Familywise error
50% chance of any false rejection
50%
False discovery rate
0.5 × 0% + 0.5 × 20%
10%
This constructed distribution distinguishes two error targets. The ratio of expected false count to expected total count would be 1/5 = 20%, which differs from the 10% expected within-dataset fraction.
Work through the mechanism
Independent null tests at 5% each
Chance of at least one false positive
1
5%
10
≈ 40.13%
20
≈ 64.15%
These are family probabilities under a specified idealization, not the chance any particular reported finding is false.
Include monitoring and stopping in the rejection event
Repeated peeking creates selection across time. A fixed-sample test describes a statistic at its specified sample size. If an analyst checks after each new observation and stops at the first small ordinary p-value, the rejection event becomes the union of crossing events at many sample sizes. These events usually overlap and are dependent because later analyses reuse earlier observations. Multiplying independent no-rejection probabilities is therefore generally wrong. Ignoring all but the final chosen look is also wrong, because that look was selected by the path.
A tiny exact example shows the effect without assuming independent looks. Toss a fair coin and plan to observe two tosses. Consider rejecting for unusually many heads using a one-sided threshold of 0.5. At one toss, a head has p-value 0.5. At two tosses, two heads have p-value 0.25, while one head has p-value 0.75. Thus a fixed two-toss test rejects only on two heads, with probability one quarter. The generous threshold makes every path visible; it is not a recommended scientific standard.
Now check after one toss as well, stopping as soon as the p-value is at most 0.5. Any path starting with heads triggers rejection immediately. The possible full paths HH and HT each have probability one quarter, so the monitored procedure rejects with probability one half. The fixed endpoint rejected only HH. Early stopping has added HT, even though that full dataset would not reject at the planned endpoint. The claimed fixed-sample probability fails because the rejection event changed, not because any individual tail calculation was numerically incorrect. Here the monitored rate equals, rather than exceeds, the nominal 0.5 threshold. This example shows a change in actual rejection chance, not a nominal-bound violation.
The same problem can occur with conventional thresholds and larger samples, but its size depends on the full monitoring design. Checking only a few planned times differs from checking after every observation. A time limit differs from continuing indefinitely. Correlation between looks matters. Valid sequential tests account for these features through boundaries or other guarantees constructed for monitoring. One possible conservative approach allocates a total error budget across a fixed set of looks and applies a union bound, provided each look's underlying test is valid under the null.
Stopping early adds a path to the rejection event
The full paths represent what both tosses would have been, even if monitoring stops after the first head. At threshold 0.5, the fixed two-toss rule rejects with probability 1/4 and monitoring rejects with probability 1/2. The monitored rate equals the nominal 0.5 threshold here; this example changes actual size without exceeding that nominal bound.
Selective reporting can occur without early stopping. Imagine running the whole planned study but presenting only the subgroup that showed the largest increase. The reported subgroup estimate tends to exaggerate the underlying increase because selection favors both real signal and favorable sampling variation. This is often called selection bias in the estimate. Correcting a threshold addresses an error-rate question; it does not automatically remove the exaggeration in the selected effect estimate. Estimation after selection needs its own method or a fresh evaluation sample.
Use the small coin paths to audit a monitoring proposal. Write down which complete paths would eventually lead to a claim, including paths stopped early, and add their probabilities once. Then compare that event with the original fixed-sample rejection event. For larger studies, enumeration may be impractical, but the question remains concrete. A valid guarantee must cover every way the procedure can produce a claim. Stating the stopping rule afterward does not restore the guarantee of a plan that never included it.
Stopping and reporting belong in the plan
Repeatedly collecting data and stopping as soon as an ordinary fixed-sample p-value becomes small generally does not preserve that test’s nominal false-positive rate. Valid sequential methods exist, but their boundaries and guarantees must match the monitoring procedure. “We stopped when it looked convincing” is not such a method.
Preregister the main question, outcomes, exclusions, and analysis when possible; label exploratory work and test it with genuinely new data. Preregistration does not make a poor design good. Full reporting, held-out evaluation, and replication reduce selection problems but still need relevant populations, reliable measurements, and independent evidence.
Evaluate selected claims with genuinely new evidence
Separate discovery from evaluation in a concrete workflow. Use an initial dataset to choose one outcome definition, detector threshold, and analysis. Freeze those choices before examining an independent evaluation dataset. If the evaluation test has valid size 5% for every possible frozen choice under its null, selecting the choice with independent discovery data does not itself increase that evaluation test's null rejection probability. Averaging error probabilities that are each at most 5% still gives at most 5%. The guarantee depends on keeping evaluation outcomes out of selection.
An untouched evaluation set is not untouched if its results repeatedly guide revisions. Suppose the first evaluation looks weak, so the analyst changes exclusions and tries again on the same held-out records. Those records now influence which rule is selected. Calling the split a holdout does not remove this dependence. Repeated tuning needs a design that accounts for reuse or another genuinely fresh evaluation. The separation concerns information flow, not whether files have names such as training and test or whether different people opened them.
New evidence also needs a model of shared causes. Two laboratories might analyze aliquots from the same contaminated specimen, producing correlated errors even if their instruments differ. Two reports might quote the same original analysis. Multiplying likelihood ratios requires conditional independence under both the claim and its alternative, or else likelihood ratios conditioned on previous evidence. A second collection or publication is not enough. Replication is strongest when it tests the proposed mechanism while addressing shared error routes that the first study could not distinguish.
A holdout fails when its outcomes guide revisions
The upper workflow selects a rule using discovery data independently of fresh evaluation outcomes. The lower return path makes the revised rule depend on evaluation results. File separation alone does not preserve the statistical separation.
Error rates still do not directly supply posterior credibility. For illustration, suppose 1% of investigated claims are genuine, a fixed test detects 80% of genuine claims, and it falsely rejects 5% of null claims. Among 10,000 hypothetical investigations, the expected positive counts are 80 genuine and 495 false. The genuine fraction among positives is 80 divided by 575, about 13.91%. These are expected frequencies under stipulated rates, not promised counts. A posterior interpretation additionally treats that prevalence as a relevant prior model.
Keeping that same prevalence, lowering actual false-rejection probability to 0.5% while holding power at 80% gives 80 expected genuine positives and 49.5 false ones. The genuine fraction becomes about 61.78%. Holding power fixed isolates the arithmetic; stricter thresholds commonly reduce power too. The likelihood of a positive under each state and the prior mixture jointly determine the reverse conditional. A multiplicity correction controls an operating error target. It cannot alone establish the probability that a selected finding is true or guarantee a useful effect size.
Fresh data must answer the intended population and causal question. Random sampling helps describe the population supplying the observations, while random assignment helps compare intervention effects without systematic treatment selection. Neither operation by itself prevents analysis selection, bad measurements, or nontransferable populations. Report the search, frozen evaluation rule, effect estimates, and uncertainty alongside the error target. If a claim will trigger action, evaluate consequences separately. Evidence that a selected association survives evaluation does not yet measure what an intervention prevents or what a mistaken action costs.
Error control and credibility use different denominators
Actual false-rejection rate
Expected genuine positives
Expected false positives
Genuine fraction among positives
5%
100 × 0.80 = 80
9,900 × 0.05 = 495
80 / 575 = 13.9130%
0.5%
100 × 0.80 = 80
9,900 × 0.005 = 49.5
80 / 129.5 = 61.7761%
Constructed mixture of 10,000 investigations with 1% genuine claims and power fixed at 80%. Counts are expectations. In practice, a stricter threshold can lower power, and a selected search needs its own operating rates. These rows do not identify the truth of any particular report.
Exercises
Ten independent tests
Find the chance of at least one false positive among ten independent tests with actual size 0.05. Is the answer the probability a particular positive result is false?
Hint
Start with the complement: no false positives.
Worked answer
Family outcome
Probability
No false positives
0.95¹⁰ ≈ 0.5987
At least one
≈ 0.4013
A family error rate does not supply a posterior for a selected claim.
The chance is 1 − 0.95¹⁰, about 0.4013. It describes the testing procedure under all ten nulls. The probability a selected positive is false needs a model of how often hypotheses are true, power, and selection; it is a different conditional question.
Protect a family of five
Five hypotheses form a planned family with desired familywise error at most 5%. What Bonferroni cutoff applies, and does it require independence?
Hint
Divide the total error budget among the five tests.
Worked answer
Quantity
Value
Number of tests
5
Per-test cutoff
0.01
Family error upper bound
5 × 0.01 = 0.05
The bound needs valid individual tests, not arbitrary scores labeled p-values.
Use 0.05/5 = 0.01 for each valid p-value. Independence is not required: the probability of any true-null rejection is at most the sum of their individual error probabilities, at most five times 0.01. Actual error can be lower, and power can be reduced.
Sources and further reading
The explanations and exercises are complete without these optional links. The references distinguish probability identities from inference procedures and reporting guidance. The examples are constructed teaching models, not empirical estimates or professional decision advice.
The ASA’s six principles on interpretation, effect size, transparency, and avoiding threshold-only conclusions. This linked PDF is the announcement and summary, not the full technical article.