Connect base rates, evidence quality, sampling uncertainty, and consequences in one complete case.
By the end: Evaluate a decision policy without confusing a posterior, a p-value, or an expected-loss calculation with a real-world guarantee.
Start with the complete proposal
A factory proposes screening 10,000 items from a population with 1% fault prevalence. The test has sensitivity 90% and false-positive rate 5%. Acting prevents all modeled fault loss, with cost zero on faulty items and five units on sound items. Missing a fault costs 100 units. Screening itself costs 0.2 units per item. These assumptions are invented and must be validated before real use.
A positive has fault probability 90/585, about 15.38%, above the 4.76% action threshold. A negative has fault probability 10/9,415, about 0.106%, below it. The model therefore recommends acting on positives and waiting on negatives. This is a policy over possible results, not a claim that every positive is faulty.
Identify the population and the result pathways
The proposal combines several claims that require different evidence. Prevalence describes the fraction faulty in the target population before this screening result. Sensitivity describes how often faulty items test positive. The false-positive rate describes how often sound items test positive. The loss table describes consequences of acting or waiting in each state. Screening price applies to everyone tested. A good estimate of one input does not validate the others. Probability identities combine these inputs; they do not establish them.
Define an item and a time horizon before evaluating the numbers. A fault might mean a defect present at screening, a failure within a month, or an inability to meet a specified load. Those are different states. A detector that predicts eventual failure need not identify defects present today. Likewise, a 1% prevalence among items returned by customers need not describe newly manufactured stock. Changing the target population can change prevalence, and changing the mix of fault types can change sensitivity as well.
Under the stated target-population assumptions, begin with 10,000 as a convenient reference size. One percent gives 100 faulty items and 9,900 sound ones in expectation. Sensitivity 90% divides the faulty branch into ninety positives and ten negatives. A 5% false-positive rate divides the sound branch into 495 positives and 9,405 negatives. Each division uses a conditional rate within its own state. Adding the four leaves returns 10,000. Conservation of probability is an arithmetic check, not evidence that the rates apply to this factory.
Keep the state branches separate until the result is known
Schematic probability tree, not an area or length encoding. Expected frequencies refer to 10,000 items at 1% prevalence. The two positive leaves combine to 585; the two negative leaves combine to 9,415. Rates next to state branches apply within that state.
The result groups mix the state branches. There are 585 expected positives, so their fault fraction is ninety divided by 585, about 15.38%. There are 9,415 negatives, so their fault fraction is ten divided by 9,415, about 0.106%. Both are posterior probabilities under this model. Sensitivity remains ninety divided by 100, or 90%. Its denominator contains faulty items, while the positive posterior's denominator contains positive items. The different denominators explain why a sensitive detector can still produce a positive group mostly composed of sound items.
The likelihood ratio for a positive is 0.90 divided by 0.05, or eighteen. Prior odds are 100 to 9,900, equivalent to one to ninety-nine. Multiplying by eighteen gives posterior odds eighteen to ninety-nine, or two to eleven. Converting those odds to probability gives two divided by thirteen, the same 15.38%. This route compares how readily each state produces the evidence. It does not turn the likelihood 0.90 into a 90% posterior, and it requires the alternative sound state in the denominator.
Compare each posterior with the action threshold only after completing those probability calculations. The positive group is above the threshold, while the negative group is below it. That supports different actions for the two result groups under the stipulated losses. It does not establish a causal benefit from isolation or determine an acceptable disruption cost. An audit can accept the arithmetic while rejecting evidence for test rates or action effectiveness.
The same positives answer different conditional questions
Quantity
Numerator
Denominator
Value
Sensitivity
90 faulty positives
100 faulty
90%
Positive posterior
90 faulty positives
585 positives
15.38%
Negative posterior
10 faulty negatives
9,415 negatives
0.106%
False-positive rate
495 sound positives
9,900 sound
5%
Expected frequencies under the proposed model. A row conditions on its denominator; switching denominators switches the question.
Evaluate every branch, including test cost
The 495 false positives incur 2,475 loss units. The ten missed faults incur 1,000. Correct decisions incur zero by assumption. Screening costs 2,000 for all 10,000 items. Total expected loss is 5,475, or 0.5475 per item. These expected counts need not be the realized counts in one batch.
Without screening, waiting has expected loss 100 faults times 100, or 10,000; acting on every item has expected loss 9,900 sound items times five, or 49,500. Thus the best no-test policy is waiting. Screening improves modeled loss by 4,525 units per 10,000 items, not by its raw fraction of correct labels.
Compare complete policies against a relevant baseline
A screening policy specifies what happens for every possible result, not only for the positive cases discussed in a sales claim. Here it means test everyone, act on positives, and wait on negatives. The losses come from two mistaken-action branches plus a price paid on every test. The model expects 495 sound positives and ten faulty negatives. Their contributions are 495 times five, or 2,475, and ten times 100, or 1,000. Before test price, total expected loss is 3,475 per 10,000 items.
A plausible alternative is not necessarily doing nothing in every application. In this constructed case, compare both available constant policies. Waiting on everyone loses 100 on each of 100 expected faults, totaling 10,000. Acting on everyone loses five on each of 9,900 sound items, totaling 49,500. Waiting is the better no-test baseline. Choosing the worse baseline exaggerates screening's value. All comparisons use the same population, consequences, and time horizon, so differences reflect policies rather than changed assumptions.
Express the accounting per item to see what a marginal improvement buys. Missed-fault loss is prevalence times the missed-fault fraction times 100. Here that is 0.01 times 0.10 times 100, or 0.10. False-alarm loss is the sound fraction times the false-positive rate times five. That is 0.99 times 0.05 times five, or 0.2475. Screening price adds 0.20, giving 0.5475 expected units. The price applies to all tested items, even those correctly classified, whereas the other contributions apply only to particular branches.
Compare complete policies on the same population
Policy
False-alarm loss
Missed-fault loss
Test price total
Expected total
Wait on all
0
10,000
0
10,000
Act on all
49,500
0
0
49,500
Test; act on positives
2,475
1,000
2,000
5,475
Loss units per 10,000 items. The relevant no-test comparison is waiting, the better of the two constant policies. Screening price is paid on every tested item.
Suppose sensitivity improves from 90% to 95% while every other input stays fixed. Five additional faults per 10,000 move from the missed branch to the correctly acted-on branch. Avoided expected loss is five times 100, or 500, equal to 0.05 per item. Separately, suppose the false-positive rate falls from 5% to 4%. Ninety-nine sound items move out of the false-alarm branch, saving 495 units, or 0.0495 per item. Different populations and losses make these unequal percentage-point improvements similarly valuable.
These comparisons assume that acting on positives and waiting on negatives remain optimal after the change. That is true for the two changes just described under the original losses. In general, a sufficiently changed test or population can move a result group's posterior across the action threshold. Then recompute the policy rather than inserting new rates into an obsolete rule. A highly uninformative test can leave both result groups on the same side of the threshold. Paying for it would then buy no improvement for this decision.
Raw classification accuracy answers another question. Labeling every item sound is 99% accurate here, while the proposed detector is correct on ninety faulty and 9,405 sound items, or 94.95% overall. Yet the detector-guided action policy has lower expected loss after its stated price. Accuracy counts all errors equally; the loss model does not. This contrast does not prove that lower accuracy is desirable. It shows why the evaluation criterion must match the proposed use, with explicit values for costly misses and disruptive false alarms.
Value improvements by their affected branches
Change alone
Expected items moved
Loss saved per moved item
Total saving
Sensitivity 90% to 95%
5 faulty negatives become positives
100
500
False-positive rate 5% to 4%
99 sound positives become negatives
5
495
Expected savings in loss units per 10,000 items, holding price and all other inputs fixed. The original result-dependent actions remain optimal in both comparisons.
Work through the mechanism
Screening branch
Expected count
Loss per item
Total loss
False positive
495
5
2,475
Missed fault
10
100
1,000
Correct decisions
9,495
0
0
Test on every item
10,000
0.2
2,000
Total 5,475. Branch counts are expectations under fixed population and conditional-rate assumptions.
Trace uncertainty back to the evidence for each input
The policy table does not estimate rates from observations. Suppose an evaluation reports ninety positives among 100 confirmed faulty items. The measured sensitivity is 90%, but other representative samples would produce different proportions. If those items were independent and shared a fixed positive probability of 0.90, the sample proportion's standard error would be the square root of 0.90 times 0.10 divided by 100, or 0.03. This three-percentage-point spread describes sampling variation, not all uncertainty.
Sound items require a separate denominator. Five positives among 100 sound items estimate a 5% false-positive rate. Fifty among 1,000 also estimate 5%, but provide more information under the independent-trial model. Their plug-in standard errors are about 2.18 and 0.689 percentage points respectively. Ten times as many observations reduce this spread by the square root of ten, not ten. Measuring sensitivity on many faults does not provide the missing sound-item evidence. Both denominators matter because the final policy combines both state branches.
Independence deserves a collection-level check. One hundred faulty items from one production incident might share the same unusual defect and measurement conditions. Repeating the detector reading on each item twice creates 200 readings, not necessarily 200 independent examples. Separate button presses do not establish independence of their readings conditional on faulty status. If contamination or defect type affects both results, multiplying their likelihoods overstates the evidence. Identify batches, repeated items, and shared sources to assess which units genuinely vary.
More readings need not mean more independent examples
Each circle represents one item, with an inner dot marking a second reading of the same item. The left arrangement samples six separate batches; the right takes six items from one batch. The grouping is schematic. Neither repeated readings nor item count alone establishes conditional independence or a representative mix of defects.
Interval language must match its construction. A 95% confidence procedure concerns how often its intervals cover a fixed rate across repeated valid samples. It does not assign 95% posterior probability to the rate lying inside the particular reported bounds. A Bayesian credible interval uses a prior and likelihood to assign posterior probability to rate values. Both can be useful, but neither automatically covers population shift or measurement error. A narrow interval from unrepresentative, highly selected examples can precisely summarize the wrong collection process.
Selection adds another source of optimism. If a team compares many thresholds on one dataset and reports only the lowest-loss version, the same dataset helped choose and evaluate the winner. An untouched evaluation set measures the frozen choice without that direct reuse. It must still represent the intended deployment conditions. Likewise, a small p-value for detector association is not a posterior probability of low policy loss. Test size is a rejection probability under a null, and power is a rejection probability under a specified alternative, neither a probability that this chosen policy is wrong.
Guide a paper sensitivity check by changing only the false-positive rate to 8%, leaving prevalence, sensitivity, and losses fixed. The sound positives become 792, giving false-alarm loss 3,960. Missed-fault loss remains 1,000 and screening price remains 2,000, so total expected loss rises to 6,960. A positive still exceeds the action threshold, so the original result-dependent policy remains appropriate. This scenario is not an interval endpoint established by data. It identifies how a plausible input change would affect consequences and what evidence would be valuable to collect.
A changed false-positive rate changes policy loss
False-positive rate
Sound positives
False-alarm loss
Other losses and price
Total
5%
495
2,475
3,000
5,475
8%
792
3,960
3,000
6,960
Expected quantities per 10,000 items. The 8% row is a hypothetical sensitivity check, not a confidence bound. Other losses and price are 1,000 for missed faults plus 2,000 for testing.
Inspect the evidence that supports the inputs
Ask whether prevalence and test rates came from a representative population, how uncertain their estimates are, and whether the detector threshold was selected on the evaluation data. A measured rate is not a known constant. Check uncertainty and plausible changes in prevalence, dependence, fault severity, intervention effectiveness, and costs before deploying.
If many detector variants were tried, use an untouched evaluation set or another valid design to assess the selected policy. A small p-value for association cannot by itself establish the loss reduction from acting. The intervention’s causal effects, distribution of harms, and operational constraints need separate evidence. Preserve both the mathematical conclusion under assumptions and the reasons those assumptions might fail.
Check causal effects and operational transfer before deployment
A detector can identify fault without establishing that the proposed action prevents harm. Suppose the actual isolation procedure leaves 20% of fault loss even when a faulty component is identified. Keep the other consequences, including zero action cost on faults. Acting on the ninety true positives now leaves twenty units each, adding 1,800 expected units. False-positive loss, missed-fault loss, and testing price still contribute 5,475. Total screened loss becomes 7,275. This remains below waiting's 10,000, but the improvement is smaller.
Recheck the threshold after effectiveness changes. Acting now loses twenty on faulty items and five on sound ones. Its expected loss is five plus fifteen times fault probability, while waiting loses 100 times fault probability. Equality occurs at five divided by eighty-five, about 5.88%. The positive posterior of 15.38% remains above this boundary; the negative posterior remains below. Thus the policy stays act on positives and wait on negatives. Without this check, the revised total would merely price a fixed policy, not establish its optimality.
Imperfect prevention adds a loss on true positives
Expected loss per 10,000
Complete prevention
20% residual fault loss
90 acted-on faults
90 × 0 = 0
90 × 20 = 1,800
False positives, misses, testing
5,475
5,475
Screening policy total
5,475
7,275
Best constant policy: wait
10,000
10,000
Action threshold
5/105 ≈ 4.76%
5/85 ≈ 5.88%
The same classification branches remain. Only the consequence of acting on a faulty item changes. The positive and negative posteriors remain on opposite sides of the revised threshold, so the result-dependent action rule is still optimal among the two available actions under these assumptions.
Evidence for that residual loss needs a causal comparison. If operators isolate only the most visibly damaged positive items, later outcomes among isolated items can look worse despite beneficial isolation. Initial severity affects both action choice and outcomes. Random assignment among eligible items can separate the intervention comparison from that initial selection, subject to implementation and ethical constraints. Randomly sampling more factory items does not itself remove action-selection bias. It addresses representation, while assignment addresses which comparable items receive which action.
Probability forecasts require their own deployment checks. Suppose a detector assigns every component a 1% risk and approximately 1% eventually prove faulty. The forecast can be calibrated overall while offering no separation between items. Another forecast might distinguish a high-risk group from a low-risk group yet consistently exaggerate both probabilities. A result-dependent policy needs risk groups that justify different actions and probabilities on a trustworthy scale near its decision threshold. A constant forecast can still support a useful constant action. Finite observed frequencies fluctuate, so calibration checks need counts and uncertainty rather than demanding exact equality in every small group.
Operational choices can also alter the population observed later. If isolated items receive detailed inspection while negative items disappear into production, confirmed faults will be easier to find among positives. An audit based only on confirmed cases then misses faults in the negative branch. Follow-up must establish outcomes in a way that supports both denominators, perhaps through independent inspection of a representative subset of each result group. Where intervention prevents a later failure, distinguish evidence about initial fault status from the later outcome that the intervention itself can change.
State the assumptions and measurements still needed. Under the original rates, losses, and price, the proposed screening policy improves expected loss relative to the best constant action. To transfer that result, freeze the detector and threshold, evaluate representative independent units, and obtain evidence about intervention consequences and operational costs. If new measurements shift a result posterior or an action loss enough, recompute the action rule and the complete policy total. The arithmetic derives consequences; the audit checks whether inputs describe deployment.
Selective follow-up can hide faults among negatives
Schematic observation pathways, with no probability encoded by length. Detailed inspection only after a positive creates unequal verification. Outcomes among negatives need independent follow-up to estimate missed faults. If action changes later failure, later failure is not an unchanged measure of initial fault status.
Exercises
A more expensive test
Change only screening cost to one unit per item. Does the policy still beat the best no-test policy?
Hint
Classification losses remain 3,475; change the cost for all items.
Worked answer
Test cost per item
Total screened loss
Compare with 10,000
0.2
5,475
Lower
0.6525
10,000
Tie
1
13,475
Higher
Buying information and acting after receiving information are distinct decisions.
Total screened loss becomes 3,475 + 10,000 = 13,475, worse than 10,000 from waiting without screening. The maximum test cost that ties the no-test policy is (10,000 − 3,475)/10,000 = 0.6525 per item. The positive-result posterior has not changed, but purchasing the test is no longer worthwhile under this model.
A cleaner evaluation
The reported 90% sensitivity came from choosing the best of fifty detectors on the same fifty faulty examples. What would you ask for before using 90% as a known constant?
Hint
Separate selection, sampling uncertainty, and the target population.
Worked answer
Uncertainty source
Useful check
Selection of winner
Fresh or properly nested evaluation
Small sample
Counts and interval estimates
Population shift
Representative deployment data
Action effectiveness
Evidence about intervention consequences
A mathematically correct posterior cannot validate unmeasured inputs.
Request independent evaluation of the frozen detector on representative faulty and sound items, with counts and uncertainty for both sensitivity and false-positive rate. Ask how thresholds and exclusions were chosen, whether items share causes, and how the production population differs. Fifty selected examples do not establish a precise transferable rate. Also obtain evidence for what acting actually prevents and what it costs.
Sources and further reading
The explanations and exercises are complete without these optional links. The references distinguish probability identities from inference procedures and reporting guidance. The examples are constructed teaching models, not empirical estimates or professional decision advice.
The ASA’s six principles on interpretation, effect size, transparency, and avoiding threshold-only conclusions. This linked PDF is the announcement and summary, not the full technical article.