Thresholds and the cost of mistakes
🎯You'll be able to
Choose a threshold from the costs of each kind of mistake, or from a minimum recall agreed with the people who use the model, on validation data.
🏠The everyday picture
A shop owner decides how much stock to keep by weighing the cost of running out against the cost of unsold stock. A threshold is the same kind of decision: weigh missed late parcels against unnecessary calls.
📘What it really is
- A threshold is a business decision, not a property of the model. The default of 0.5 is just a habit.
- Two common rules:
- Costs: give each outcome a cost, compute the total for each threshold on validation data, and pick the lowest.
- A constraint: for example "catch at least 60% of late parcels", and among thresholds that do, pick the one with the highest precision. The kit uses this rule on May; it gave 0.30.
-
Choose on validation, apply to test. The test result tells you how well the choice holds. In the kit, recall was 0.6032 on May at 0.30, and 0.5349 on June: the constraint wasn't quite met in the new month, which the report should say.
-
Costs are estimates. Write where each number comes from and try a range.
- Capacity limits: sometimes the rule is "the hub can make 150 calls a day", so you flag the top 150 by score.
📝A worked example
Input: illustrative costs, chosen for this example and not DakDost's real figures: each call costs ₹20 of staff time; each late parcel costs ₹150 in refunds and complaints; a call prevents half of the late parcels it reaches.
cost = 20 × (TP + FP) + 150 × (FN + 0.5 × TP)
Expected output (reference figures, 2 November 2026):
| Threshold | May cost | June cost |
|---|---|---|
| No calls at all | ₹28,350 | ₹32,250 |
| 0.5 | ₹24,820 | ₹28,475 |
| 0.30 (the kit's recall rule) | ₹23,960 | ₹27,085 |
| 0.22 (lowest cost on May) | ₹23,660 | ₹26,905 |
| 0.15 | ₹23,865 | ₹26,440 |
On these costs, a threshold around 0.15 to 0.3 does best, and every threshold beats no calls. The cheapest point on May (0.22) wasn't the cheapest on June (0.15 did better there), a reminder that the choice is an estimate.
🌍Where it's used
- Banking: fraud thresholds set from losses and investigation costs, reviewed regularly.
- Health: screening thresholds set for a minimum sensitivity, then the cost of follow-up tests.
- Support teams: "flag the top N each day" matched to staff capacity.
📝Case scenario
A lender used the default 0.5 threshold for a "needs manual review" model and its reviewers sat idle while bad loans slipped through. Working out the cost of a missed bad loan against a review hour moved the threshold to 0.2, filling the review team's capacity and cutting losses. They recorded the costs and agreed to revisit them every quarter.
🖱️Try it
Change the cost of a late parcel from ₹150 to ₹300 and find the lowest-cost threshold on May. Success: it moves lower (more calls), and you can explain why.
⚠️Common mistakes
- Keeping 0.5 by habit. Sign: nobody can say why 0.5. Fix: choose from costs or an agreed constraint.
- Choosing the threshold on the test set. Sign: June used to pick it. Fix: pick on May, report June.
- Hiding the assumptions. Sign: costs appear without sources. Fix: write them down and test a range.
🛠️For your project
Project 1's test 7 recomputes the threshold on May with the kit's rule (recall at least 0.60, highest precision) and checks that it matches.
✅Quick check
Why not use 0.5 as the threshold?
Show the answer
It's a habit, not a decision; the right threshold depends on the costs of each mistake.
Where should the threshold be chosen?
Show the answer
On validation data, then reported once on the test set.