Guides
Part of Field guide to estimate methodology: the five-step pipeline
Scoring estimate methodology methodology: what the absence causes
Estimate methodology and the missing correction loop: a number nobody can ever prove wrong is not cautious research, it is an unfalsifiable claim.
Every field that publishes numbers has a way to be shown wrong. A forecast meets the event. A survey meets a census. A financial statement meets an audit.
This subject has none of that. The absence is not a minor gap in quality control. It lets a wrong figure survive indefinitely and accumulate credibility.
What to take away
- Without ground truth there is no error signal, so nothing ever improves.
- A claim that cannot be tested is not a careful claim, it is an untestable one.
- Fields with real correction loops publish their limitations openly.
What a correction loop looks like elsewhere
| Field | The claim | What corrects it |
|---|---|---|
| Weather forecasting | Tomorrow's conditions | Tomorrow arrives |
| Election polling | A vote share | The count |
| Company accounts | A financial position | An audit and a regulator |
| Official statistics | A population measure | Revisions, benchmarking and published methodology |
| Scientific measurement | A physical quantity | Replication by other laboratories |
| Net worth estimates | A person's wealth | Nothing |
The last row is the entire argument of this page. Every other row has a mechanism that punishes error, and over time that mechanism improves the method. Where there is no mechanism, methods do not improve, they merely persist.
Correction Loops by Field
Field
- Weather forecasting
- Tomorrow arrives
- Election polling
- The count
- Company accounts
- Audit and regulator
- Official statistics
- Revisions and benchmarking
- Scientific measurement
- Replication by labs
- Net worth estimates
- Nothing
What corrects it
- Weather forecasting
- Election polling
- Company accounts
- Official statistics
- Scientific measurement
- Net worth estimates
What the absence causes
Errors never surface. A figure that was wrong at the outset stays wrong. Nothing arrives to contradict it, so nobody revisits it.
Repetition looks like confirmation. The same number in many places reads as corroboration when it is propagation. With no test available, agreement is the only signal a reader has, and it is the wrong one.
Confidence and accuracy decouple. In a field with feedback, publishing overconfident numbers is costly, because you are seen to be wrong. Here it is free, so confidence is selected for.
Methods drift toward what is publishable. With no accuracy signal, the surviving practices are the ones that produce usable output rather than the ones that produce correct output. The commercial side of that pressure is on richest rankings.
The falsifiability test
Take any published figure and ask what observation would prove it wrong. Not what would make you doubt it, but what specific, obtainable piece of evidence would falsify it.
For a company's disclosed pay, the filing itself provides the answer.
For someone's total wealth, every possibility fails, because each source falls short. A statement from the person is an interested assertion, another article repeats the claim, and a property record shows a purchase, not a net position.
No available document could contradict the number, so the number is safe and worthless.
Honest work differs: statistical agencies publish designs and known limits, as the Bureau of Labor Statistics does in its Handbook of Methods.
Survey researchers have conventions for what must accompany a result, set out in the disclosure standards of the American Association for Public Opinion Research. Measurement science requires reporting a value with its uncertainty, and guidance on expressing measurement uncertainty sets out how.
Every one of those conventions exists to make a claim contestable. None of them appears in this genre.
What a method with no feedback should do
Three things, all of which are available and none of which is difficult.
- State the inputs held, with issuers and dates, so a reader can see the evidence base rather than infer it.
- State the assumptions used and the range each one spans, so the output can be re-run at different settings.
- Publish an interval rather than a point, derived from those ranges.
Do all three and the reader can perform the correction that the world will not. That is the substitute for a feedback loop, and it is what a field without one owes its audience. The honest outputs it leads to are described on estimate methodology.
The absence of a correction loop does most damage to lifetime totals, which span too many years for any single document to contradict, as set out on career earnings. The one place where a real correction mechanism partly exists is on executive net worth.
Common questions
Do estimates ever get corrected when someone dies and an estate is settled?
Occasionally something surfaces, and it tends to show large divergence from published figures. It is also rare, delayed by years, and reflects an estate after taxes and settlements rather than the position that was estimated.
Is a long-standing figure more likely to be right?
The opposite. Longevity means it has been copied more often, not tested more often, and in a field with no test, age measures circulation.
Would better data fix this?
Better data on the visible parts would help those parts. Personal liabilities are not published anywhere for anyone, so the loop stays broken regardless.
