Test predictions fairly with probabilities and Brier scores
Write specific claims before the deadline, preserve uncertainty, and compare probabilities with observed outcomes without inventing evidence.
Make the claim resolvable
“Something important will happen soon” is difficult to disprove. A useful entry defines an observable event, a deadline and a rule for deciding whether it happened. Write these before the event. If you later change the definition to fit an outcome, you are testing a different claim. Keep a note of ambiguity rather than treating it as a success.
Base rates matter. A high probability for a common event can beat a dramatic-sounding rare prediction. This tracker stores only your own entries. There is no population leaderboard and no claim about how accurate other visitors are.
Score probability, not confidence in hindsight
For a resolved binary event, the Brier score is (p − o)², where p is the probability from 0 to 1 and o is 1 if the event occurred or 0 if it did not. A 50% forecast scores 0.25 either way. A certain, correct forecast scores 0; a certain, incorrect forecast scores 1. Lower average scores are better.
Undecided events are excluded from both the average and calibration counts. Missing evidence is not evidence of failure. The table groups probabilities into five ranges; the final range includes 100%. Fewer than five resolved cases in a range receive a visible caution. Even larger samples can be biased by which claims you chose to record.
Watch how the evidence was selected
Confirmation bias can lead us to search for matches and overlook misses. Selection bias can arise when only impressive predictions are recorded. A self-fulfilling prediction changes behavior in ways that help produce the outcome; that does not establish supernatural foreknowledge. Separating these possibilities requires more than a personal score.
Preregister wording, keep every outcome, compare against a simple baseline and be willing to change your view. The exercise is useful whether you enjoy divination or are skeptical of it. Its purpose is to make reasoning inspectable, not to label a person as gullible.
How it works
References
- Brier (1950): Verification of forecasts expressed in terms of probability
- APA Dictionary: confirmation bias
For entertainment and reflection; not scientific prediction or professional advice.