probability calibration
DESCRIPTION
A presentation from 2008 to a private security conference in the UK.TRANSCRIPT
Phil HugginsPrivate Conference Spring 2008
RiskUncertaintyCardinals versus OrdinalsProbability Estimation AccuracySuppressing UncertaintyFoxes and HedgehogsUncertainty Calibration
A state of uncertaintySet of possible ?negative?
outcomes
A set of possibilities with quantified probabilities and consequences
Lack of complete certaintyMore than one possibilityTrue outcome not known
A set of probabilities assigned to a set of possibilities
Cardinal -> Size Cardinal -> A number or %
Ordinal -> Position in Set Ordinal -> High/Medium/Low or 1/2/3/4/5
Ordinals often have different meanings depending on the person
An estimate of realityThe usefulness of an estimate has
two measures
Calibration ▪ How close to reality is the estimate
Discrimination▪ The range of the estimate
We’re the ‘experts’
Client doesn’t want doubt
Client pushes the risk of the decision onto you
Hitting targets for risk reduction focuses on goals not outcomes
‘The fox knows many things but the hedgehog knows one big thing’ – Isiah Berlin
Hedgehogs• Probability is a deductive exercise• Highly confident in their estimates• Intolerant of uncertainty
Foxes • Probability is an ad-hoc estimation• Low confidence in their estimates• Tolerant of uncertainty
Foxes are generally better calibrated and discriminated
Less ‘foxy’ foxes are slightly worse at discrimination
Best overall at making short term predictions in field of expertise
Hedgehogs worst overall when making long term predictions in field of expertise
Hedgehogs win less but win big
90% Confidence Interval (CI) ‘You are 90% certain the probability of
occurrence is between 60% and 50%’ 95% sure below the upper bound 95% sure above the lower bound
Ask yourself 20 general knowledge questions asking for numerical answers Assign lower and upper bounds to your 90% CI of being
correct Ask yourself 20 true/false questions
Set a value to how certain you are 50%, 60%, 70%, 80% 90%, 100%
General Knowledge You expected to get 9 out of 10 right (90% CI) What did you get? You probably got less than that right
True / False Convert % to decimal (50% = 0.5) Add them up Compare to number of questions you got right If number of correct answers was 2.5 or more
less then you are overconfident
Take multiple tests in successionEquivalence Bets (next slide)Pros and Cons
Consider two pros why you should be confident and two cons why you could be wrong
Avoid Anchoring 95% certain below upper bound 95% certain above lower bound
Two choices:1. Win £1000 if answer falls within your upper and
lower bounds.2. Spin a dial split into 90% and 10% slices. You win
£1000 if it finishes on the 90% slice
Most people (80%) prefer to spin the dial Ask yourself which you would prefer
Preferring 1 is over confident Preferring 2 is under confident
Just pretending to bet money makes you better
After 3 tests 70% of people are perfectly calibrated
After 3 tests 20% of people are significantly improved
10% of people never really get it
Testing has shown that most people relied on to make estimates are in the first two groups already
Thanks for listening
http://blog.blackswansecurity.com