080317 auditing byandrewhale
TRANSCRIPT
Improving Safety Performance Through Measurement
Mike ThomasManaging Director Prof. Andrew Hale
DirectorIOSH London Health& Safety Group
17 March 2008
Overview
Key questions What is the role of performance measurement in a
Safety Management System (SMS)? (“What gets measured gets done”, Drucker and HSE in HSG65)
How can measurement be used to improve performance and drive continual improvement?
How can we select appropriate key performance indicators (KPIs)?
How can we develop meaningful targets? How can we promote performance measurement by
line managers and link this to audits?
A Success Story: Corus NL
Year -3 -2 -1 0 +1 +2
LTI Frequency/1000 man
17.3 16.1 14.3 7.9 0.9 1.9
Reported dangerous situations
186 237 642 1401 1643 1252
Maintenance department of an integrated steel works, employing about 1100 used an intervention driven by KPIs to improve performance.
Improvement driven by KPI’s for managers: • Compulsory + free choice, ‘proactive’ + ‘reactive’.• Targets/KPI + score (1-10 with 6 for hitting target) • KPIs given different weights & totalled for overall score/period• Reported on a ‘dashboard’ every 6 weeks
STOP-GO cards for workforce – dynamic task risk assessment
Key Performance IndicatorsOutput: LTI frequency + Recordable frequency
Compulsory: Reporting of dangerous situations (0.5pppy) Dealing with dangerous situation reports (80%) Toolbox meetings (including safety) (80%)
Choice (at least 1 of 4): Behaviour observation rounds (80%) Safety communication rounds (80%) ‘5 S’ housekeeping inspections (80%) Risk assessment + plan of action (80%)
OHSAS 18001
Continual improvement
Checking and corrective action
OH&S Policy
Planning
Implementationand operation
Management Review
Measurement
Within 18001 “Checking and corrective action”
Includes Measurement of proactive data Measurement of reactive data Internal audit (independent)
Reactive and proactive are key issues
Risk assessment & control
Review & improvement
Risk management system Audit
EvaluationDirect risk controls:
Hard-, soft- & liveware
Primary processes
Inspection
Incident analysis
Risk assessment
Known risk
Comparison
Unknown riskReview RIE system
Types of indicator
Damage, injury, loss Precursors – for each scenario, different
precursors (incidents, leaks, breakdowns)
Re-design
Choice and design of (sub)system
Normal situation
with in-built hazards
Recovery
Deviations from normal situation
Loss of control(release of energy
+ exposure
Transmission
Damage process
Stabilisation
Elimination of hazard
Hazard control measures
Detection & recovery
Escape
Secondary safety
Rescue, damage limitation, treatment
Reporting
Learning
Choice & design of prevention &
control measures
Accident Deviation Model
For each hazard, different deviation scenarios, with different control measures, failing in different ways
LTI’s don’t predict disasters
The lesson of Texas City: BP became distracted from efforts to control
process safety because they were lulled into a false sense of security by their very excellent control of lost time injuries.
The fallacy of Bird’s interpretation of Heinrich’s triangle
Heinrich’s thesis
Reasoning from top down
Each scenario or accident type apart
Average ratio over scenarios
1
29
300
Disabling Injury
Minor Injury
No injury
From Bird onwards
Overall statistics of a department, factory, or country
Different categories of severity
1
100
500
Property damage
(Bird 1966)
Disabling Injury
Minor Injury
Types of indicator
Damage, injury, loss Precursors – for each scenario, different
precursors (incidents, leaks, breakdowns) Presence & readiness of risk control measures
(hardware, software, people’s behaviour) Management system processes to deliver the
preventive measures
Resources and controls in the SMS
- Competence of staff- Availability of manpower- Commitment to safe operations & conflict resolution- Communication within and between teams- Procedures, goals & rules- Hardware & software
- Selection & training- Manpower planning- Incentives, supervision, appraisal, culture, management priority- Handover, briefing, communication channels- Task/policy analysis & design- Design, layout, maintenance
These processes are the ones to be audited
Problems with Auditing
Seen as a panacea “Bolted on” – imposed from outside May not match policy & own management
system: independent of management Passive process for manager “Economical with the truth” Wait for next audit No ownership
Types of indicator
Damage, injury, loss Precursors – for each scenario, different
precursors (incidents, leaks, breakdowns) Presence & condition of risk control measures
(hardware, software, people’s behaviour) Management system processes to deliver the
preventive measures Attitudes and values (culture) to use the SMS
processes and the risk control measures
Safety climate/culture measurement
Existing climate questionnaires: limited validation (TRIPOD is one exception) Interpretation is still more an art than a science – lack
of clarity over underlying models of culture Gap between measuring (profile) and improving
Safety culture maturity scales (Hearts & Minds): Similar criticisms Individual completion followed by group discussion
generates much useful debate
Improving Performance
Once you are clear on meaning & objectives Look at measurement and improvement
Three key issues How to define performance in measurable terms? How can performance be measured? How to extend to target setting and to drive improved
performance?
How is Improvement Achieved?
“What gets measured gets done” Apply sound management techniques to safety Performance standards - what people must do
Who is responsible? – whose KPI? What are they responsible for? – tasks, processes When should the work be done? - plans What is the expected result? – (intermediate) outputs
Set targets and measure performance in terms meaningful to each individual
Criteria for performance measures
Validity Reliability Representativeness Sensitivity Openness to bias Cost-effectiveness
NB. Exposure measures
Typical targets in the past
“eliminate lost time (or all) accidents” “reduce all accidents to n or by x% per annum” “reduce number of days lost to n or by x%” “reduce cost of claims or other losses by y%” “eliminate notices as a result of enforcement action”
Obvious approach = what we want! Commendable, but typically, reactive
Accidents as performance measure Validity Reliability
Representativeness Sensitivity Reporting bias Cost-effectiveness PM Exposure measure
Very high High, but absence
serious Good if enough Low for good organisation Strong for minor High cost of analysis /man or /manhour
Problems with Reactive Targets
Problems - well recognised? Not helpful to managers!
They accept the value and sense but … What must they do?
“Promotes” under-reporting and “manipulation” of outcome
What if zero accidents already? Past performance may be a poor predictor of the
future
Accident Rates
"A low accident or incident rate over a period of years is no guarantee that risks are being effectively controlled.This is particularly so in organisations involved in major hazard activities, where the probability of an accident may be low but where the consequences could be extremely serious.In this type of organisation, the historical incidence of reported accidents alone can be an unreliable indicator of safety or environmental performance and can lead to complacency."
Guidance on COMAH Regulations, UK
Reactive Targets
Why still set? Easy option that sounds good! Mirrors UK Government’s targets Accident reduction IS the desired output of the SMS
Are Reactive “Targets” Still Valid? At a high level within the organisation as “aim” or
“vision” But NOT as targets for managers Need to move to better data: step 1
Minor injuries and health effects Near misses (surprises) or dangerous situations
Way Forward?
Aim Reduce accidents but numbers of accidents are NOT
used as exclusive targets for managers
Move to proactive measures Set targets on proactive measures of performance
Compare with quality management
Contrast: Safety and Quality
ISO approach to quality as: An essential feature Not an optional extra
Emphasise: managing quality in, not inspecting defects out get the management processes right, but still measure defects
Apply this philosophy to safety
What is Proactive Measurement?
Traditional measures (input measures): e.g. PPE being used Guards in place Documents reviewed and updated All employees trained Toolbox meetings held Maintenance conducted to plan
More dynamic measures (input + intermediate) Hazards identified and put right Competence tested & used Behaviour observed, discussed & improved
Planning and Target Setting
If you accept the philosophy, how can measures be translated into targets?
BS8800 approach Annex C (BS 8800:1996) Simplified version (BS 8800:2004, Annex D)
Planning and Implementing
Planning for Safety
Measure Baseline Data
Devise Plan(Management Programme)
Topics withQuestions as Targets
IMPLEMENT PLAN
Aim
Objective
Define Outcome Indicator
Measuring Success
IMPLEMENT PLAN
Measure Outcome Data
Achieving Objective ? No
Yes
Take Corrective Action as Necessary
Continue Process
Measure Compliance with Programme (Targets)
Meeting Targets ?
Yes
Combining Data Compliance Monitoring Data
(Management Programme)
Outcome Data (Objective)
Improving Not Improving
Improving Not Improving
Why questions posed in plan?
Forces clear logical thought Define the programme Define the targets
Questions are a blue print for the managers Questions can be used to measure success
“Correct” answer is “yes” Link to audit (e.g. CHASE)
Proactive Monitoring and Audit
HASTAM’s CHASE system Questions developed by planning process Used by managers to measure performance Used by specialists for audit
Audit by verification of Managers’ answers Recommended remedial action
Reinforces ownership by managers
Conclusion
Explained Need for and effective means of setting realistic
targets Targets NOT based on “reactive” measures Based on proactive measures of system
However, no conflict with reactive aims Targets provide
Clear implementation mechanism for managers Effective measurement of improved performance
