seminar om kpi – anleggsintegritet/prosess-sikkerhet ... · leading vs lagging indictors – is...

OLF pres JEV 12122011 1Risk management research and developmentPreventorUsumacinta (Pemex), nov 2007

Seminar om KPI – Anleggsintegritet/prosess-sikkerhet

Lagging and Leading Key Performance Indicators Definitions

How are they are used within our industry

Jan Erik Vinnem, Preventor AS12.12.2011

OLF pres JEV 12122011 2Risk management research and developmentPreventor

Overview

• Background• Terminology• Alternative perspectives• Criteria for good indicators• Challenges• Recommendations


Brief history of indicators

• Traditionally– Indicators based on occupational injuries– Was proven disastrously misleading by Texas City

explosion• Major hazard risk indicators have received

significant attention after Texas City– Lagging indicators appear to be the main focus in the

process industries• Also the starting point for RNNP in 1999/2000

– Leading & lagging indicators• Total E&P Norge started with major hazard

indicators in 199722

30

21 21

17 17

11 10

18

11 11 1011 11

7

0

5

10

15

20

25

30

35

40

9702 9703 9704 9801 9802 9803 9804 9901 9902 9903 9904 0001 0002 0003 0004

Year/Quarter

Rel

ativ

e ris

k

Overall major hazard risk indicatorfor Frigg Central Complex


Why special emphasis on major hazards?• Rare occurrences (0 fat. after ’85)• For individual installations in NCS

– Worst: 1–2 major accident precursors per year– Best: many years between precursor events

• Causal mechanisms different– Example: is number of leaks < 0.1 kg/s a good

indicator for leaks with escalation potential?• Leaks >0.1 kg/s has escalation potential• Causes of minor leaks often different from >0.1 kg/s

– Root causes may have more in common


Terminology confusion?

• KPI; Key Performance Indicator• Risk indicator• Safety indicator• Major hazard risk indicator

• Asset integrity• Process safety• Major hazard risk

• Indicators– Leading, lagging, proactive, barrier, precursor, output, input

• Focus on definitions and subtle differences may be interesting academically, but rarely very fruitful!

Focus in presentation:

• Major hazard risk is the focus area


-1,0

-0,5

0,0

0,5

1,0

1,5

2005 2006 2007 2008

Rel

ativ

e ov

eral

l bar

rier

avai

labi

lity

Installation 1Installation 2

Important distinction

• On which level are indicators considered?– National level (RNNP)

• National trends, challenges, focus areas, effect of attention, etc.

– Company level (variable)• Benchmarking between companies

– Installation level• Benchmarking between installations• Trends on installations• Problem areas on some installations

0,00,1

0,2

0,3

0,4

0,50,6

0,7

0,80,9

1,0

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 Int01-07

Year

Leak

s pe

r ins

talla

tion

year

s

0

0,1

0,2

0,3

0,4

1 2 3 4 5 6 7 8 9 10 11

Operator

Leak

s pe

r ins

talla

tion

year

> 1 kg/s

0,1 - 1 kg/s


Leading vs Lagging indictors – is there a problem?

• Recent statement:– ”I consider the definitions to be rather vague & unclear,

especially when some lagging indicators also may be leading indicators.

– Some companies consider HC leaks as lagging indicators, (no of leaks), but may also be considered a leading indicator for fire and explosion.”

• Safety Science– Special issue (2009) Process safety indicators (≈10 papers)– Safety Science (2010)

• Vinnem, JE. Risk indicators for major hazards on offshore installations, 48, 770-787

http://www.sciencedirect.com/science/article/pii/S092575351000055X?_alid=1859279708&_rdoc=1&_fmt=high&_origin=search&_docanchor=&_ct=3&_zone=rslt_list_item&md5=098d5800e6c49334c8bcfcb8fe93e65d

http://www.sciencedirect.com/science/journal/09257535/47/4


Leading indicators

• Kjellén:– Leading safety performance indicator is…….

an indicator that changes before the actual risk level has changed

– Consistent with what is commonly used in economy, but significantly different from what other researchers in safety use

– What is now typically called ‘leading indicators’ earlier was called ‘proactive indicators’


Implications

• Indicator based on precursor event occurrences are lagging indicators, according to Kjellén’s definition– Risk level has changed, at least temporarily,

when a precursor event occurs• What constitutes a leading indicator appears

to be more controversial


Further confusion?

• Prof Ben Ale– No indicator for which the values are based on

observations over time can be regarded as leading indicators

• According to this interpretation, none of the indicators used in RNNP would be called ‘leading’


Leading & lagging indicators

• Relativistic approach (Prof Andrew Hopkins)– A specific indicator could be lagging in relation

to one objective, while leading in relation to another

– Example:• Failure at testing is lagging in relation to the

performance of the individual barriers, while leadingin relation to the performance of the overall process safety management system



• Absolutist approach (Hopkins)– Bowtie based

• All indicators to the left of the ‘knot’ leading• All indicators to the right of the ‘knot’ lagging

– Is all ambiguity thus removed?– Certainly not!– Where is the ‘knot’ located in the sequence:

• Error in flange mounting after PSV replacement• Verification not performed due to ‘silent deviation’• Gas leak• Ignition• Failure of active fire protection• Escalation to neighbour module• Escape and evacuation



• Hopkins– Bowtie is not a good way to

distinguish between leading & lagging indicators

– Relativist approach is not good either

– Alternative classifications needed


Alternative perspective – lead vs. lag indicator

• Alternative perspective (HSE)– Input – Throughput – Output – Outcome indicators

• Input– Human or organization error leading to gas leak

• Throughput– Culture affecting the chance that errors are not detected

• Output– Failure of fire detector in test

• Outcome– Failure of fire detector to detect real fire Lagging indicator


Criteria for good indicators – major hazard risk

• Criteria should be strictly tailored to major hazard risk

• Major hazards are special in some senses– Reference made to arguments about frequency– Return period between occurrences of major

hazard precursors• Typical offshore employee• In the range once per 10–15 years


Avoid “garbage in – garbage out”

• Data basis quality is sometimes a challenge– Unambiguity of reporting criteria

• HC leaks– RNNP: > 0.1 kg/s– HSE: no lower limit

• System scope– Process systems– Fuel gas system– Well systems– What about gas injection system, lift gas system???

– Need to focus continuously on improvements• QC, QC, QC, QC……..



• Combination of lagging and leading indicators• Easily observable performance• Intuitive indicators• Not require complex calculations• Not be influenced by campaigns that give

conflicting signals• Reflect hazard mechanisms• Sensitive to change• Show trends• Robust to manipulation• Validity for major hazard risk Vinnem, Safety Science (2010)



• Combination of lagging and leading indicators– Leading often preferred over lagging– Both should be used, for different needs

• Attention more focused if number of indicators is kept at a reasonable level– Max 20–25?



• Easily observable performance– Measurable by applying a recognized data collection

method and scale of measurement– Usually expressed on a ratio scale of measurement

• Hydrocarbon leaks (>0.1 kg/s) per installation year• Hydrocarbon leaks per 1 million manhours• Reporting reliability may be challenge, such as

– Unclear basis for calculation of rate (kg/s)– Both leak & injury– Both leak & environmental spill

– Limiting the data collection to those events with medium and large severity will usually improve the reliability significantly



• Intuitive indicators– What is measured is considered intuitively by

the workforce to be important for the prevention of major accidents

– Dubious intuitivism• Number of years in the same position (e.g. as

process operator) as an organizational major hazard risk indicator for frequency of HC leaks



• Not require complex calculations– Major hazard risk indicators should not require

complex calculations (intuitiveness)– If the number of observations goes down, it

should correspond to an improvement– If very complex calculations are required, the

confidence may be lost



• Not be influenced by campaigns that give conflicting signals– Psychological and organizational reasons could

in many cases result in a too low reporting– Example

• Think of an organizational incentive structure where absence of incidents is rewarded (for instance by a bonus scheme)

• May result in some incidents not reported, as the incentive structure is interpreted as rewarding the absence of reported incidents



• Sensitive to change• Show trends

– Major hazard risk indicator should allow for early warning by capturing changes in a socio-technical system that have significant effects on accident risks

– “Good" set of indicators reflects changes in risk as well as where improvement areas

– Number of observations (incidents, barrier faults, etc) should for lowest level systems be in order of a dozen or so per period (year?)



• Robust to manipulation– The major hazard risk indicators should not

allow the organization to “look good” by for example changing reporting behaviour, rather than making the necessary basic changes that reduce accident risk



• Validity for major hazard risk– Validity is a critical aspect for goodness of indicators

• Is the indicator a valid indicator for the major accident risk? • Is indicator actually measuring what we intend to measure?

– Consider for example the indicator defined by the number of lost time injuries

• Clearly, this indicator says something about accident risk• Accident risk is more than the number of lost time injuries, so

we cannot use just this indicator to conclude on development in the accident risk level as a whole

– Validity of a statement concerning the major accident risk based on observations of the injury rate only, would thus in most cases be low


Recommended approach – major accident risk

• Abandon– Terminology: Lagging & leading indicators

• To be preferred– Precursor based indicators– Barrier (proactive) indicators

• Technical barrier elements• Organizational barrier elements

– Still observing the proposed criteria for good indicators

0

5

10

15

20

25

30

35

40

45

50

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Num

ber of leaks

>10 kg/s

1‐10 kg/s

0.1‐1 kg/s

0,000

0,005

0,010

0,015

0,020

0,025

0,030

0,035

Firedetection

Gasdetection

Total Closure test Leak test Total Closure test Leak test DHSV BDV PSV BOP Deluge valve Fire pumpstart

Riser ESDV Wing & Master valve

Ave

rage

frac

tion of

faults

0

1 000

2 000

3 000

4 000

5 000

6 000

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536

Num

ber o

f hou

rs backlog

PM backlog, hours HES critical PM backlog, hours other equipement



• Precursor based indicators– Will usually function only on a national level– Installation level

• Too infrequent to provide useful feedback• Should after all be used in order to maintain

awareness & focus

0

5

10

15

20

25

30

35

40

45

50

19961997 1998 1999 20002001 2002 2003 200420052006 2007 2008 20092010

MODU

NUI

Complex

FPU

Fixed production

DFU5 Ships on collisioncourse



• Barrier indicators– Important for major hazards in particular

• to focus on indicators that change before the actual risk level has changed

– Will usually function on all levels• National level• Company level• Installation level

0,00

0,05

0,10

0,15

0,20

0,25

D 15

F 27

G 139 H 4 J 8 L 2

M 2

N 4

P 20 Q 4

U 13

V 2

W 6 Y 2

Ø 2

AA 35

AB 27

AC 15

AD 6

AE 46

AF 15

3AH 4

AI 30

AJ 1

2AK 3

AL 11

AM 39

AN14

0AP 8

AQ 18

AR 50

AU 39

AV 7

AW 20

AX 4

AY 8

AZ 22

AÆ 21

AØ 4

AÅ 4

BA 30

BB 12

BC 36

BE 4

BF 61

BG 9

BH 62

BJ 28

BK 42

BL 102

BM 28

BQ 12

BU 4

BV 4

BW 20

BX 2

BZ 2

BØ 2

CA 123

CB 6

CC 10

Fractio

n of fa

ults

2010 Average 2002-10

Riser ESDV

seminar om kpi – anleggsintegritet/prosess-sikkerhet ... · leading vs lagging indictors – is...

Documents