using inspection information to identify and ameliorate risks - andrew robinson

Using Inspection Information to Identify andAmeliorate Risks,

and Monitor Performance of Risk Management

Andrew Robinson

CEBRA, University of Melbourne

September 12, 2013

Outline

Background

Measuring Performance

Reporting

Risk–Based Approach to Inspection

Lessons Learned

Background

Quarantine is Expensive

Program Annual Revenue1

Airports $79,810,000Import Clearance $112,862,000International Mail $22,100,000... ...

TOTAL $376,165,000

Why (and how) do we spend all this money?

1DAFF 2007–8 Figures, Table 4, Beale et al. (2008).

Terror on the High Seas

Picture: AP Source: AP

http://www.news.com.au/lifestyle/home/

panic-as-giant-african-land-snail-invades-us-state-of-florida/

story-fneuz6rh-1226620722434

Giant African Snail

I Achatina fulica

I Up to 30 cm long

I Up to 1 kg in weight

I Omnivorous (eats 500 species of plants)

I Up to 9 years lifespan

I Hermaphroditic reproduction

I Up to 1200 eggs p.a. after matingI In the 1960s, 3 snails accidentally released in FL.

I 10 years and a million dollars to catch 18,000 snails.

I First recorded in American Samoa in the mid–1970sI a million snails were collected by hand in 1977I more than 26 million snails were collected over the following

three years.

ECIR & RDI

International Vessels

ULD

Plant–Product Pathways

Measuring Performance

Good Questions.

1. How risky is pathway X ?

2. How much effort are we investing in our monitoring of X ?

3. How appropriately are we focusing our effort?

4. How useful is our intervention?

5. How can we (better) reduce the risk?

Performance Indicator Qualities.

A Performance Indicator is simply a Statistic.

A good set of performance indicators . . .

1. is easy to interpret;

2. does not mislead the decision-maker;

3. can be computed using readily-available data; and

4. provides an appropriate measure of uncertainty.

Pathway

Examine? ExamineNot

Compliant

Compliant

LeakageSurvey?

ExamineNot

Compliant

Released Compliant

v

i

Rectification

bv − i

v

n

v − n

i− b

b

n− y

y

Rectification

y

Figure 1: Flow chart for sampled intervention of pathway with leakage survey. Rectifica-tion means that the BRM is captured, and the mail item is then assumed to be compliant.The leakage survey records whether the unit was released or inspected after screening.

3

Performance Indicator Examples.

I Leakage (Slippage)

I Before-intervention compliance rate (BIC),

I Post-intervention compliance rate (PIC),

I Effectiveness (E, for intervention), and

I Hit Rate (HR, or Odds Ratio, for screening / profiling).

Needs to Know

In order to measure passenger cohort risk, we need to know

I v , the volume, which is the number of units on the pathway;

I i , the number of units inspected after screening;

I b, the number of units that were non-compliant;

I n, the number of units processed in the leakage survey; and

I y , the number of units that were found to be non-compliantin the leakage survey.

NB: n and y may have subscripts i for inspected and r for released.

Intervention Leakage Count l : Point Estimate

l = i × yini

+ (v − i)× yrnr

(1)

A Worked Example With Completely Fabricated Data

> v = 10000

> i = 3000

> b = 30

> n_i = 100

> y_i = 5

> n_r = 300

> y_r = 5

The estimated leakage count is

> (l.hat = i * (y_i / n_i) + (v - i) * (y_r / n_r))

[1] 266.6667

Intervention Leakage Count l : Interval Estimate

Let pi = (yi + 1)/(ni + 2) and pr = (yr + 1)/(nr + 2), and

sl =

√i2 × pi × (1− pi)

ni + 2+ (v − i)2 × pr × (1− pr )

nr + 2(2)

Then the interval estimate for the leakage count is

lI = i × pi + (v − i)× pr ± 1.96× sl (3)

Estimated Leakage Count

The standard error of this estimated leakage count is computed by

> p_i = (y_i + 1) / (n_i + 2)

> p_r = (y_r + 1) / (n_r + 2)

> (s_l = sqrt(i^2 * p_i * (1 - p_i) / (n_i + 2) +

+ (v - i)^2 * p_r * (1 - p_r) / (n_r + 2)))

[1] 89.69113

The 95% confidence interval estimate for the leakage count is

> (l.int = i * p_i + (v - i) * p_r +

+ c(-1,1) * 1.96 * s_l)

[1] 139.7488 491.3380

Estimated Approach Count a: Point and Interval

a = b + l (4)

aI = b + lI (5)

The estimated approach count is

> (a.hat = b + l.hat)

[1] 296.6667

and the interval estimate of the approach count is

> (a.int = b + l.int)

[1] 169.7488 521.3380

Before-Intervention Compliance

BIC =v − a

v(6)

BICI =v − aI

v(7)

Then BIC is

> (BIC = (v - a.hat) / v)

[1] 0.9703333

and the interval estimate of BIC is

> (BIC.int = (v - a.int) / v)[2:1]

[1] 0.9478662 0.9830251

Post-Intervention Compliance

PIC =v − l + yi + yr

v(8)

PICI =v − lI + yi + yr

v(9)

Then PIC is

> (PIC.hat = (v - l.hat + y_r + y_i) / v)

[1] 0.9743333

and its interval estimate is

> (PIC.int = (v - l.int + y_r + y_i) / v)[2:1]

[1] 0.9518662 0.9870251

NCE (Inspection)

NCE =b

a(10)

NCEI =b

aI(11)

The E of the inspection, expressed as a percentage, is

> (NCE.insp.hat = b / a.hat) * 100

[1] 10.11236

with interval estimate

> (NCE.insp.hat = b / a.int)[2:1] * 100

[1] 5.754424 17.673170

Hit Rate

HR =b + lii

(12)

Let

li = i × yini

(13)

> (HR.hat = (b + i * (y_i / n_i)) / i)

[1] 0.06

Hit Rate

An estimate of the SE is

sli = i ×√

(yi + 2)× (ni − yi + 2)

(ni + 4)3(14)

then

liI = i × yi + 2

ni + 4± 1.96× sli (15)

following Agresti and Coull (1998), and

HRI =b + liI

i(16)

Hit Rate

> (s_hat_li = i * sqrt((y_i + 2) * (n_i - y_i + 2) /

+ (n_i + 4)^3))

[1] 73.70656

> (l_i.hat = i * (y_i + 2) / (n_i + 4) +

+ c(-1,1) * 1.96 * s_hat_li)

[1] 57.45822 346.38793

> (HR_I = (b + l_i.hat) / i)

[1] 0.02915274 0.12546264

Challenge 1

In some systems, we don’t know i so we estimate it using n,∑

i ,and raking.

Cohort A & R X-Ray Manual Total

A a1 a2 a3∑

N AB b1 b2 b3

∑N B

C c1 c2 c3∑

N C

Total∑

N AR∑

N X∑

N M∑

Raked up to 4 dimensions — cohort, port, declaration, screeningmethod.

Challenge 2

n is small so p = yn

is highly variable, and sometimes i < b!

I Smooth i using a ridge.

I Smooth p using Empirical Bayes methods.

Bias–variance trade-off (Could also create larger groups).

Empirical Bayes

L(α, β; x ,n) =

k∑i=1

− ln Γ(α+ xi)− ln Γ(β + ni − xi)

+ ln Γ(α+ β + ni)

+ ln Γ(α) + ln Γ(β)− ln Γ(α+ β) (17)

pi =xi + α

ni + α+ β(18)

Comparing Profiles: Leakage curves and ROC

DAFF Biosecurity: International Passenger Profiling

●

●

●

●

0 500 1000 1500 2000

0.0

0.5

1.0

1.5

Risk

Effort (passengers)

Leak

age

(%)

●

●

●

●

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

AUC: 0.65

False Positives

True

Pos

itive

s

Figure 3.1: Example graphical assessment of profiling strategy for three categories of passengers,as reported in Table 3.1. The left panel shows the leakage curve, and the right panel shows theROC curve, and reports the AUC in the panel title. The solid line is the 1:1 line, representing arandom allocation of passengers to risk categories.

the overall operational outcome. In particular, the higher volume, lower risk segments are goingto be difficult to profile. Profiling has its operational limits, and is a complement to the other190

interventions.2

3.3.2 Balance

Here we refer to balance in the informal sense that the profiling will be more likely to providepredictions with desirable statistical properties if the categories into which the passengers aredivided are, more or less, the same size. The counts of citizenships of international passengers are195

quite variable; the best-represented is Australia with nearly 6 million arrivals, and a number ofcategories are very small, for example, it appears that there was only one arrival from AmericanSamoa during the data collection period. It would be straightforward to set a lower limit forcategory sizes, and merge all categories that are smaller than that lower limit, however in doingso it is possible that some small but useful efficiencies could be missed. For our purposes we200

retained as many of the categories as we had data for, and used smoothing statistics (specifically,the empirical Bayes procedure, see Appendix C) to provide some robustness for the estimatesof the smaller categories.

3.3.3 Monitoring

As noted in earlier ACERA reports ( e.g. Robinson et al., 2008), all pathways should be inspected205

at some level, even if that level of inspection is purely for the purposes of monitoring the pathwayto ensure that it remains within operationally acceptable limits. This monitoring does not referspecifically to the use of a leakage survey, but rather to a random infrequent inspection ofpathways that are deemed likely to be compliant. It may be convenient for this monitoring tobe performed as part of the leakage survey, as is done presently.210

The consequence of this monitoring is that the leakage curve will not quite reflect the trueamount of effort required to manage the pathways, because the monitoring component is not

2Comment 16: Text from CW.

12

Reporting

Reporting: PIC by Port, Channel, and Declaration

A/R K9 Manual Xray●

●

● ● ●

●

● ●

● ● ● ● ●●

● ●

● ● ● ● ●

●

●●

● ●●

● ●● ● ●

● ●●

● ●● ● ●

●

●

●

● ● ● ●●

●●

●

● ● ●●

●

● ● ●●

●

● ● ●

92.5

95.0

97.5

100.0

92.5

95.0

97.5

100.0

Declarant

Non−

Declarant

Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8

Port

PIC

(%

)

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

BALAOFKFNJJABFFEMHCCNFQCMG

PIKANBCHKDKCLEGEHAGAOCBJAJ

MBMI

DDOBCIDJJEBBQGKGABLHRI

KHQHGH

FIREOABEHJ

90.0 92.5 95.0 97.5 100.0BIC (%)

Citi

zens

hip

Lessons Learned

I Start small — solve case studies.

I Visit.

I Operationalise–Light.

I Sustain engagement.

I Be patient.

I Deliver useful outcomes.

I Build bridges inside and outside.

Discussion and Questions

Background

Measuring Performance

Reporting

Risk–Based Approach to Inspection

Lessons Learned