Assessment of Agreement between SkyBit Predictions and

On-site Measurements

Henry K. Ngugi, PhD.

Penn State Fruit Research & Extension Center, Biglerville, PA.

Quantitative Epidemiology and Commercial Fruit Production

Scientist

Industry Orchardist

Adjudicate between the grower and industry to maintain a commercially viable tree fruit industry in PA

Diseases of concern in the mid-Atlantic region

• SkyBit---Marketed by ZedX Inc.

• Growers pay ~$60 per month for pest management information

• Fire blight and apple scab models developed by Dr. James W. Travis (Penn State Univ.)

• Models were never validated

• How do SkyBit forecasts perform?

Assessment of Agreement Between Forecasts

Adjudicating: The SkyBit Example

How do SkyBit weather data predictions compare with data

collected on-site?

Gold standard = Campbell Scientific and Spectrum Technologies weather stations at PSU-FREC, Biglerville, PA

• How do SkyBit data predictions compare with data collected on-site?

• Perform statistical agreement tests between data collected on-site and SkyBit predictions

• Lin’s concordance analysis for continuous variables

• Limits of agreement statistics for continuous variables

• Concordance tests for categorical variables

Reliability of SkyBit Disease Forecasts

Agreement in weather data

I. Temperature

SkyBit mean temperature (oC)

0 5 10 15 20 25 30S

pect

rum

mea

n te

mpe

ratu

re (

oC

)0

5

10

15

20

25

30

SkyBit temperature (oC)

0 5 10 15 20 25 30

Cam

pbel

l Sci

. te

mpe

ratu

re(o

C)

0

5

10

15

20

25

30

r = 0.988ρ = 0.987

r = 0.992ρ = 0.991

Highly significant agreement between SkyBit data and on-site daily mean temperature measurements

Shapiro-Wilk test W = 0.986, P = 0.223 i.e., cannot reject normality hypothesis

Differences must be normally distributed to compute limits of

agreement

Confidence limits agreement

Mean of SkyBit and Spectrum (oC)

5 10 15 20 25

Diff

eren

ce (

Spe

ctru

m-S

kyB

it; o

C)

-3

-2

-1

0

1

2

3Over a 4-month period, only in 4 out of 152 days did the SkyBit measurements significantly differ from on-site temp. measurements

95CL = d 1.96SD, in this case = -1.82 to 1.65

Rainfall data: April to June 2009

SkyBit (mm)

0 10 20 30 40

Nat

iona

l Wea

ther

Ser

vice

(m

m)

0

10

20

30

40

B

Good agreement with on-site data from the Spectrum Tech. weather station but SkyBit underestimates the rainfall amounts

SkyBit (mm)

0 10 20 30 40

Spe

ctru

m (

mm

)

0

10

20

30

40 A

No agreement with data from National Weather Service

r = 0.882ρ = 0.875

r = 0.405ρ = 0.403

Spectrum Tech. data Lin’s concordance coefficient, ρ = 0.860; r = 0.876

Campbell Sci. data Lin’s concordance coefficient, ρ = 0.746; r = 0.788

SkyBit (wetness hours)

0 5 10 15 20 25

Spe

ctru

m (

wet

ness

hou

rs)

0

5

10

15

20

25

SkyBit (wetness hours)

0 5 10 15 20 25

Cam

pbel

l Sci

. (w

etne

ss h

ours

)

0

5

10

15

20

25

Wetness hours: April to June 2009

Wetness confidence limits analysis

Mean Cambell & SkyBit (hrs)

0 5 10 15 20 25

Diff

eren

ce (

Cam

pb.

- S

kyB

it; h

rs)

-15

-10

-5

0

5

10

15

Generally good agreement but SkyBit over-estimates in wet days and underestimates in dry

Good agreement between the ‘gold standards’

Campbell Sci. (wetness hours)

0 5 10 15 20 25

Spe

ctru

m (

wet

ness

hou

rs)

0

5

10

15

20

25

r = 0.929ρ = 0.919

Test for agreement between fire blight predictions (MaryBlyt and

SkyBit)Concordance coefficients and McNemar's test statistics for agreement between SkyBit and MaryBlyt fire blight forecasts for Biglerville, PA April and May, 2009 Coefficient Estimate SEx RemarkKendall’s tau-B 0.607 0.103 ModerateStuart’s tau-C 0.482 0.112 No concordanceCohen's Kappa 0.570 0.115 ModeratePearson’s correlation 0.607 0.115 Weak

McNemar's testy χ2 = 6.40 P = 0.014; df = 1; n = 55xAsymptotic standard erroryFor the null hypothesis that the two data sets disagree

SkyBit predictions are more cautious

Concordance coefficients and McNemar's test statistics for agreement between SkyBit and MaryBlyt fire blight forecasts for Biglerville, PA April and May, 2009 Coefficient Estimate SEx RemarkKendall’s tau-B 0.884 0.065 High Concordance Stuart's tau-C 0.826 0.086 High concordanceCohen's Kappa 0.883 0.065 High concordancePearson’s correlation 0.884 0.064 Strong

McNemar's testy χ2 = 0.333 P = 0.564; df = 1; n = 55xAsymptotic standard erroryFor the null hypothesis that the two data sets are disagree

When MaryBlight “H” is counted as “I”

Disease forecast assessments• No agreement between SkyBit and MaryBlyt

forecasts (χ2 = 6.4; P < 0.011; McNemar’s test) for infection events

• Agreement only when MaryBlyt ‘H’ is counted as = ‘I’ (χ2 = 0.333; P = 0.564)

• Good agreement in apple scab forecast (χ2 = 2.0; P = 0.15 and χ2 = 3.6; P = 0.058 for Spectrum and Mill table models, respectively)

Summary of Results

• SkyBit delivers reliable data to growers in the mid-Atlantic region Temperature measurements highly reliable Underestimates rainfall amounts Wetness measurements unreliable in very dry

or wet conditions

• SkyBit forecasts for apple scab are as reliable as Mill’s Table or Spectrum model

• SkyBit fire blight predictions are conservative relative to those of MaryBlyt

Acknowledgements

People in Ngugi Lab

$$$ FOR Ngugi Lab

USDA –CSREES

SHAP, PSU-CASDrs. Jim Travis & N.O. HalbrendtDrs. Jim Travis & N.O. Halbrendt