Using Standing Setting in evaluation: Exploring

differences between the ordinary and excellent

Janet Clinton University of Auckland, New Zealand

University of Connecticut 2009

The chocolate chip cookie exercise

Chocolate chip standards of success

Characteristic

Poor Basic Good Excellent

Chewy centreHealth factor

Number of chipsSize

The x factor

Aims

Explore judgment of successIllustrate standard setting as an approach

to systematically judge levels of successDemonstrate that standard setting can be

a powerful tool in program evaluation in either large or small evaluations

Standard setting particularly useful in health settings

What is a standard?

A standard is a statement about

whether a performance is good

enough for a particular purpose

Evaluation is about judgments

The judges

How have we done this in the past?

The use of benchmarks to understand success:

Common practice Developed from literature reviews &

input by stakeholders however Often inaccurate or not representative

The lack of strong evidence and diverse cultures often makes the process of determining standards difficult

Praying……….

What is standard setting?

Standard setting is a means of systematically

harnessing human judgment to define and

defend a set of standards

Standard setting methodology

Describes the attributes of each category e.g., what makes a program fall into the category of excellent

A process of rationally deriving, consistently applying, and describing procedures on which judgments can be made

The process decides on a “cut-score” that then creates distinct and defensible categories e.g., pass/fail, allow/deny, excellent/poor.

Theoretical basis & method

Based on theories of judgment analysis & attribution theory

Used in educational measurement e.g. Jaegar, 1997

Methods◦ Angoff◦ Bookmark◦ Hofstee◦ Lens Modelling

The method has to be:DefensibleCredible Supported by body of

evidence in the literature

Feasible Acceptable to all

stakeholders

All about the process and the right fit

Standard Setting in an Evaluation of

National Chronic Care Management

Initiatives

The project

Understand best practice of chronic care management programs --- nationally.

Develop a user friendly best practice workbook: e.g. COPD, Stroke, CVD & CHF

◦ The evidence---Literature review◦ What is happening---Stock take survey◦ Key stakeholders--Expert interviews◦ Acceptable practice--Standard setting exercise◦ Evaluation of individual programs or sites◦ Development of a best practice work book

Applying the methodology

Conducted Stand Setting Workshops

◦5 sites around NZ ◦Invite experts/stakeholders◦Group and individual rating exercises ◦Analyze the assessments◦Define a standard ◦Determine success indicators

Preparation work

1. Understand the evidence base for success

2. Develop a set of dimensions 3. Develop profiles of programs

4. Plan the workshops

5. Determine the composition of stakeholders

6. Develop assessment rubrics—scoring schedule

Exercise 1

Importance of each factor ◦Rank the importance of the factors

that make up best practice◦Present the information back◦Discuss as a group e.g. missing

dimensions

DIMENSION EXPLANATION

LEVEL OF SUPPORT FOR CHRONIC CARE

Limited Basic Reasonably Good Fully Developed 1 2 3 4 5 6 7 8 9 10

Conceptual understanding of CCM

Appropriate levels of collaboration

Active engagement of leadership

Development of sustainable community links

Focus on health (inequalities)

Decision support systems in place

Appropriate delivery design system

Knowledge transfer

Attention to efficiency/ cost/ output

Attention to effectiveness/ outcomes

Adherence to clinical guidelines

Overall perception of the programme

ID CODE: PROFILE NAME:

Site W

0

2

4

6

8

10

12

Con

cept

ual

unde

rsta

ndin

g of

CC

MPa

tient

Appr

opria

te le

vels

of c

olla

bora

tion

Activ

e en

gage

men

tof

lead

ersh

ip

Appr

opria

tede

velo

pmen

t of

sust

aina

ble

com

mun

ity lin

ks

Focu

s on

hea

lth(in

equa

litie

s)

Dec

isio

n su

ppor

tsy

stem

s in

pla

ce

Appr

opria

te d

eliv

ery

desi

gn s

yste

m

Know

ledg

e tra

nsfe

r

Atte

ntio

n to

effic

ienc

y / c

ost /

outp

ut

Atte

ntio

n to

effe

ctiv

enes

s /

outc

omes

Adhe

renc

e to

clin

ical

gui

delin

es

Ave

rage

Rat

ing

(+1

SD)

Regional Comparison

0

1

2

3

4

5

6

7

8

9

10

11C

once

ptua

lun

ders

tand

ing

ofC

CM

Appr

opria

te le

vels

of c

olla

bora

tion

Activ

e en

gage

men

tof

lead

ersh

ip

Appr

opria

tede

velo

pmen

t of

sust

aina

ble

Focu

s on

hea

lth(in

equa

lities

)

Dec

isio

n su

ppor

tsy

stem

s in

pla

ce

Appr

opria

te d

eliv

ery

desi

gn s

yste

m

Know

ledg

e tra

nsfe

r

Atte

ntio

n to

effic

ienc

y / c

ost /

outp

ut

Atte

ntio

n to

effe

ctiv

enes

s /

outc

omes

Adhe

renc

e to

clin

ical

gui

delin

es

Ave

rage

Rat

ing Rotorua

WhangareiChristchurchAucklandWellington

Regions

The Exercise 2: Round 1

• Form representative groups • Describe and discuss the dimension• Read the profiles & make a judgment about

the standard of the dimensions• Rate the dimensions• Each individual records the reason or

explanation for their choice

DIMENSION EXPLANATION

LEVEL OF SUPPORT FOR CHRONIC CARE

Limited Basic Reasonably Good Fully Developed 1 2 3 4 5 6 7 8 9 10

Conceptual understanding of CCM

Appropriate levels of collaboration

Active engagement of leadership

Development of sustainable community links

Focus on health (inequalities)

Decision support systems in place

Appropriate delivery design system

Knowledge transfer

Attention to efficiency/ cost/ output

Attention to effectiveness/ outcomes

Adherence to clinical guidelines

Overall perception of the programme

ID CODE: PROFILE NAME:

Analyzing the information

Analyze all the information

◦Present the information back to the established groups of mixed stakeholders

◦Individual information was returned

Exercise 2: Round 2

Information returnedGroups discuss their individual

choicesAim to get group consensus on

rating best practice of dimensionsInformation recorded along with

the explanations

DHB1 Self Perceptions - Average Ratings for Dimensions

0123456789

1011

Empo

wer

men

tan

d Se

lf-M

anag

emen

t

Collabo

ratio

n

Lead

ersh

ip

Com

mun

ityLink

ages

Red

ucing

Ineq

ualiti

es

Dec

ision

Supp

ort

Deliver

y:or

ganisa

tion

of h

ealth

care

Deliver

y:pr

ogra

mm

ede

liver

y

Know

ledg

eTr

ansf

er

Effic

ienc

y

Effe

ctiven

ess

Clin

ical

Guide

lines

Roles defined,

attendance at meetings;

program spokes person

No collaborati

on with PHO or NGOs

No plan to disseminate info to community

Results for each profile

Analysis of results

Regression to determine importance of dimensions

Create a matrix of explanations first by individuals then by group consensus

Consult the evidence & the experts Determine the ‘cut scores’ for each

assessment area—used the bookmark method

N Mean sd

Engagement of leadership 479 5.23 1.81Focus on health (inequalities) 479 4.97 2.09Collaboration 478 4.96 1.86Adherence to clinical guidelines 477 4.88 1.95Conceptual understanding 475 4.76 2.00Community links 475 4.67 1.86Attention to effectiveness / outcomes 476 4.65 2.03Attention to efficiency / cost / output 468 4.63 2.01Delivery design system 469 4.40 1.70Decision support systems 478 4.39 1.79Knowledge transfer 476 4.21 1.81

Overall 480 4.72 1.76

Overall means

Factor analysis Individuals Groups

Program & Organization 1 2 3 1 2 3Community links .92 .06 -.18 .69 .06 .06Collaboration .74 .04 .05 .90 -.00 -.07Focus on health inequalities .72 .01 .00 .75 -.10 .17Conceptual understanding .61 -.07 .21 .86 .06 -.08Delivery design system .54 .05 .22 .52 .36 -.01Engagement of leadership .40 .06 .39 .48 .24 .24

Effectiveness/EfficiencyAttention to efficiency -.03 .97 -.03 .04 .91 -.02Attention to effectiveness .10 .73 .12 .15 .79 -.03

InformationDecision support systems .08 .10 .72 .30 .19 .70Adhere to clinical guidelines .05 .21 .59 -.11 .67 .32Knowledge transfer .14 .36 .39 .43 .48 -.00

Factor inter correlationsProgram 1 1Efficiency/Effectiveness .62 1 .68 1Information .66 .68 1 .38 .51 1

Across the exemplars

1 5 10 3 8 6 7 2 4 9 Total1.00

2.00

3.00

4.00

5.00

6.00

7.00

Grp- Final 4 Ind - Final 4 Gp - All Ind - All

Reasonably good

Fully Developed

Limited

Basic

DIMENSION

LEVEL OF SUPPORT FOR CHRONIC CARE

LIMITED BASIC REASONABLYGOOD

FULLY DEVELOPED

1 2 3 4 5 6 7 8 9 10

CONCEPTUAL UNDERSTANDING

COLLABORATIONACTIVE LEADERSHIPCOMMUNITY LINKSFOCUS ON INEQUALITIESDECISION SUPPORTDELIVERY SYSTEMKNOWLEDGE TRANSFER

EFFICIENCY COST/ OUTPUT

EFFECTIVENESS-OUTCOMES

USE OF CLINICAL GUIDELINES

OVERALL

The standard

Dimension Limited Basic Reasonably good

Excellent

Collaboration No work with communityNo evidenceHospital focusPoor referral system

Low engagement with Maori low levels of trustLittle primary/ secondaryintegration

Recognises weaknessesEvidence of partnershipsGood initiatives

Community approach evidencedWhole system collaborationLots of alternatives

Leadership No championPoor managementLack of evidence

Some leadershipNo championsFoundation of leadership

Strong clinical leadershipWeak championsIdentified problems but no change

Evident leadership at programStrong championsEvidence of vision

Did it work?

Analysis illustrates that we can create cut scores for basic through to excellent.

We can define the judgment Have an explanation or criteria for

each level

Reactions to the workshops

◦ We have a voice◦ Systematic◦ Inclusive for roles,

regions & view◦ Useful for self review

◦ Hard work◦ We got nothing◦ This is dangerous◦ Stupid!

Variable

Draw backs

Resource intenseAnalysis can be problematicTime PressuresSetting league tables in healthScary

Consequences

Creates a benchmark for judgmentValidates research through practiceUnderstand different views of successFacilitates self-reviewEncourages learningEncourages an evaluation perspective

Working with Community groups

Adapt the processNeed 8-10 in groupGreat focus of the explanation Qualitative analysisPreparation takes community into

accountUse of evidence to build rubrics &

exemplarsFeasibility-Propriety- Accuracy-Utility

Final word

Not so much◦the METHOD as the PROCESS

The judgment