spotlight webinar: evaluability assessments in public health

Follow us @nccmt Suivez-nous @ccnmo

Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The

views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..

Evaluability Assessments in Public Health

Presenters:

Laura C. Leviton, PhD

Laura Kettel Khan, PhD

November 10, 2017 1:00 – 2:30 PM ET

Follow us @nccmt Suivez-nous @ccnmo2

Housekeeping

Use Chat to post comments and/or

questions during the webinar

• ‘Send’ questions to All (not

privately to ‘Host’)

Connection issues

• Recommend using a wired

Internet connection (vs.

wireless),

• WebEx 24/7 help line

• 1-866-229-3239

Participant Side

Panel in WebEx

Chat


After Today

The PowerPoint presentation (in English and

French) and English audio recording will be made

available.

These resources will be available at:

http://www.nccmt.ca/previous-webinars

http://www.nccmt.ca/previous-webinars


How many people are watching

today’s session with you?

Poll Question #1

A. Just me

B. 1-3

C. 4-5

D. 6-10

E. >10


Your profession? Put a √ on your answer (or RSVP via email)

/

Epidemiologist Management (director,

supervisor, etc.)

Allied health

professionals (nurse,

dietician, dental

hygenist, etc.)

Librarian Physician / Dentist Other

5


Evaluability assessments in

public healthhttp://www.nccmt.ca/resources/search/290

Episode 35

6

http://www.nccmt.ca/resources/search/290

NCC

Infectious

DiseasesWinnipeg, MB

NCC

Methods

and ToolsHamilton, ON

NCC Healthy

Public PolicyMontreal, QC

NCC

Determinants

of HealthAntigonish, NS

NCC

Aboriginal

HealthPrince George, BC

NCC

Environmental

HealthVancouver, BC

7

http://www.nccid.ca/en/home

http://www.nccid.ca/en/home

http://www.nccmt.ca/


http://www.healthypublicpolicy.ca/2/Home.htm

http://www.healthypublicpolicy.ca/2/Home.htm

http://www.nccdh.ca/

http://www.nccdh.ca/

http://www.nccah-ccnsa.ca/

http://www.nccah-ccnsa.ca/

http://www.ncceh.ca/

http://www.ncceh.ca/

Registry of Methods and Tools

Online Learning

Opportunities

WorkshopsMultimedia

Public Health+

Networking and

Outreach

NCCMT Products and Services

8


Poll Question #2

How familiar are you with the

method or tool we are discussing

today?

A. I am not familiar with the method or tool

B. I have heard of the method or tool

C. I have used the method or tool


Presenters

Dr. Laura C. Leviton, PhD

Senior Advisor for Evaluation

Robert Wood Johnson Foundation

USA

■ Summative evaluation:

o Designing and using evaluation to judge merit

■ Formative evaluation:

o Designing and using evaluation to improve intervention

■ Evaluability assessment: Assessing whether

o the intervention is ready to be managed for results

o what changes are needed to do so

o whether evaluation would contribute to improved performance

Definitions


Resources

• Wholey, Hatry & Newcomer, Handbook of Practical Program

Evaluation, Wiley 2010

• Leviton et al., Evaluability assessment to improve public

health. In Annual Review of Public Health, Volume 31:213-

234.

• Leviton, Kettel Khan & Dawkins, New Directions in Evaluation,

No. 125, January 2010-- chapter 3 has templates, procedures


Evaluations are Often Handed to Us

“Here, evaluate this.”


Some of the Interventions Just Aren’t Very Good

“Here, evaluate this.”


Why?

• Perhaps 95% of interventions are not fully developed at the

time of evaluation.

• Also, ongoing problems with measurement, design and

analysis.

• Wilson and Lipsey, 2001 review of 319 meta-analyses:

• Proportion of effect sizes associated with study features

• Variance accounted for:

Study methods = features of the intervention

Biggest sources: research design, operationalizing the dependent variable, sampling

error


Don’t Rush to Summative Evaluation


Steps in Evaluability Assessment

1. Involve intended users of evaluation information

2. Clarify the intended intervention

3. Explore intervention reality

4. Reach agreement on needed changes in activities or goals

5. Explore alternative evaluation designs

6. Agree on evaluation priorities and intended uses of information.


It’s Cyclic,

Not Linear

Involve end users of

evaluation

Determine scope of project

Review program

documents

Consult stakeholders

Agree-ment on goals?

no

no Create / revise logic model or

theory of change

Agree-ment on model?

yes

Feedback to program manager or policy maker

Interview staff; “scout” the

program reality

yes

Report on assessment of: Plausibility Areas for program development Evaluation feasibility Options for further evaluation Critique of data (quality,

availability)

Develop Program

Stop


Early Steps

Involve end users of

evaluation

Determine scope of project

Review program

documents



no


Stop


Middle Steps

Review program

documents



no


theory of change


yes


Develop Program

Stop


Logic Models

Inputs Outputs Outcomes (Impact)

Resources

&

Staff

Activities,

Outreach to

Target Group

Products,

Target Group

Participation

Achieve

Short Term

Objectives

Achieve

Intermediate

Objectives

Achieve

Long Term

Objectives


Theory of Change

Goals Strategies Target Group If-Then

Statements

Short Term

Outcomes

Long Term

Outcomes

Statements How goal will

be

accomplished

Define group If (activity)

then outcome

Measurement Was strategy

achieved?

Were

participants in

the target

group?

Did the

activity result

in the

outcome?

Was short

term objective

achieved?

Was long term

objective

achieved?

Data Sources E.g., program

records

E.g., survey of

group


Later Steps


theory of change


Interview staff; “scout” the

program reality

yes

Report on assessment of: Plausibility Areas for program development Evaluation feasibility Options for further evaluation Critique of data (quality,

availability)


What Now?

• Intervention development

• Data collection to inform improvements

• When to do summative (outcome) studies:

• Logic model or TOC is sharpened and agreed to

• The model looks like the reality, and vice versa

• It’s plausible to achieve the outcome(s)

• Formative evaluation indicates intermediate steps are being

accomplished.


Presenters

25

Dr. Laura Kettel Khan, PhD

Senior Scientist and Advisor

Centers for Disease Control and

Prevention

USA




Systematic Screening and

Assessment: Past & Present

Examples from the Field


Overview

•4 Examples from the Field

• Community Obesity Prevention – New York City Child

Care

• Healthy eating & Active living

• Hypertension Control

• Traumatic Brain Injury

•Coming full circle with SSA

• Childhood Obesity Declines

•Benefit for Public Health

27


Obesity Prevention: NYC Child Care

Policies

Convene a panel of experts to identify and review

potential programs and policies

Assess programs and policies’ readiness for evaluation

Synthesize findings and share promising practices with the

field

Develop a network of professionals with the skills to

conduct evaluability assessments

Inform funders of programs and policies ready for

evaluation28


Systematic Screening & Assessment

29

Inputs Steps Products

Guidance 1. CHOOSE priorities

2. SCAN environmental

interventions

3. REVIEW AND IDENTIFY

INTERVENTIONS that warrant

evaluability assessment

4. EVALUABILITY

ASSESSMENTS of priority

interventions

5. REVIEW & RATE for

promise/ readiness for eval

6. USE information

7. SYNTHESIZE what is known

Focus

Brief descriptions

Constructive feedback

Plan for rigorous

evaluation

List of

interventions

Nominations, existing

inventories, descriptions

Communicate with all

stakeholders

Expert review panel

Report of intervention and

evaluation issues

Ratings and Reports

List of Interventions

Brief DescriptionsNominations, existing

inventories, and

descriptions

Expert review panel

Network of

practitioners/research

Communicate with all

stakeholders

Expert review panel


Systematic Screening

•Criteria expert panel used for screening:

– Potential impact

– Innovativeness

– Reach

– Acceptability to stakeholders

– Feasibility of implementation

– Feasibility of adoption

– Sustainability

– Generalizability/transportability

– Staff/organization capacity for evaluation

30


Systematic Screening & Assessment

Results

Nominations

Met Inclusion

Criteria

Selected for

EA

After School/Day Care 167 61 23

Food Access 84 34 18

School District Local

Wellness Policies146 58 6

Comprehensive Physical

Activity39 7 2

Built Environment for

Physical Activity 22 14 4

TOTAL 458 174 53

31


Evaluability Assessment

Review of documents• Draft logic model

2-3 day site visit• Interviews: program description, logic model,

staffing, funding, sustainability, evaluation activities

• Observations

• TA /debriefing session

Reports and recommendations

Follow-up TA call with CDC experts

32


Readiness for Evaluation

•Review of site visit reports identified classifications:

1. Ready for stand-alone, outcome evaluation

2. Appropriate for cluster evaluation

3. Theoretically sound but need further development

4. Technical assistance needed in specific areas

33


Big Pay Off: NYC Child Care Policy

Evaluation

NYC child care regulations on nutrition, physical

activity & screen time

Positive evaluation of implementation and

outcome

Large sample of centers in low-income urban

neighborhood population

Outcome verified with direct observation &

accelerometry

Reference study for national recommendations for

center policy and practice

34


CDC: Healthy Eating & Active Living

Identify innovative policies and strategies

related to public property shared & open Use

Schools, swimming pools, parks

Opportunity to identify promising practices

feasible for state and local implementation

Focused on interventions for communities with

fewer resources

Factsheets on successful policies and

strategies

35


CDC: Hypertension Control

7 HTN Control project using SSA &/or SSA and

new Enhanced Evaluability Assessment (EEA)

Depending on the topic area and amount of

evidence, both methodologies are used.

EEA is an adaptation of SSA which permits

rapid assessment & reports effectiveness of

intervention JL Losby, et al. Arriving at Results Efficiently: Using the Enhanced

Evaluability Assessment Approach. Prev Chronic Dis 2015;12:150413.

DOI: http://dx.doi.org/10.5888/pcd12.150413

36


CDC: Return to School post-Traumatic

Brain Injury

Best practices of Return to School after TBI

Students who sustain TBI need post-injury

supports at school

Families have difficulties accessing resources

What is the best program model or components

for understanding outcomes

Scalability and applicability for all 50 states

Improving identification & management of TBI in

healthcare systems & schools

37


Childhood Obesity Declines

A retrospective use of SSA…. Is the evaluation valid? Is the BMI/outcome data

correct?

If so, what factors could have played a role?

Same process, different questions for site visit

Used SSA as a confirmatory process for outcome and an exploratory process for what happened.

4 sites, similar patterns of interventions and programs yields anticipated outcome

38


It’s a Process…..

39

1. Choose priorities for the scan

2. Scan environmental programs & policies

3. Review and identify those that warrant evaluability assessment

4. Evaluability assessment of programs & policies

5. Review and rate for promise and readiness for evaluation

6. Use Information:

• Position for rigorous evaluation

• Feedback to innovators

• Cross-site synthesis


Why SSA adds Value…..

What is the same?• Review documents

• Discuss with

stakeholders

• Develop logic model

• Iterate the process

• Determine what can

be evaluated

What is different?• EA as one component of a

process of discovery

• SSA explicitly provides

feedback to innovators

• SSA provided insights on

clusters of projects

• SSA helped to identify

policies and programs

worthy of further

attention.

40


Finding the best bet faster…..

Of 458 innovations nominated in Year I:

• 174 met criteria for inclusion;

o 53 were selected for evaluability assessments;

– 27 were ready for stand alone evaluation.

• Yet all of the nominations were viewed as

important by stakeholders.

• If all of them underwent evaluation,

o there would be a 5% chance of encountering something

with even a chance of concluding success!

41


AND finding it cheaper…..

Without a systematic process,

one would need to conduct at least 20 evaluations to discover 1 that might be successful.

The process is cost-effective for and decision makers.

It reduces uncertainty about investments.

42


Experts & practitioners benefit….

Expert and practice “Innovators” found the

process a learning experience.

Learned to value interventions outside their

area of expertise

The evaluability assessment itself plays a program

development role.

Opportunity to provide technical assistance

43


Impact on the field

Themes and issues emerged for clusters of

policies and programs

Evaluability assessments can be configured to

cast new light on

• developments in the field

• families or clusters of policies and programs

Researchers are fascinated by what practice is doing

Stimulated discussion of new research agendas

44


Your Comments/Questions

• Use Chat to post comments

and/or questions

• ‘Send’ questions to All (not

privately to ‘Host’)

Chat

Participant Side

Panel in WebEx


Poll Question #3

Could this method or tool be useful

in practice?

A. Very useful

B. Somewhat useful

C. Not at all useful

D. Don’t know


Your Feedback is Important

Please take a few minutes to share your thoughts

on today’s webinar.

Your comments and suggestions help to improve

the resources we offer and plan future webinars.

The short survey is available at:

https://nccmt.co1.qualtrics.com/jfe/form/SV_71yjf5

AwP51pRdP

https://nccmt.co1.qualtrics.com/jfe/form/SV_71yjf5AwP51pRdP


Poll Question #4

What are your next steps? (Check all

that apply)

A. Access the method/tool referenced in the

presentation

B. Read the NCCMT summary about the

method/tool described today

C. Consider using the method/tool in

practice

D. Tell a colleague about the method/tool


Share your story!

• Are you using EIDM in your practice? We want

to hear about it!

• Email us: [email protected]

• Need support for EIDM? Contact us for help!

• Email us: [email protected]

• We typically respond within 24 business hours

49


Join us for our next webinar

Spotlight on Methods and Tools: EENet:

Mental health and addictions knowledge translation resources

Date: Thursday January 25, 2018

Time: 1:00 – 2:30pm ET

Do you work in mental health and addiction? Interested in a

knowledge exchange network that helps to create and share

evidence to build a better mental health and substance use system?

Join us to find out!

Register at: https://health-evidence.webex.com/health-

evidence/onstage/g.php?MTID=ef97dccfad971be665e8e7550172d7

ada

https://health-evidence.webex.com/health-evidence/onstage/g.php?MTID=ef97dccfad971be665e8e7550172d7ada


Webinar Series from NCCMT

www.nccmt.ca/webinar-series

• Spotlight on Methods and Tools

• Topic-Specific Methods and Tools

• Online Journal Club

• Peer-to-peer Webinars

51

http://www.nccmt.ca/webinar-series




For more information about the National Collaborating Centre for Methods and Tools:NCCMT website www.nccmt.ca

Contact: [email protected]


mailto:[email protected]


Systematic Selection

•7 After School/3 Daycare Programs

• 5 programs: PA time, nutritious snacks

• 4 programs: PA time, nutrition education

• 1 policy: PA, nutrition, TV screen time

•10 Food Access Programs

• 5 farmers’ markets

• 3 supermarket or corner store programs

• 2 restaurant programs

•6 School District Local Wellness Policies

• All selected addressed PA and nutrition

53


Results

•Expert panel determined:

– 14 ready for stand-alone, outcome evaluation

– 2 best suited for cluster evaluation

– 3 theoretically sound but need further development

– 6 need TA in specific areas

54

spotlight webinar: evaluability assessments in public health

Education