Outcome measures for assessing progress of meeting PHP or SPH Goals.

Lynn D. Woodhouse M Ed, Ed D, MPHAssociate Dean of Faculty Affairs, Accreditation and Assessment and

Professor of Community Health and Health BehaviorJiann-Ping Hsu College of Public Health

Georgia Southern University

CEPH Technical Assistance SessionAPHA November, 2007

Washington, DC

Outline for this presentation:

What are outcomes?How do they fit into the larger picture of assessment and evaluation?Why do they matter?Why are they such a challenge to develop?Can a more effective process for developing outcomes be implemented?

Outcome measures or performance measures:

the quantifiable indicators that gauge productivity or effectiveness

the measurable variables by which attainment of objectives may be judged

Purpose of using outcome measures:

Monitoring of multiple indicators enables a PHP or SPH to document:– the successes, – areas in need of improvement – and, ultimately, the effectiveness of efforts to meet

goals.

To support and sustain deliberative evaluation that is also meaningful to the multiple stakeholders of a PHP or SPH.

Who are these stakeholders?

Students, graduatesCommunity members, public health professionalsFaculty, Administrators, larger institutionGroups described in mission or vision of SPH or PHPThe publicFunders, etc.

Caveat:

The use of outcome measures is not a substitute for thoughtful evaluation

SPH and PHP will continue to assess the less tangible parts of the academic mission, the things that are not easily measured

Remember . . “You are what you measure”

“Fuzzy Goals” and Outcome Measure Development (Patton ’97, Weiss ’72))

Problem: Lack of clear, specific, prioritized (and measurable with objectives) goals.Human cognitions are more intuitive than analytical – hard to think in “goal” formatLack of clarity can mask:– lack of understanding of WHAT to accomplish– divergent intents – underlying conflict

Lack of contextual analysis to assess various stakeholders’ beliefs about what goals should beMany disciplines define these terms differently

Determine Purpose and Direction(goals for I, R, S and Org)

THEN objectives and outcome measures of these

Examples of questions to use:– What are you trying to achieve?– If successful what will be different?– How will stakeholder groups be different if

successful? – Who will behave differently, why and how?– What will people be able to report is different

because of changes?– What would you (as stakeholder) be able to see in

one specific/or all stakeholders that would be different?

Developing link between goal(s) and outcomes

Goals can be system level, organizational level, or at level of stakeholder groups– BUT should lend themselves to

objectives/activities AND outcomes that measure purpose and direction

Definition of outcomes should precede any discussion of HOW to measure them

Example of how people attempt to develop outcomes: You can lead a horse to water . .

(Patton ’97)

You can lead a horse to water, measure proximity to the water and even count her “sips” of water - but, . . . . . . . . would you really prefer to demonstrate reduced horse dehydration?

Relevance . . .You can control (and we tend to measure):– Leading horse to water– Making sure the water quality is good– Measuring the “sipping ability” of the horse– Keeping the horse happy and satisfied

You can NOT control (and tend NOT to measure):– If the horse actually drinks the water– How much water the horse drinks– Horse urine– The ultimate health status (dehydration?) of the horse

Main point here: Our tendency is to focus on what we can control

It is difficult to move from counting “services or activities” to measuring meaningful outcomes

Tendency is to focus on what you can control (what YOU do and what YOU can count) – not on what you are trying to facilitate (healthy horses)

Developing valuable outcome measures:

Involves linking type of change to specific indicators and measures of those indicatorsConsider:– Time frames– Resources for measuring– Validity and reliability of measures– How information/outcome findings will be used– Willingness of primary stakeholders to engage in this

type of assessment– Clarity, specificity and measurability (Patton ’97)

2 Examples from various levels of development: ESU and JPHCOPH

Outcome measures link the evaluation and planning process feedback loop for PHP and SPH

ESU Goal V: To Prepare graduates who are self-motivated, work collaboratively, apply ethical principles to their work, exercise initiative,

have critical thinking skills and develop into leaders in public health.

Goal Program level Objectives Outcome Measure Assessment’04

Students as leaders

1.Maintain a diverse student body that reflects the diversity of the community we serve.

1.Program student and graduate diversity will enhance the diversity of the region as compared to the regional statistics. demographics

Met

2.Maintain a student body with diverse educational and professional backgrounds

2.Assessments of records kept of the educational and professional backgroundsof the students to ensure our students body is diverse in these areas.

Met

3.Graduates will be employed at the level appropriate to their education and experience in a position that contributes to population-based public health

3.Graduates employment history will be documented on regular surveys. Rates of employment in areas that contribute to population-based public health will be recorded. AT THIS POINT A TARGET LEVEL IS NOT SET.

>60% in PH

4.Graduates will report attainment of these outcomes on our regular surveys. 4.All students will be assessed favorably by their internship preceptor on these aspects during their internship experience

4. Graduates will respond with a 3.0 or better on their rankings of the outcomes related to Goal E on the evaluation survey.

Met

5. During their tenure in our MPH program all students will successfully participate in multiple (no less than 3) group level experiential or service learning activities.

5.Documentation from faculty of the number of and types of collaborative, team building activities in which students participate in the required MPH courses.

Met – Vita and student database

Outcome statements are clearly separated from operational criteria for measurement of

the outcome:

Specification of desired outcome (connected to objectives/goals)Determine what is a reasonable measurement of the desired outcomeDetermine performance target (s)Determine time frames

GSU - JPHCOPH processpeeling back the onion

WorkforceGoal:

Sample objective with activity

1 Outcome expected from the activity

Proposed measure of the this outcome

Data source

Criteria for success or benchmark -baseline

Provide three exceptional competency-based DrPH programs ( )that graduate leaders for the public health workforce who contribute to the mission of JPHCOPH during their academic training and throughout their careers.

All graduates will demonstrate the core competencies for all 3 DrPH concentrations through effectively designed and delivered coursework, practicum and dissertation.Additional:1.Demonstrate Comp for Concentrations2.Exceptional programs3.Graduate leaders for 4.Contribute to mission in training5.Contribute to mission during career

Demonstration of core competencies

Successful completion of and evaluation of coursework, practicum and dissertation

Alumni self reports

Exit interview?

Preceptor Evaluation, Course grades and assessments, dissertation defense and presentation

Alumni Survey

Practicum = 90% students will receive a 4 or better (5 point scale) on competency evaluation completed by Preceptor

Alumni report – 80% 4 or better for each competency

Outcomes could reflect change in:

Policy/rules/regulationsCircumstances for organizations, groups or individualsStatus – health status?Behavior, skills or competenciesAbility to functionKnowledge or attitudePreventionActivities – process measures

Setting meaningful performance measures:

Amount or level of outcome attainment that is expected or required.Use past performance/baselineSet progress measures at reasonable levels – normativeWatch for the relationship between resources and performance

Data collection

What data would give insights to this outcome: existing or new?Who “keeps” the data?Who is responsible? Who has oversight?How frequently will indicator data be collected? More often early on . . . Will sampling processes be used?How will findings be determined? Used? Shared?

Summary of developing outcome measures:

Process should: – resonate with stakeholders – contribute to valuable feedback loop for planning =

give insights based on purpose and direction– enable resource and performance based planning

and changeSpecifically planned outcomes should be the driver of planning rather than “available” data driving what the PHP or SPH will measure

Values, Vision and Mission

Goals for Instruction, Research and Service (Organization/System?)

Purpose and Direction

Statement ofObjectives

and Activitiesfor each goal-

What do you need to accomplish to meet goals?

DevelopingOutcome Measures

for ALL Objectives - Assessment

Development and Implementation of

Evaluation and Planning Overview

Concept:OngoingProcess