ctsa program common metrics initiative maximizing ctsa
TRANSCRIPT
Maximizing CTSA Program Impact
Module Two
Philip L. LeeResults Leadership Groupwww.resultsleadership.org
CTSA Program Common Metrics Initiative
Based on the principles of Results-Based Accountability from Trying Hard is Not Good Enough: How to Produce Measurable Improvements for Customers and Communities by Mark Friedman (Trafford 2005)
Module Two
1. Review of Interim Learning Activities
2. Maximizing the Impact of the Whole (as well as the parts)
3. Selecting Headline Performance Measures
4. Interim Learning Activities
Review of Interim Learning Activities
1. Frequently Asked Questions
2. Observations
3. Recap
Module Two
1. Review of Interim Learning Activities
2. Maximizing the Impact of the Whole (as well as the
parts)
3. Selecting Headline Performance Measures
4. Interim Learning Activities
Population Health
Health Care
Biomedical Research
Whole Versus PartsMaximizing the Impact of the Whole (as well as the parts)
Environment
Public Safety
Education
Economy
Socio-economicStatus
Biomedical Research
NIH
Whole Versus Parts
IndustryPatients/Communities
Universities/Medical Centers
CTSAProgram
CTSAProgram
Biomedical Research
CTSA ProgramHub
Hub
Hub
Hub
HubHub
Hub
TICS RICs
Trial Innovation Network
Whole Versus Parts
Hub
Hub
Hub Hub
Hub
Hub
HubHubHub
Hub
Hub
Hub Hub
HubHub
Hub
Workforce Development
Hub
BMIRegulatory Knowledge
PilotFunding
Community Engagement
Whole Versus Parts
BERDBERD
BERD
Whole Versus Parts
10
How much work did an
individual partdeliver?
Maximizing the Impact of the Whole (as well as the parts)
Distinguishing Parts vs. Whole; Effort vs. Impact
What was the impact of the
individual part?
What was the impact of the
whole?
Parts Whole
Impa
ct
Effo
rt How much work did the
whole deliver?
FPSI/RLG11
Effort versus impact?
Benchmark
Target
Indicator Goal
Result
Objective
Outcome
Measure
ModifiersMeasurable CoreUrgent QualitativePriority ProgrammaticTargeted PerformanceIncremental Strategic
Systemic
Whole Versus Parts
Definitions
Result
Indicator
Performance Measure
Babies Born Healthy, Safe Communities, Clean Environment, Free from Death and Suffering Due to [disease]
Rate of low birth weight babies, Crime rate, Air quality index, Mortality and morbidity rates for [disease]
1. How much did we do? 2. How well did we do it? 3. Is anyone better off?
A condition of well-being for children, adults, families or communities.
A measure which helps quantify the achievement of a result.
A measure of how well a program, agency, or service system is working.
= Customer Results
POPU
LATI
ON
C
UST
OM
ERS
Three types:
(Language Discipline)
Performance Measures1. How much did we do?
Who are our customers and what services do we provide to them?
2. How well did we do it? How well do we provide those services?
3. Is anyone better off?What is the desired impact of those services for/on our customers (a.k.a. “customer results”)?
Indicator orPerformance
Measure?
What kind ofPerformance
Measure?
Provide Training
Mastered New Job Skills
Employed
Employed @ skill level
Employment@ Skill Level & > year
CityEmployment
Rate
Job Training Program
How much?
Population = Indicator
Certified Instructors
1. Safe Community
2. Crime Rate
3. Average Police Dept response time
4. Free from death and suffering due to hypertension
5. Mortality and morbidity rates for hypertension
6. Average blood pressure of pulmonary hypertension clinic patients
7. Youth Enter College or Good Jobs
8. % of youth who graduate from high school
RESULT, INDICATOR OR PERFORMANCE MEASURE?RESULT
INDICATOR
PERF. MEASURE
RESULT
INDICATOR
RESULT
INDICATOR
PERF. MEASURE
HOW WELL?
BETTER OFF?
PERF. MEASUREBETTER OFF?
Indicator or Performance
Measure?
What kind ofPerformance
Measure?
Provide CT resources & services and CT
education/training & early career support
Improved translation & translation processes
(speed, cost, quality, etc.)& successful careers in
CT Research
Improved Mortality & Morbidity
Rates
Population = Indicator
CTSAPROGRAM
Develop, Demonstrate,& Disseminate
How much?
How much?
Whatactivities, services,
and/orresources
do we perform/provi
de?
Who are our customers?
What is the desired impact
of those activities/services/resourceson/for those customers?
How are we doing at
achieving that desired impact?
Common Metrics = Data/Feedback on ImpactEffort/How much? Impact/Better off?
What PIs/NCATS want to track
(“Requirements”)
Common Metrics(Data)
1. Free from death and suffering due to [disease]
2. Mortality and morbidity rates for [disease]
3. Median IRB duration
4. % CT scholars with grants/publications
5. Median study start up time
4. Rates of Pilot Funding publications and subsequent funding
4. Accrual rates
RESULT
INDICATOR
PERF. MEASURE
PERF. MEASURE
PERF. MEASURE
CTSA Program
BETTER OFF?
BETTER OFF?
BETTER OFF?
PERF. MEASUREBETTER OFF?
PERF. MEASUREBETTER OFF?
RESULT, INDICATOR ORPERFORMANCE MEASURE?
The Whole, the Parts & Ends-to-Means Decision Making
(a.k.a., Turn-the-Curve Thinking)
1. Aligning our activities in service of our ends.
2. The matter of contribution.
Population HealthLung Cancer
NIH
Biomedical Research
Industry
Patients/Communities
Universities/Medical Centers
CTSAProgra
m
Whole Versus Parts
Population HealthLung Cancer
Health Care
Biomedical Research
Environment
Public SafetyEducation
Economy
Socio-economicStatus
Strategic Priorities?
Whole Versus Parts
Story behind the curve
Partners (with a role to play in turning the curve)
What would work to turn the curve
Strategy
How are we doing?
Why?
Help?
Options?
Propose to do?
(c) Results Leadership Group, LLC 23
Mortality & Morbidity Rates (the “Curve”)
Free from Death and Suffering Due to Lung Cancer
1. Cigarette smoking2. Socio–economic status3. Environmental hazards4. Inadequate knowledge of the causes
of lung cancer
What is the story behind the curve?
What is our strategy to turn the curve?1. Smoking Cessation Campaign2. Behavioral research on smoking3. Targeted education campaign4. Lung cancer research
Whole
Part
Priority
Doing the
right things?
Ends
Means
Lung Cancer Mortality and Morbidity Rates
Smoking Cessation Program
Whole Versus Parts
Story behind the curve
Partners (with a role to play in turning the curve)
What would work to turn the curve
Strategy
(c) Results Leadership Group, LLC 26
ProgramPerformance
Measures
Whole
Smoking Cessation Program
Doing things
right?
Start withthis End
Mortality and Morbidity rates for ...
National/Community-wide Strategy/Partners
Smoking Cessation Program Performance Measures (baselines)
WH
OLE
PAR
TS
Customer Impact END
X
1. Doing the right things?
2. Doing those things right?
Population Accountability
Performance Accountability
?
Joint Accountability
Sole Accountability
END
M
EAN
S
People are Healthy
The Whole, the Parts & Ends-to-Means Decision Making
People are Healthy
The Matter of Contribution (Joint vs. Sole Accountability)
People are HealthyMortality and morbidity rates
Health Care Patients are healthy/not suffering
Biomedical Research New biomedical knowledgeNew treatments
Translational Research/Innovation • New knowledge about translation• Improvements to translation processes &
translation rates
What Is Your “End”?
Population HealthIndicators:
Story:Partners:Strategy (including the role of the CTSA Program):
Population Accountability
CTSA ProgramCTSA Program Common Metrics:
Story:Partners:Strategy to “turn the curve” of a CTSA Program Common Metric:
Performance Accountability
31
The CTSA Program shares joint accountability with multiple stakeholders to “turn the curve” of an indicator (e.g., mortality and morbidity rates). The CTSAProgram has a role within the larger strategy.
The CTSA Program is solely accountable to “turn the curve” of the CTSA Common Metrics –(which contributes to the larger strategy to turn the curve of the indicator).
!
32
I sure am glad we don’t have that problem!
I sure am glad we don’t have that problem!
!
33
!
34
Joint Accountability
!
35
Sole AccountabilitySole Accountability
Right Strategy Tools
Right Strategy Roles & Partnerships
Right Implementation?
Right Data for Useful Feedback?
CTSA ProgramHub
Hub
Hub
Hub
HubHub
Hub
TICS RICs
Trial Innovation Network
Hub
Hub
Hub Hub
Hub
Hub
HubHubHub
Hub
Hub
Hub Hub
HubHub
Hub
Whole Versus Parts
Workforce Development
Hub
BMIRegulatory Knowledge
PilotFunding
Community Engagement
BERD
Whole Versus Parts
Hub Accrual Rates
What is the story behind the curve?
What is our strategy to turn the curve?
Whole
Doing the right things?
End
1. Non-investigator clinicians not screening patients for potential study participants during clinic consultations.
2. Lack of systems for data-driven cohort discovery in planning clinical studies
3. Inadequate knowledge/skills in recruitment planning/implementation
4. Study designs that impede participation5. Sites inconveniently located.
Priority
1. Research, develop, and institute targeted strategies to motivate non-investigator clinicians to screen patients.
2. Develop and implement social media strategies.
PartsMeans
“People and their managers are working so hard
to be sure things are done right, that they hardly have time
to decide if they are doing the right things.”
Stephen R. Covey
Common Metrics for the CTSA Program
Hub Strategy to Turn-the-Curve on Accrual Rates
Performance Measures (baselines)
WH
OLE
PAR
TS
Customer Impact END
X
1. Doing the right things?
2. Doing those things right?
?
END
M
EAN
SHub Accrual Rates
Whole or Part?End or Means?
Clinician PatientScreening Program
Hub Accrual Rates
Hub Strategy to Turn-the-Curve on Accrual Rates
Performance Measures (baselines)
Customer Impact END
X
1. Doing the right things?
2. Doing those things right?
?
Whole or Part?End or Means?
Clinician Patient Screening Program
PM
What is the story behind the curve?
• Non-investigator clinicians not screening patients for potential study participants during clinic consultations. (See: CTSI report)
• Lack of systems for data-driven cohort discovery in planning clinical studies
• Inadequate knowledge/skills in recruitment planning/implementation
• Study designs that impede participation
• Sites inconveniently located.
% of studies that achieved accrual goal within time specified in study designPM
50 %Current Value:Q2 2016
PM
What are we going to do?
% of studies that achieved accrual goal within time specified in study designPM
Research, develop, and institute targeted strategies to motivate non-investigator clinicians to screen patients. Engage institutional leadership.
Develop and implement social media strategies.Partner with the communications school.
50 %Current Value:Q2 2016
PerformanceMeasures? Performance
Measures?
Clinician Patient Screening Program
Whole Versus Parts
Story behind the curve
Partners with a role to play in turning the curve
What would work to turn the curve
Strategy to turn the curve
(c) Results Leadership Group, LLC 49
Performance Measures?
Whole
Clinician Patient Screening Program
Doing things
right?
Start withthis End
Accrual Rates:
Story:
Strategy (including the role of the Clinician Patient Screening Program):
Hub Common Metric
Headline Performance Measures:
Story:
Strategy to “turn the curve” of a program headline performance measure:
Clinician Patient Screening Program
50
The manager of the Clinician Patient Screening Program shares joint accountability with multiple stakeholders to “turn the curve” of the accrual rates for the Hub. The Clinician Patient Screening Program has a role within the larger strategy.
The manager of the ClinicianPatient Screening Program is solely accountable to“turn the curve” of the HeadlinePerformance Measures for the Clinician Patient Screening Program.
Module Two
1. Review of Interim Learning Activities
2. Maximizing the Impact of the Whole (as well as the parts)
3. Selecting Headline Performance Measures
4. Interim Learning Activities
52
How much service did we deliver?
Selecting Headline Performance Measures
How well did we
deliver it?
How much change / effect
did we produce?
What quality of change / effect
did we produce?
Quantity Quality
Effe
ct
E
ffort
How muchdid we do?
Three Kinds of Program Performance Measures
How welldid we do it?
Is anyonebetter off?
Quantity QualityE
ffect
E
ffort
# %
Quantity Quality
Efficiency, Admin overhead, Unit costStaffing ratios, Staff turnoverStaff morale, Access, Waiting time, Waiting lists, Worker safetyCustomer Satisfaction1. quality service delivery2. customer benefit
Cost / Benefit ratioReturn on investmentClient results or client outcomes
EffectivenessValue addedProductivity
Benefit value
ProcessInput
Effe
ctEf
fort
CostTQM
1. Did we treat you well?
2. Did we help you with your problems?
ProductOutput Impact
Performance Measure Categories
# of students
Education
Student-teacher ratio
#/% of high school graduates
Quantity QualityEf
fect
Effo
rt
# of persons treated
Drug/Alcohol Treatment
% of staff with training certification
#/% of clients off of drugs/alcohol
> at exit> 12 months after exit
Quantity QualityEf
fect
Effo
rt
# of responses
Fire Department
Response time
#/% of fires kept to room
of origin
Quantity QualityEf
fect
Effo
rt
Criteria for SelectingHeadline Performance Measures
Communication Power
Proxy Power
Data Power
Does the performance measure communicate to a broad range of audiences?
Does the performance measure say something of central importance about the result/program?
Does the performance measure bring along the data HERD?
Quality data available on a timely basis.
(c) Results Leadership Group, LLC59
Selecting Headline Performance Measures WorksheetUnit/Program _______________________________
H = High; M = Medium; L = Low
Candidate Performance Measure
CommunicationPower
ProxyPower
DataPower
H M L
H
Measure 1Measure 2Measure 3Measure 4Measure 5Measure 6Measure 7Measure 8
HDataDevelopmentAgenda
H M L H M L
H H
H L
Provide Training
Mastered New Job Skills
Employed
Employed @ skill level
Employment@ Skill Level & > year
CityEmployment
Rate
How much?
Population = Indicator
Certified Instructors
Right Question
Data Development Agenda
Feasible “Proxy”
Job Training Program
Selecting a HeadlinePerformance
Measure
Attendance
Reading on Grade Level
Graduation Rate
College Success Rate
Career SuccessRate
Right Question
Feasible “Proxy”
Data DevelopmentAgenda
Schools
Selecting a HeadlinePerformance
Measure
Module Two
1. Review of Interim Learning Activities
2. Maximizing the Impact of the Whole (as well as the parts)
3. Selecting Headline Performance Measures
4. Interim Learning Activities
Interim Learning Activities1. Revise and enter into Results Scorecard the headline performance
measure and turn-the-curve plan for the component you selected for Module Two.
2. Complete the Asthma Sorting Exercise.
3. Practice facilitating a turn-the-curve process with a group of colleagues. Have your learning partner observe and provide you with feedback.
4. Using the RBA Turn-the-Curve Plan Rubric, assess the sample Turn-the-Curve Plan.
5. Read the sample performance report and reorganize its elements into a turn-the-curve plan in Results Scorecard.