how to measure the success of learning and … swls...how to measure the success of learning and...
TRANSCRIPT
How to Measure the Success of Learning and Development:
Capturing Impact and ROI of a Learning Program
Patti Phillips, PhD President & CEO ROI Institute, Inc.
Objectives
Identify the five levels of evaluation Identify steps in ROI Methodology Discuss a case study on ROI Plan next steps to build capability
2
The ROI Calculation
BCR =
ROI =
Program Benefits Program Costs
Net Program Benefits Program Costs
X 100
The ROI Calculation
$750,000 $425,000 BCR =
$750,000 - $425,000 $425,000
ROI = X 100
=
=
$750,000 $425,000 BCR =
$750,000 - $425,000 $425,000
ROI = X 100
= 1.76:1
= 76%
The ROI Calculation
LLeevveellss ooff EEvvaalluuaattiioonn MMeeaassuurreemmeenntt FFooccuuss
0. Inputs and Indicators
The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing
1. Reaction & Perceived Value
Reaction to the project or program, including the perceived value
Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action
2. Learning & Confidence
Learning to use the content and materials, including the confidence to use what was learned
Skills, Knowledge, Capacity, Competencies, Confidence, Contacts
3. Application & Implementation
Use of content and materials in the work environment, including progress with actual items and implementation
Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use
4. Impact and Consequences
The consequences of the use of the content and materials expressed as business impact measures
Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement
TTyyppiiccaall MMeeaassuurreess
5. ROI Comparison of monetary benefits from program to program costs
Benefit-Cost Ratio (BCR), ROI%, Payback Period
Program Business Alignment and Forecasting The ROI Methodology
Business Alignment
Learning Needs
Preference Needs
Evaluation
Reaction
Learning
Application
Impact
ROI
Reaction Objectives
Learning Objectives
Application Objectives Performance Needs
Impact Objectives Business Needs
Payoff Needs ROI Objectives
End Here Start Here
5
4
3
2
5
4
3
2
1 1
Initial Analysis
Input Needs 0 Input Objectives Input 0
8
Data Collection During and After Program
Method Level 1 2 3 4
Surveys üü üü üü Questionnaires üü üü üü üü
Observation üü üü üü Interviews üü üü üü Focus Groups üü üü üü Tests/Quizzes üü Demonstrations üü Simulations üü Action planning/improvement plans üü üü
Performance contracting üü üü
Performance monitoring üü
Factors to Consider
When selecting methods, consider:
Time required for participants Time required for supervisors Costs of methods Amount of disruption
Accuracy Utility Culture/Philosophy
!When selecting sources of data, consider:
Participants Supervisors Direct reports Peer groups
Internal staff External sources Organizational records
!When determining timing, consider:
Availability of data Ideal time for behavior
change/application
Ideal time for business impact Convenience of data collection Constraints on data collection
!Reaction and Learning data are collected during the program because that is when you need them.
Use of a control group arrangement Trend line analysis of performance data Use of forecasting methods of performance data Participant’s estimate of program impact (percent) Supervisor’s estimate of program impact (percent) Manager’s estimate of program impact Use of expert/previous studies Calculate/estimate the impact of other factors Customer input
Methods to Isolate Program Effects
Data are converted by:
Converting output to contribution – standard value Converting the cost of quality – standard value Converting employee’s time – standard value Using historical costs Using internal and external experts Using data from external databases Linking with other measures Using participants’ estimates Using supervisors’ and managers’ estimates Using staff estimates
1. Report the complete story
2. Conserve resources
3. Use the most credible sources
4. Choose the most conservative alternatives
5. Isolate the effects of the program
6. No data no improvement
7. Adjust estimates for error
8. Throw out the extreme and unsupported claims
9. Use first year benefits for short-term programs
10. Include fully-loaded costs
11. Report intangible benefits
12. Communicate results to all stakeholders
Operating Standards
Reaction
Learning
Application
Impact
ROI
Isolate the Effects of the Program
Intangible Benefits
Inputs
ICR ROI Analysis Plan Program: ___________________________________ Responsibility:____________________________ Date:_____________
Data Items
(Usually Level 4)
Methods for Isolating the Effects of the
Program/ Process
Methods of Converting
Data to Monetary
Values Cost Categories Intangible Benefits
Communication Targets for Final
Report
Other Influences/ Issues During
Application Comments Varies, depending on measures selected
Participant
estimate
Standard
value Expert
value Participant
estimate
Needs Assessment (Prorated)
Program Dev. (Prorated)
Facilitation fees Pro. materials Facilitation & coordination
Meals and refreshments
Facilities Participant salaries & benefits - for time away from work
Mgrs. salaries & benefits -for time involved in program
Cost of overhead
Evaluation
Job satisfaction for first level managers
Job satisfaction for team members
Improved teamwork
Improved communication
Participants (first level managers)
Participants’ managers
Senior executives L&D staff Prospective participants
Learning & development council members
Several process improvement initiatives are on-going during this program implementation
Must gain commitment to provide data
A high response rate is needed
17
Plan for High Response Rate
18
Before administering the survey Design for confidentiality, simplicity, and ease of completion Describe the time it will take to the complete the survey Provide a due date (3 weeks before launch) Decide on incentives for early response consistent culture and
practices Program Manager send announcement Program Manager send link to respondents
During the response period Send a reminder with survey link Send a reminder with survey link and up-to-date response rate Send final reminder with survey link, up-to-date response, and
announcement of close
After the response period Send thank you for responding with final response rate Send brief overview of results and plan for their use Send summary of actions taken based on results
19
Reaction
20
Learning
21
Application
22
Barriers & Enablers
23
Business Impact
24
Business Impact
Estimates are reasonable Measure Fact % Contrib. Est. Impact Confidence Adjusted
Impact
Sales (Profit) $13,100 60% $7,860 80% $6,288
Fact: Sales (Profit) increased $13,100 % Contribution: 60% Est. Impact: $7,860 Uncertainty 20% Margin of Error: +/- $1,572
$9,432 $7,860 $6,288
Program Costs
26
27
Return on Investment
If: Benefits = $329,201
Costs = $160,754
What is the BCR?
What is the ROI?
28
Return on Investment
It depends, in part, on your worldview.
If you are measuring,
you are estimating.
There are no absolutes.
Sometimes the crowd
knows best.
Be able to explain what you did, how
you did it, and why you did it
that way.
A statistic is an estimate of what
probably is… maybe.
Process without
standards is no process.
The rest is a balancing act.
30
Benefits
Costs
If you would like a copy of the case study download a copy of
the Global Car Rental Case Study at
www.roiinstitute.net/free-tools
31