how to measure the success of learning and … swls...how to measure the success of learning and...

How to Measure the Success of Learning and Development:

Capturing Impact and ROI of a Learning Program

Patti Phillips, PhD President & CEO ROI Institute, Inc.

Objectives

  Identify the five levels of evaluation   Identify steps in ROI Methodology   Discuss a case study on ROI   Plan next steps to build capability

2

The ROI Calculation

BCR =

ROI =

Program Benefits Program Costs

Net Program Benefits Program Costs

X 100

The ROI Calculation

$750,000 $425,000 BCR =

$750,000 - $425,000 $425,000

ROI = X 100

=

=

$750,000 $425,000 BCR =

$750,000 - $425,000 $425,000

ROI = X 100

= 1.76:1

= 76%

The ROI Calculation

LLeevveellss ooff EEvvaalluuaattiioonn MMeeaassuurreemmeenntt FFooccuuss

0. Inputs and Indicators

The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing

1. Reaction & Perceived Value

Reaction to the project or program, including the perceived value

Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action

2. Learning & Confidence

Learning to use the content and materials, including the confidence to use what was learned

Skills, Knowledge, Capacity, Competencies, Confidence, Contacts

3. Application & Implementation

Use of content and materials in the work environment, including progress with actual items and implementation

Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use

4. Impact and Consequences

The consequences of the use of the content and materials expressed as business impact measures

Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement

TTyyppiiccaall MMeeaassuurreess

5. ROI Comparison of monetary benefits from program to program costs

Benefit-Cost Ratio (BCR), ROI%, Payback Period

 Program  Business Alignment and Forecasting The ROI Methodology

Business Alignment

Learning Needs

Preference Needs

Evaluation

Reaction

Learning

Application

Impact

ROI

Reaction Objectives

Learning Objectives

Application Objectives Performance Needs

Impact Objectives Business Needs

Payoff Needs ROI Objectives

End Here Start Here

5

4

3

2

5

4

3

2

1 1

Initial Analysis

Input Needs 0 Input Objectives Input 0

8

Data Collection During and After Program

 Method Level 1 2 3 4

  Surveys üü üü üü     Questionnaires üü üü üü üü

  Observation üü üü üü     Interviews üü üü üü     Focus Groups üü üü üü     Tests/Quizzes   üü       Demonstrations   üü       Simulations   üü       Action planning/improvement plans     üü üü

  Performance contracting     üü üü

  Performance monitoring       üü

Factors to Consider

When selecting methods, consider:

Time required for participants Time required for supervisors Costs of methods Amount of disruption

Accuracy Utility Culture/Philosophy

!When selecting sources of data, consider:

Participants Supervisors Direct reports Peer groups

Internal staff External sources Organizational records

!When determining timing, consider:

Availability of data Ideal time for behavior

change/application

Ideal time for business impact Convenience of data collection Constraints on data collection

!Reaction and Learning data are collected during the program because that is when you need them.

  Use of a control group arrangement   Trend line analysis of performance data   Use of forecasting methods of performance data   Participant’s estimate of program impact (percent)   Supervisor’s estimate of program impact (percent)   Manager’s estimate of program impact   Use of expert/previous studies   Calculate/estimate the impact of other factors   Customer input

Methods to Isolate Program Effects

Data are converted by:

  Converting output to contribution – standard value   Converting the cost of quality – standard value   Converting employee’s time – standard value   Using historical costs   Using internal and external experts   Using data from external databases   Linking with other measures   Using participants’ estimates   Using supervisors’ and managers’ estimates   Using staff estimates

1.  Report the complete story

2.  Conserve resources

3.  Use the most credible sources

4.  Choose the most conservative alternatives

5.  Isolate the effects of the program

6.  No data no improvement

7.  Adjust estimates for error

8.  Throw out the extreme and unsupported claims

9.  Use first year benefits for short-term programs

10. Include fully-loaded costs

11. Report intangible benefits

12. Communicate results to all stakeholders

Operating Standards

Reaction

Learning

Application

Impact

ROI

Isolate the Effects of the Program

Intangible Benefits

Inputs

ICR ROI Analysis Plan Program: ___________________________________ Responsibility:____________________________ Date:_____________

Data Items

(Usually Level 4)

Methods for Isolating the Effects of the

Program/ Process

Methods of Converting

Data to Monetary

Values Cost Categories Intangible Benefits

Communication Targets for Final

Report

Other Influences/ Issues During

Application Comments Varies, depending on measures selected

Participant

estimate

Standard

value Expert

value Participant

estimate

Needs Assessment (Prorated)

Program Dev. (Prorated)

Facilitation fees Pro. materials Facilitation & coordination

Meals and refreshments

Facilities Participant salaries & benefits - for time away from work

Mgrs. salaries & benefits -for time involved in program

Cost of overhead

Evaluation

Job satisfaction for first level managers

Job satisfaction for team members

Improved teamwork

Improved communication

Participants (first level managers)

Participants’ managers

Senior executives L&D staff Prospective participants

Learning & development council members

Several process improvement initiatives are on-going during this program implementation

Must gain commitment to provide data

A high response rate is needed

17

Plan for High Response Rate

18

Before administering the survey   Design for confidentiality, simplicity, and ease of completion   Describe the time it will take to the complete the survey   Provide a due date (3 weeks before launch)   Decide on incentives for early response consistent culture and

practices   Program Manager send announcement   Program Manager send link to respondents

During the response period   Send a reminder with survey link   Send a reminder with survey link and up-to-date response rate   Send final reminder with survey link, up-to-date response, and

announcement of close

After the response period   Send thank you for responding with final response rate   Send brief overview of results and plan for their use   Send summary of actions taken based on results

19

Reaction

20

Learning

21

Application

22

Barriers & Enablers

23

Business Impact

24

Business Impact

Estimates are reasonable Measure   Fact   %  Contrib.   Est.    Impact   Confidence   Adjusted  

Impact  

Sales  (Profit)   $13,100   60%   $7,860   80%   $6,288  

Fact: Sales (Profit) increased $13,100 % Contribution: 60% Est. Impact: $7,860 Uncertainty 20% Margin of Error: +/- $1,572

$9,432 $7,860 $6,288

Program Costs

26

27

Return on Investment

If:   Benefits = $329,201

  Costs = $160,754

  What is the BCR?

  What is the ROI?

28

Return on Investment

It depends, in part, on your worldview.

If you are measuring,

you are estimating.

There are no absolutes.

Sometimes the crowd

knows best.

Be able to explain what you did, how

you did it, and why you did it

that way.

A statistic is an estimate of what

probably is… maybe.

Process without

standards is no process.

The rest is a balancing act.

30

Benefits

Costs

If you would like a copy of the case study download a copy of

the Global Car Rental Case Study at

www.roiinstitute.net/free-tools

31