m101 and tu101 evaluating training effectiveness: the ... · evaluating training programs 2. expand...

M101 and TU101Evaluating Training Effectiveness:

The Kirkpatrick Four LevelsTM

Dr. Donald L. Kirkpatrick

© 2009- 2011, Kirkpatrick Partners, LLC. All rights reserved.

Trademark and Copyright Warning© Copyright 2010-2011, Kirkpatrick Partners, LLC. All

rights reserved. This presentation remains the exclusive property of Kirkpatrick Partners, LLC. Distribution or duplication of these slides and content, in whole or in part, in any way is strictly prohibited. These are for the personal use of program attendees only.

The following marks are the property of Kirkpatrick Partners, LLC as related to educational / training materials, programs and books: Kirkpatrick®

Kirkpatrick Four LevelsTM

The One and Only Kirkpatrick®


The Trainer’s Song

Words by Donald KirkpatrickTune: Clementine

They’re the greatest in the world,There is nothing more to say,No one dares to contradict it,They’re the Packers from Green Bay.



There were anti-Packer people.To run them down was just their goal.Now they’re eating crow and crying, ‘Cause they won the SUPER BOWL!



We are trainers teaching “teamwork”,We need a guide to show the way,Let us follow the example,Of the Packers from Green Bay!


HRD is:

A SCIENCE– Concepts– Theories– Principles– Techniques

AN ART– Application to your situation


Teaching Approach

SMART

vs.

PIE


Quiz1-4. In sequence, what are the 4 levels?5. What do we call a test we administer prior to a training

program?6. What do we call a test we administer after a training

program?7. What do we call the group that receives the training?8. What do we call the group that doesn’t?9. T or F We can borrow results from other organizations

if the program is the same.10. T or F Those who plan, coordinate, and teach training

programs are better able to judge its effectiveness than those who attend.


Requirements for an Effective Program

1. Based on __ __ __ __ __.2. Aimed at __ __ __ __ __ __ __ __ __ __3. Scheduled at the right __ __ __ __.4. Held at the right __ __ __ __ __.5. For the right __ __ __ __ __ __.6. Conducted by an effective __ __ __ __ __ __.7. Using effective __ __ __ __ __ __ __ __ __ __.8. __ __ __ __ __ __ __ __ __ __ are reached.9. Participants are __ __ __ __ __ __ __ __ __.10. Program is __ __ __ __ __ __ __ __ __.


Requirements for an Effective Program

1. Based on needs.2. Aimed at objectives.3. Scheduled at the right time.4. Held at the right place.5. For the right people.6. Conducted by an effective leader.7. Using effective techniques.8. Objectives are reached.9. Participants are satisfied.10. Program is evaluated.


Workshop Objectives

• To Understand– Why Evaluate?– The Four Levels– Guidelines– Forms and Approaches


Why Evaluate?

• Should the program be continued?

• How can the program be improved?

• How can trainers justify their existence?


Level 1: REACTIONHow participants react to the training

Kirkpatrick Model


Level 2: LEARNINGThe extent to which participants acquire intended knowledge, skills, and attitudes

Kirkpatrick Model


Level 3: BEHAVIOR The extent to which participants apply what

they learn on the job

Kirkpatrick Model


Level 4: RESULTS The final outcomes that occur as a result of the

training and reinforcement

Kirkpatrick Model


Evaluation Principles

1. Evaluation methods and forms can be borrowed from other organizations and used or adapted to your situation.

2. Evaluation results cannot be borrowed even if the program is the same.


Guidelines for Evaluating Reaction

• Determine what you want to find out.• Design a form that will quantify reactions.• Encourage written comments.• Get 100% immediate response.• If desirable, get delayed reactions.• Develop acceptable standards.• Measure future reactions against the

standard.


Methods for Evaluating Reaction

• Reaction sheet• Trained observer• Focus group


Reaction Sheet #1 Instructions: Please complete the form by checking theappropriate answer for each item:SD – strongly disagree D – disagree N – neutral A – agree SA – strongly disagree DNA – does not apply

SD D N A SA DNA1. The meeting started on time. 2. The program was practical and met

my needs. 3. The program maintained my interest

and attention. 4. The objectives were clearly stated.


Reaction Sheet #2 Instructions: Select your response after the

following items. 21. The length of the meeting was: ____ Too long ____ Just right ____ Too short

22. I am clear on how to apply what I learned: ____ very clear ____ clear ____ quite clear ____ not clear

23. What are three things you learned during the program?


Guidelines for Evaluating Learning

• Measure before and after attitudes, knowledge, and/or skills.

• Use a paper and pencil test for knowledge and attitudes.

• Use a performance test for skills.• Use a focus group for Levels 1 and 2

combined.


Methods for Evaluating Learning

• Pre and post test• Performance observation

– Teach back– Role play

• Focus group


Selected Items from “Management Inventory on Time

Management”A DA 1. Managers should be rewarded on the basis of the

amount of time they devote to the organization.A DA 2. It’s a good idea to handle each email or piece of

paper only once.A DA 3. It’s a good idea to do two things at once in order to save time.A DA 4. Unwanted visitors should be treated ‘coolly’ so they won’t

stay.A DA 5. When a direct report comes to you with a problem

which you can’t immediately solve, it’s a good policy to say, “I’ll look into it and get back to you”

A DA 6. Delegation should always include the results to be achieved as well as the activities to be performed.


FormPlease circle the items that you answered “incorrectly”according to the scoring key. Then, subtract the number wrong from 45 to get your score.

1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 1819 20 21 22 23 24 25 26 2728 29 30 31 32 33 34 35 3637 38 39 40 41 42 43 44 45

Your Score 45 - ___ = ___


Response to MITM Item 4“Unwanted visitors should be treated “coolly”

so they don’t stay long.”

Experimental Group Control GroupAgree Disagree Agree Disagree

Pretest 5 20 5 20

Posttest 10 15 6 19

Gain +5 +1


Response to MITM Item 6“Delegation should always include the results to beachieved as well as the activities to be performed.”

Experimental Group Control GroupAgree Disagree Agree Disagree

Pretest 7 18 6 19

Posttest 23 2 7 18

Gain +16 +1


Decision MakingManager

QUALITY Subordinates ACCEPTANCE

Past FutureAvg Range Avg Range25% 10-50% 16% 5-40% 1. Manager decides and sells decision37% 10-50% 30% 10-50% 2. Manager asks for input, considers it,

decides, and sells decision22% 10-50% 25% 5-50% 3. Manager leads subordinates in making

a “consensus” decision16% 0-40% 29% 0-60% 4. Manager empowers subordinates to ____ ____ make the decision100% 100%

ServiceMaster Executives August, 1996


Transferring Learning to Sustainable Behaviors

Give a man a fish and he

will eat today. Teach him how to fish

and he will eat forever


Change in Behavior Requires:

1. Desire to Change2. Necessary Knowledge and Skills3. The “Right” Job Climate4. Encouragement and Help5. Rewards for Change


The Job ClimateSupervisor Department Head

CLASSROOMKnowledgeSkillsAttitudes

CLIMATEPreventingDiscouragingNeutralEncouragingRequiring


Guidelines for Evaluating Behavior

• Measure on a before / after basis if practical.

• Allow time for behavior change to take place (perhaps 3-6 months).

• Get 100% response or a sampling.• Repeat at appropriate times.• Consider cost vs. benefits.


Methods for Evaluating Behaviors

• Checklists and observation• Surveys• Interviews (trainee, boss, co-workers,

direct reports)


Behavior Change(3-6 months after program)

Survey or Patterned Interview1. Explain purpose of survey/interview.2. Review program content.3. Ask the program participant:

“To what extent have you changed behavior as a result of the program you attended?”__ Large extent __ Somewhat __ Not at all

If “Large extent” or “somewhat”, please explain:


Behavior Change(3-6 months after program)

4. If “not at all”, please indicate why not:___ Program content wasn’t practical___ No opportunity to use what I learned___ My boss prevented or discouraged me to change___ Other higher priorities___ Other reason (please explain)

5. In the future, to what extent do you plan to change your behavior?

___ Large extent ___ Somewhat ___ Not at allPlease explain:


Guidelines for Evaluating Results

• Measure on a before and after basis. • Allow time for possible results to take place

(possibly 6-12 months). • Repeat at appropriate times. • Use a control group if practical. • Consider short and long term metrics. • Consider “evidence” if “proof” is not possible.


Methods for Evaluating Results

• Business / HR metrics• Survey (manager)• Interview (manager)


Reducing TurnoverScenario:It was determined that the main reason for the high turnover was the

poor way supervisors inducted and trained new employees.A training program was conducted in April on “How to Induct and Train

New Employees”Goal:Reduce monthly turnover to 2% or less

Employee Turnover FiguresTraining program held

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec4% 5% 6% 5% 2% 2% 1% 2% 2% 1% 1% 2%

Proof or evidence?


Increased SalesIncrease in sales for 3 month period over the same

period last year.

Experimental ControlRegions Regions Difference $987,885 ($356,920) $1,344,805

J.P. Huller, Sales Training Manager, Hobart Corporation, Troy, Ohio

Proof or Evidence?


Suggestions for Evaluating Training Programs

1. Measure reaction to all programs• Develop your own form• Gather data• Establish standards of acceptable performance• Improve programs


Suggestions for Evaluating Training Programs

2. Expand evaluation to measure:• Learning• Behavior• Results (including ROE)

3. Consider:• Available evaluation resources• Frequency of programs• Cost of programs• Cost of the evaluation• Executive interest / demand


Register With Us for FREE Resources

Links to:• Online resource library with 40+ items• White paper and article• LinkedIn Kirkpatrick Evaluation discussion group

Subscription to:• Weekly e-newsletter

– Kirkpatrick Quick Tips– Articles– News, specials and events

Go to kirkpatrickpartners.com

Thank you!

[email protected]

(443) 856-4500

m101 and tu101 evaluating training effectiveness: the ... · evaluating training programs 2. expand...

Documents