kirk patrick four levels
TRANSCRIPT
-
8/6/2019 Kirk Patrick Four Levels
1/12
A Kirkpatrick White Paper
April 20April 200909
The Kirkpatrick Four Levels:A Fresh Look After 50 Years
1959 - 2009
By Jim Kirkpatrick, PhDand Wendy Kayser Kirkpatrick
April 2009
-
8/6/2019 Kirk Patrick Four Levels
2/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
A Golden Anniversary Celebration
The Kirkpatrick Four Levels are turning 50 this year! November1959 marked the first time Don Kirkpatrick published histhoughts on training evaluation with a series of four articles in the
Journal of the ASTD. Don will tell you he did not call themlevels, and he did not coin the phrase Kirkpatrick Four LevelEvaluation Model. He did, however, use the legendary fourwords that comprise the foremost training evaluation modeltoday: Reaction, Learning, Behavior, and Results.
On the anniversary of the four levels, we thought it would beappropriate to first honor Don and his work, and second to sharewith you how the four levels have both evolved and remainedrelevant over the last half century.
The Day of Reckoning Has Arrived
When we conduct workshops on the Kirkpatrick Model all overthe world, we find a surprising number of misconceptions aboutit. After 50 years, we figured it was time to set the record straight.Rather than attempt to clarify the many misunderstandings aboutthe four levels, we will focus on the salient elements of themodel, and show you how to leverage its power at a time whenthe day of reckoning has arrived for the entire learning industry.
Dons incredible vision is evident in a quote he obtained from acolleague back in 1959:
Managers, needless to say, expect their manufacturingand sales departments to yield a good return and will goto great lengths to find out whether they have done so...likewise, training directors might be well advised to takethe initiative and evaluate their programs before the dayof reckoning arrives.
TheExperimental Evaluation of Management Training:Principles and Practice, Daniel M. Goodacre III, TheB.F. Goodrich Company, Personnel, May 1957
This statement represents what the ultimate intent of the fourlevels of evaluation was then, and is today: to show the businessvalue and worth of training.
Donald L. Kirkpatrick, PhD
The ultimate intent ofthe Kirkpatrick Four
Levels was then, and istoday:
To show the businessvalue and worth of
training.
2009. All rights reserved
-
8/6/2019 Kirk Patrick Four Levels
3/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
Before unveiling our new Kirkpatrick Model visual, we willpresent what we consider to be the five foundational principles.The following are the cornerstones of this model as adifferentiator from others, and the keys to successfulimplementation of training programs. Some will likely be asurprise to you.
Five Foundational Principles
1. The end is the beginning
Most learning professionals have heard of the four levels, andmany can recite them. But relatively few know how to effectivelyget beyond Level 2. When presenting these concepts to groupsof professionals, we often refer to current evaluation practice assmile sheets (L1), pre and posttests (L2), and hope for the best(L3 and L4). Most of our workshop participants know exactlywhat we are saying.
In 1993, Don wrote his first book on the Four Levels to explainthe entire model (which is, and has always been, morecomprehensive than four simple levels). On page 26 ofEvaluating Training Programs: The Four Levels (1
stEdition,
Berrett-Koehler, 1993), Don writes:
Trainers must begin with desired results and thendetermine what behavior is needed to accomplish them.Then trainers must determine the attitudes, knowledge,and skills that are necessary to bring about the desiredbehavior(s). The final challenge is to present the trainingprogram in a way that enables the participants not only
to learn what they need to know but also to reactfavorably to the program.
It is unfortunate that the message above has been missed bymany learning professionals. For decades, practitioners haveattempted to apply the four levels aftera program has beendeveloped and delivered. It is difficult, if not impossible, to createsignificant training value that way.
All this is said to distinguishthe development of the planto buildeffective programs and evaluation methodology (which startswith Results, as shown to the right) from the actual datacollection, which doesbegin with Level 1 and works forwardthrough Level 4.
Kirkpatrick Four Levels
Level 4:Results
To what degree
targeted outcomes
occur, as a result of
the learning event(s)
and subsequent
reinforcement.
Level 3:Behavior
To what degreeparticipants applywhat they learnedduring training whenthey are back on the
job.
Level 2:Learning
To what degreeparticipants acquirethe intendedknowledge, skills, andattitudes based on
their participation inthe learning event.
Level 1:Reaction
To what degreeparticipants reactfavorably to thelearning event.
2009. All rights reserved
-
8/6/2019 Kirk Patrick Four Levels
4/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
2. Return on Expectations(ROE) is the ultimate indicatorof value
What normally happens when executives ask for new training?Despite what they might say, most learning professionals jump tothe task by all-too-quickly retreating to their departments and
commencing to design and develop suitable programs. While acursory needs assessment may be conducted, it is rarely takento the point that expectations of the training contribution arecompletely clear.
Much of the training and consulting that the Kirkpatricks doinvolves helping learning professionals negotiatekey businessstakeholder expectations. This is a process where learningprofessionals ask questions to clarify and refine the expectationsof the key business stakeholder so that they are satisfying to thelatter and realistically achievable to the former. Learningprofessionals then need to convert those rather genericexpectations into observable, measurable success outcomes byasking the question, What will success look like to you? Thosesuccess indicators then become the Level 4 outcomes thetargets to which you can sharply focus your collective efforts toaccomplish return on stakeholder expectations.
3. Business partnership is necessary to bring aboutpositive ROE
Note in the last sentence, we wrote, collective efforts.Contraryto training myth and deep tradition, we do not believe thattraining events in and of themselvesdeliver positive, bottom lineoutcomes. Much has to happen beforeand afterformaltraining.Learning professionals need help to do it, and that help needs tocome from our business partners. Historically, the comfort zone
performance processes.
Before training, learning professionals need to partner withsupervisors and managers to prepare participants for training.Supervisors should explain to their direct reports what they will
arn in training, why its importa
of learning professionals has been Levels 1 and 2. We believethis is one of the major reasons why many learning professionalsspend almost all of their time there. But the actual executionoflearning programs and overall corporate strategy occursprimarily at Level 3.
2008 research by Bersin and Associates shows us that as muchas 70% of employee learning occurs on-the-jobin many differentways. Thus, we needto reinventour roles bybecoming expertsin and deeply involved with learning that extends beyondournormal comfort level, or we run the risk of losing our positions.
Therefore, not only do we need to call uponour businesspartners to help us identify what success will look like, but we will
need a cooperative effort throughout the learning and
nt, expectations for actions
08
leduring and after the training event, and the kind of support theywill receive throughout the process.
Return OnExpectations (ROE):Focusing training and
reinforcement efforts onthe stated bottom-line
expectations ofbusiness partners, withthe goal of deliveringthe key outcomes /results they expect.
Josh Bersin and Associates, 20
Where Learning Takes Place
Prior to
being
"trained":
20%
During
"training"
10%On the job:
70%
2009. All rights reserved
-
8/6/2019 Kirk Patrick Four Levels
5/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
Even more critical is the role of the supervisor or managthe training. They are the
er afterkey people to reinforce newly learned
knowledge and skills through support and accountability. Thedegree to which this reinforcement and coaching happensdirectly correlates to improved performance and positiveoutcomes.
It is often said that a picture is worth a thousand words. To theright is one of our signature pictures that we believe accuratelyillustrates the concept of business partnership with thebusiness on the left, training on the right, and the bridge as ourway of crossing over.
This is no easy sell to many executives and managers, becausewe have all been in cahoots to perpetuate the myth that goodtraining leads to positive results on its own. Going forward, wemust make a business case to our key stakeholders that weneed them to work with us at different points along the learningand performance continuum. Business partnership, not the
delivery of training programs, is the secret to positive outcomes.
4. Value must be created before it can be demonstrated
A colleague of ours, Sandy Almeida, MD, MPH, recently sent usa summary of her research that identified statistical correlationsbetween the four levels. There is good correlation betweenLevels 1 and 2, in that positive learner engagement led to ahigher degree of learning. Similarly, the correlation betweenLevels 3 and 4 was significant if employees consistentlyperform critical on-the-job behaviors, individual and overallproductivity increased. There was not, however, a significantcorrelation between Levels 2 and 3. In short, Sandy states thateven providing excellent training does notlead to significant
transfer of learning to behavior and subsequent results without agood deal of deliberate and consistent reinforcement.
Consider those findings in combination with a study conductedby Dr. Brent Peterson at Columbia University in 2004. Hecompared the amount of time that is spent developing trainingand related activities, and what actually contributes to learningeffectiveness. He found that the typical organization invests 85%of its resources in the training event, yet those events onlycontributed 24% to the learning effectiveness of the participants.The activities that led to the most learning effectiveness were
e training event.
we are putting mostof our time intog training (L1 and L2) andter of the benefit. And we are
e follow-up activities thatnge and subsequent resultsng programs to deliver.
Activities Contributing to
Learning Effectiveness
Typical Learning Investment
Dr. Brent Peterson,Columbia University, 2004
Pre-
Work
26%
Learning
Event
24%
Follow-
Up 50%
Pre-
Work
10%
Learning
Event
85%
Follow-
Up 5%
follow-up activities that occurred after th
What does this mean? Thatdesigning, developing and deliverinonly getting about one-quarspending virtually noneof our time on thtranslate into positive behavior cha(L3 and L4) that we intend our traini
2009. All rights reserved
-
8/6/2019 Kirk Patrick Four Levels
6/12
2009. All rights reserved
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
were
rned
h, and an ineffective culture of follow-up and
That is one of the biggest reasons why a comprehensive ASTD
study in 2006 identified the fact that more than 70% of training
failure comes aftertraining is completed. The major factors
that employees didnt get a chance to apply what they lea
oon enougscoaching existed. Bottom line? It is essential that learning
professionals redefinetheir roles and extend their expertise, their
involvement, their influence, their impact, and their value into
Levels 3 and 4 before the day of reckoning arrives.
5. A compelling chain of evidencedemonstrates yourbottom line value
What do we mean by corporate jury? We believe that each ofus individuals, departments, our entire industry is on trial,accused of not bringing enough value to the bottom line to
justifyour costs. And in all cases, there is either an individu
group of business leaders who are sitting in judgment of our
al or
erformance and our impact to the business.
sok
y
tribution
e last
pOur friend and colleague, Nick DeNardo from Edward Jones,refers to data collected at Levels 1 and 2 as consumptivemetrics and data at Levels 3 and 4 as impact metrics. If weare only providing our jury members with numbers of programand attendees (L1 and L2 data) we are basically shouting, Lohow much this is costing you! and, thus, have a bulls eyepainted on our backs. If, on the other hand, we work our wabackwards from Level 4 and include L3 and L4 metrics,particularly for mission critical programs, we are saying just asloudly, Look how much value we are bringing.
A chain of evidence consists of data and information thatsequentially connects the four levels and shows the con
Consumptive metrics:Level 1 and Level 2data indicating howmuch time and how
many resources havebeen invested in
training
Impact metrics:
Level 3 and Level 4e
and tangible results thattraining has delivered
data indicating the valulearning has made to the business. We recently wrote a bookentitled Training On Trial(AMACOM, 2009) that details theseand other principles of what we call the Kirkpatrick BusinessPartnership Model. We talk about how we do notbelieve inattempting to isolate the impact of training in order to prove ourvalue. Instead, we advocate presenting a chain of evidence thatillustrates the value of the entire business partnership effort.
The New Kirkpatrick Model
It is now time to pull the wraps off not only the trueKirkpatrickmodel, but also the completemodel. While it contains the sameideas that Don (and we) have been communicating over th50 years, we felt a more complete illustration of the model wouldhelp both learning professionals and their business partners tocreate more training value together.
-
8/6/2019 Kirk Patrick Four Levels
7/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
Training Needs Analysis: The End is the Beginning
a formal connection between theresolving a business problem, or taking
nd is the beginning because you aremine what success will look like (L4), and then
. We identify critical behaviors as the few,ir jobs in order to bring about thee the actions and processes that are
programs, etc.) that are designed andheir critical behaviors.
for participants to be able to performaddress the learning environment and
conditions that will support learning effectiveness and enjoyment (L1). We define environment as theonditions as prerequisites for ultimate success (i.e., corporate
ulture, participant readiness, etc.).
ctive, it is time to get your instructionaleen identified as necessary to bringis is als the time to determine the besteasurement plan you will use. Next
cted earlier. This is alsoel 1 and Level 2. This is probably familiar territory for most
arning professionals.
Start in the upper left corner of the model. Note that it starts withbusiness and learning, whereby a request is made to work towardadvantage of a market opportunity. As mentioned already, the eworking from L4 to L1 in the first four steps. You will detersubsequent critical behaviors and organizational drivers (L3)targeted behaviors that program graduates must apply back at theidentified outcomes. Organizational drivers, on the other hand, arcarried out by others (e.g., coaches, mentors, peers, softwareimplemented to reinforce the participants to perform t
Next, we deal with the required KSAs or competencies neededtheir new on-the-job behaviors (L2). Finally, we subsequently
venue and modality for training, and cc
Once the four levels have been dealt with from a planning perspedesigners working on the actual program or programs that have babout the targeted outcomes. Importantly (and often forgotten), thway to measure each of the four levels, and build the tools and mcomes the delivery of the program(s), in whatever modality and venue you sele
hen formal evaluating begins with Lev
o
wle
2009. All rights reserved.
-
8/6/2019 Kirk Patrick Four Levels
8/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
Entering New Territory For Many LearningProfessionals
drivers, and preliminary outcomes (L4). Use the data to makesure that behaviors and driversare meeting standards to ensurepositive outcomes, and provide your stakeholders jurymembers with a periodic dashboard to show them initiativeprogress.
During this step is also the time to partner with managers andsupervisors who are responsible for ongoing coaching and
reinforcement. Offer any assistance you can and follow up withthem frequently. While this step in the process looks like onesmall box, it actually represents months or even years of effort.This is where you will see 50% of learning effectiveness occur.Its where you can prevent 70% of the potential learning failuresif done properly. In short, we believe ongoing reinforcement and
The diagram (now moving back from right to left) starts to moveinto new territory for many learning professionals at this point.The next step is called initiate ongoing reinforcement andmonitoring. This has everything to do with what happens afterlearning events, and deals with what we talked about under theprinciple of creating value. It is true that what gets measured,gets done, and the best suggestion we have here is to monitorand measure the critical behaviors (L3), the organizational
monitoring is among the most critical steps in the entire model!
The gray box on the left side of the diagram is a reminder toanalyze your findings along the way, and make positiveadjustments where you can to eliminate snags at Levels 1, 2, or3. If you determine that conditions are limiting the ability ofassociates to learn or apply the information, now is the time to
call attention to this and create a backup plan. There is no valuein going through the entire process to find out at the end thatintended results arent achieved because of something that couldhave been corrected months earlier!
Building Your Chain of Evidence
pefully) this value-filled and
and compelling presentation.
When you are satisfied with your preliminary findings at each ofthe levels, its time to prepare final data and testimonials that willmake up your closing argument to the jury. Presentation ofyour chain of evidence goes from right to left, starting with Level1 and moving step by step to create your value story throughLevel 4, Results. Ultimately (and ho
evidence-based story will lead to a positive verdict: one whereyour stakeholders say that beyond a reasonable doubt, theoverall learning initiative has successfully contributed to theintended results. For mission critical cases (programs), be sureto practice how you will make your oral presentation to your jury.It is always sad for us to see training departments that do greatwork fall short of impressing their stakeholders because theydidnt make an effective
Ongoingreinforcement and
monitoring
May reflect monthsor years of effort
50% of learningeffectiveness occurs
here 70% of potentiallearning failures occurhere if not doneproperly
2009. All rights reserved.
-
8/6/2019 Kirk Patrick Four Levels
9/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
The Four Levels Are Not Just For Training Evaluation
t the Four Levels are used only tobe used for this purpose (and most
ally an integral part of every phase of a
show that programs begin byen each of the other threodel. In fact, all fo
Another commonly held misconception we hope to debunk is thameasure the impact of training after it is complete. While they can
typically are by learning professionals), the Four Levels are actutraining initiative, from beginning to end.
We have illustrated this by shading the new Kirkpatrick Model toconsidering what Results the business wants to accomplish. Thconsidered. And this pattern continues all the way through the m
e levels is
ur levels are addressedat almost every step in the process! To reflect this reality, we areKirkpatrick Four Level Evaluation Model, as this is, to some degrename of the model is simply the Kirkpatrick Model.
no longer referrie, a misnomer. As yo ew
ng to the model as theu can see, the n
2009. All rights reserved.
-
8/6/2019 Kirk Patrick Four Levels
10/12
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
The Window Washers
y taxi-atching post (but I did snap his picture as I walked away).
The next day, I was in the country of Brunei, conducting anotherworkshop. During a break, I wandered over to this young man
and asked him,
What is your job here at the resort?
Chai (who later introduced himself to me by name), stoppedwhat he was doing, walked over to me, looked me in the eye andsaid;
I am part of the team that creates exceptionalexperiences for our guests.
After I picked my jaw off the ground, we carried on quite aconversation, which included Chai asking me about my stay, if I
had everything I needed, what I thought of hisgrounds, and whatI thought of hiscountry.
On my way back to the U.S., I thought about Window Washer #1and Chai. They were both about the same age, so why weretheir answers to the same question so different? Of course, mythoughts went to training and reinforcement. I never found outwhat kind of training Window Washer #1 received (he probablytook a job down the street when he was offered a modest raise).Chai, on the other hand, told me about his orientation, training,and the coaching and encouragement he received from hissupervisor. He was taught that he was in training in order tolearn, perform, enhance his career possibilities, and ultimately
serve as an ambassadorfor hisresortand his country.
short, Chai received training and reinforcement that modeledhat we have presented here. While his windows were no
s, he knew that the purpose ofose were tied to the bigger
ows. He made meto return.
Window Washer #1
Chai
ets put some real faces on what we have talked about. In theL
right margin there are photos of two men who seemingly havethe same job. The first is a man that I (Jim) saw in front of a hotelin Asia. While I was waiting for a taxi, I went up to him and askedhim,
What is your job here at the hotel?
Without looking at me, he answered,
I wash windows.
ince we had little else to talk about, I went back to mSw
Inwcleaner than Window Washer #1
aining and development and hispurpicture. So he did more than just clean wind
feel like a welcome guest so I would want
trp
2009. All rights reserved.
-
8/6/2019 Kirk Patrick Four Levels
11/12
The Kirkpatrick Four Levels: A Fresh Look After 50 YearsAn Urgent Call to Action
o there you have it. A new visual for the Kirkpatrick Model that
ith the recent changes in the economy and the world, there isno bette the learning
dustry. Allow me to explain with a personal story. We have af our home. As I (Jim)
as walking up the path shortly after we were married, I noticedthe form the footprints of (I assume) theirmall child, and carved his or her initials into the sidewalk.
use,sked Wendy where a screwdriver was, and proceeded back
You, ho rk in the learningdustry. During this time of economic challenge, and when our
ble challenges. You have the unique opportunity torovide significant answers for them and to carve out a new
learning tners.Fortuna t last forever. Thisreates a limited time frame and urgency for you to take action
e says it all:
to
ning
ng us
Swe hope better conveys the entire process and essence of whatDon Kirkpatrick designed 50 years ago. The sound, timelessprinciples upon which it is based. And now, a challenge and
opportunity for you.
Wr time to create a new standard within
incement walk leading to the front door ow
er owners depresseds
Upon seeing this legacy left for all to see, I went into the hoaoutside to add my own mark. Well, obviously the cement haddried since 1995. I was too late to leave my mark on the front
walk.
wever, are nottoo late to leave your mainindustry is under fire, the good news is that business leadersour jury members are looking everywhere for solutions to theirincredip
legacy by becoming true strategic business partely, these economic times won
cbefore the cement dries (or the rock bridge collapses).
Following one of our training sessions, I received an e-mail fromone of the participants. Sh
This was one of the most insightful and worthwhiletraining events that Ive taken in awhile. I look forwardreading more of your work your father and you arevery inspiring. Too often we think training design anddevelopment and we dont spend nearly enough time inevaluation at the beginning. This training was so eye-opening you made a Brunei Window Washer out ofme.
Kat Siedlecki, Booz Allen Hamilton, OS LearTeam
hanks, Don, for a very useful 50 years. Thanks for showiTa path to creating true value to the business world, and thepeople we serve. Now it is up to all of us to heed your warning,and effectively apply what you and others have taught us.
2009. All rights reserved.
-
8/6/2019 Kirk Patrick Four Levels
12/12
2009. All rights reserved.
The Kirkpatrick Four Levels: A Fresh Look After 50 Years
most sought after speakers in the
rd
w.linkedin.com/in/donaldkirkpatrick
About Don Kirkpatrick
Don Kirkpatrick, PhD, is professor emeritus at the University ofWisconsin. He is the creator of the Kirkpatrick Four LevelEvaluation Model, the foremost training evaluation tool in the
world. He is one of theworkplace learning and performance field.
Don is the author of numerous books on training evaluation andHR topics.
Don is a past president of the American Society for Training andDevelopment (ASTD). He has been honored with the LifetimeAchievement in Workplace Learning and Performanceawafrom ASTD, and is a member of the HRD Hall of FameofTraining Magazine.
Read more about Don at
http://ww . Contact him [email protected] .
ing
the
im has co-written three books with his father, Don Kirkpatrick,
ead more about Jim at smr-usa.com
About the Authors
JimKirkpatrick, PhD, is the Vice President of Global Trainand Consulting for SMR USA. He presents workshops andprovides consulting to Fortune 500 companies around the world,including Harley-Davidson, LOreal, Ingersoll Rand, Honda,Royal Air Force, and GE Healthcare.
J
the creator of the Kirkpatrick Model.
R andhttp://www.linkedin.com/in/kirkpatrickfourlevelevaluation. Con
him atj .
Kayser Kirkpatrick is the director of Kirkpatricks, LLC. She is a certified instructional designer. Wendyts seminars and facilitates corporate meetings. Shen 16 years of expe
WendyPartnerconducdraws o rience in the business world to makeher training releRead more about Wendy at kirkpatrickpartners.com
vant and impactful with measurable results.
andlinkedin.com/in/wkkirkpatrick. She welcomes your comments.Please contact her [email protected].
Jim and Wendy have written a new book entitled Training Onrial, to be published by AMACOM books in late 2009.T