scorecards, learning metrics and measurement strategies
DESCRIPTION
TRANSCRIPT
You can listen to today’s webinar using your computer’s speakers or you may dial into the teleconference.
If you would like to join the teleconference,
please dial 1.866.469.3239 and enter access code xxx xx xxx.
You will be on music hold until the seminar begins.
Scorecards, Learning Metrics and Measurement Strategies
#CLOwebinar
Speaker: Dennis Bonilla Senior Vice President GP Learning Solutions Group
Andrew Snoey Senior Program Manager, Operations Microsoft Corporation
Moderator: Kellye Whitney Managing Editor Chief Learning Officer magazine
Scorecards, Learning Metrics and Measurement Strategies
#CLOwebinar
• Q&A
– Click on the Q&A panel (?) in the bottom right corner
– Type in your question in the space provided
– Click on “Send.”
Tools You Can Use
#CLOwebinar
• Polling
– The poll will appear on the right side of your screen
– Select the best option for each question
– Click on “Submit”
#CLOwebinar
Tools You Can Use
Frequently Asked Questions 1. Will I receive a copy of the slides after the webinar?
YES
2. Will I receive a copy of the webinar recording? YES
Please allow up to 2 business days to receive these materials
#CLOwebinar
#CLOwebinar
Kellye Whitney Managing Editor Chief Learning Officer magazine
Scorecards, Learning Metrics and Measurement Strategies
#CLOwebinar
Dennis Bonilla Senior Vice President GP Learning Solutions Group Andrew Snoey Senior Program Manager, Operations Microsoft Corporation
Scorecards, Learning Metrics and Measurement Strategies
LEARNING METRICS, SCORECARDS MEASUREMENT STRATEGIES
9
POLL QUESTION
Where does your role fit in your Learning and Development Organization? • Not at all
• Instructional Designer
• Instructor
• Training Coordinator
• Training Manager
• In the Line of Business
10
POLL QUESTION
Does your organization formally track and report learning metrics?
• Not at all
• Sometimes
• I Don’t Know
• Yes
11
POLL QUESTION
Does your organization formally track and report learning metrics?
• Not at all
• Sometimes
• I Don’t Know
• Yes
If you answered yes to the previous poll, are the metrics:
• Just Learning Process and Event Statistics (butts in seat, courses completed, etc….)
• Tied to Organizational Business Outcomes
• Both
SECTION 1 – CRITERIA FOR DEVELOPING EFFECTIVE SCORECARDS
12
SCORECARD OPERATING PRINCIPLES
• Drive Business Leadership • Aim for Simplicity • Set Realistic, But Stretched, Targets • Ensure Consistency in Taxonomy and Calculations
• Ensure Similar Metric Types have Similar Thresholds • Align Related Scorecards’ Metrics and Targets
13
IS LEARNING MAKING A DIFFERENCE?
• Are business objectives being met?
• Business leaders are demanding more linkages to business outcomes and measurements of results, yet it’s the outcome least often measured.
• Huge confusion between internal process metrics and actual results.
• Sponsors want to see evidence of IMPROVED BUSINESS PERFORMANCE, not internal training operations metrics.
14
SECTION 2 – DISCUSS THE EVOLUTION OF LEARNING MEASUREMENT FRAMEWORKS
15
16
POLL QUESTION
How does your organization / company measure the quality of the training programs for your key strategic initiatives?
• Not at all
• Data collection measuring attendee satisfaction
• Data collection measuring attendee competency
• Data collection measuring manager satisfaction in attendee performance
• Data collection measuring business impact after course completion
WHAT METHODS ARE USED IN EVALUATION?
• Evaluation is traditionally represented as the final stage in a systematic approach to training
• More recent Instructional Systems Design (ISD) models incorporate evaluation throughout the process, not as end state
• There are six general approaches to educational evaluation with Goal-based and Systems-based approaches as the predominant modes used in the evaluation of training programs
• Various frameworks have been proposed under the influence of these two approaches
• The most influential framework has come from Kirkpatrick which follows the Goal-based approach and the most commonly used today
17
KIRKPATRICK LEVELS
18
BASELINE INFO SUCH AS STUDENT PRESENCE AND PARTICIPATION
ATTENDANCE LEVEL 0
STUDENT PERCEPTIONS OF THE COURSE AND ITS VALUE
REACTIONS LEVEL I Most organizations have
Level 0 to I
STUDENT UNDERSTANDS AND CAN RECALL PRIMARY CONCEPTS FROM THE TRAINING
LEARNING LEVEL II
STUDENTS’ BEHAVIORAL CHANGE RESULTING IN PERFORMANCE AT A HIGHER LEVEL
TRANSFER LEVEL III Assessing Level II and III takes
more time and effort –this is where our certification
program is located
BUSINESS IMPACT AS A CONSEQUENCE OF TRAINING
RESULTS LEVEL IV
But Level IV is the only place where we can easily perform
an ROI calculation
SECTION 3 –ALIGNING TRAINING MEASURES WITH BUSINESS MEASURES
19
TALK ABOUT A DISCONNECT
Data from the Forum Corporation, Customer Driven Training Organization 20
0
1
2
3
4
BUSINESS LINKAGE REGISTRATION & ALIGNMENT
PROGRAM DELIVERY MEASUREMENT
WHAT EXECUTIVES WANT
TALK ABOUT A DISCONNECT
Data from the Forum Corporation, Customer Driven Training Organization 21
0
1
2
3
4
BUSINESS LINKAGE REGISTRATION & ALIGNMENT
PROGRAM DELIVERY MEASUREMENT
CURRENT EFFORT ALLOCATION WHAT EXECUTIVES WANT
MEASUREMENT STRATEGIES
• 1st Question that should always be asked: “Does this metric actually measure an outcome for which the training program was created?”
• For some, it could be a behavior change; for others, improved perception by stakeholders, and others a quantitative measure of business impact (revenue, market share, market position, etc.).
• Measure what matters to the customer in a way that leads to informed action.
• Don’t pick a process before defining the purpose. Let form follow function!
22
MEASUREMENT STRATEGY CORE PRINCIPLES
• Relevant
• Credible and Compelling
• Simple and Consistent
• Efficient
23
“For every complex problem there is an answer that is clear, simple and wrong!”
– H.L. Mencken
24
POLL (TRIVIA) QUESTION
When was the first, documented, formal use of an educational program evaluation conducted in the USA?
• 1792
• 1815
• 1897
• 1916
• 1958
“NOT EVERYTHING THAT CAN BE COUNTED, COUNTS.” So What Counts?
• Outcomes that are of interest to the business to which training contributed in a demonstrable way.
• Sponsors may not be as wedded to proof of financial ROI as many HR professionals assume.
• An evaluation that provides sound evidence to support informed decisions that are in the best interest of the organization.
• They prove to an appropriate level of certainty that the program did or did not achieve the desired outcomes and they provide insights to improve subsequent courses or iterations; actionable Information (Bersin)
25
PRESENTING TRAINING RESULTS Power of Narrative
• “Stories are easier to remember – because in many ways, stories are how we remember.” Pink (2005): A Whole New Mind
• Stories bring statistics to life
• Stories will be repeated and retold
Sell the Sizzle: It doesn’t matter how good your results are if no one knows about them.
Results x Marketing = Perception of Value
26
SECTION 4 – SHARING EXAMPLES OF MICROSOFT’S EFFORTS
27
PROJECT PICASSO In the Fall of 2009, the Microsoft training organization to applied our measurement principles to our course assessment survey.
Remember • Relevant: questions refer to key course success metrics à eliminated non-action-
oriented questions • Simple: shortened the question quantity to improve response rate • Consistent: aligned format, wording, and calculations à minor wording revisions
Compelling: added business impact questions
• Manager engagement on the material • Different actions will the employee take due to the course
28
OLD SURVEY NEW SURVEY
OLT 12 + 4 6 + 4
cILT 11+7 12 + 4
vILT 11+7 12 + 4
PROFESSIONAL CERTIFICATION
29
• Microsoft has an extensive certification program for internal and external candidates using third-party testing organizations
• Our technical staff can be certified in 18 tracks; when looking across the technologies and levels, resulting in 64 discrete certifications
TECHNICAL EMPLOYEES
• Sales staff was taught the “soft” (professional) skills, but how do you measure the knowledge?
• In January 2010, Microsoft started an effort to train Sales & Margeting staff through an academy program that would have certifications associated with certain skills
SALES & MARKETING EMPLOYEES
http://www.microsoft.com/learning/en/us/certification/cert-default.aspx
SALES AND MARKETING ACADEMY Our Sales & Marketing Academy is part of our People focus
Purpose • Accelerate sales and marketing staff readiness • Promote management coaching and re-enforcement
Options considered • Certification of the growth of a skill • Certification of the growth of a skill set • Separate certifications for each skill • Certify high-priority skills
Built
• Curriculum roadmaps • Course knowledge assessments – demand pass rate of 80% • Single certification activity across all courses in a level involving case study/simulation,
live work product review, role play, oral review panel
30
Basic Competent Master
Business Management A101 A201 A301
Customer Relationship Management B101 B201 B301
Products & Services C101 C102
-- C301
Organizational Awareness
A101 A102
B201 D201
--
… E101 B201 --
… F101 F201 F301 F302
MEASURING TRAINING IMPACT Almost two years ago, Microsoft started an effort to measure the business impact of our core sales training curriculum (CST)
Key questions asked • Does CST increase the Account Manager’s annual quota attainment? • Does CST increase Customer Satisfaction? • Does CST increase the Account Manager’s pipeline velocity?
Tried to control for • Region / country
• Tenure / experience level • Account Manager’s satisfaction level with CST courses • Account Manager's self-assessed impact of CST courses
31
ALIGNING TRAINING TO A BUSINESS SCORECARD
32
Metric Color?
Root Cause?
Y/R
G
OTHER PLANS
OTHER
DEVELOP CORPORATE SCORECARD
Sell $ Share % Cust Sat #
SALES & MARKETING EXECUTES TO STRATEGIC INITIATIVES
QUARTERLY SCORECARD RESULTS
Course Metric A Metric B Metric C Metric D Metric E
Sell… X X X
Market… X X
Present… X
Create… X
Close… X X
FORMULATE RESPONSE PLAN 1. Drive existing course(s) 2. Bolster manager reinforcement 3. Re-vamp existing course(s) 4. Develop new course(s)
ASSESS TRAINING ADOPTION
TRAINING
Feedback Loops
SUMMARY
33
• Align Your Scorecard with the Desired Business Results
• Do Not Abandon Measurement at the Smiley Face
• Count Training Results, Not Effort
LET’S CONTINUE TALKING
Andrew Snoey Senior Program Manager, Operations Microsoft Corporation [email protected] 425-722-8510 office
Dennis Bonilla Senior Vice President, Learning Solutions General Physics Corporation [email protected] 425-636-3410 office 702-885-5814 mobile
Join Our Next CLO Webinar
To the Agile Go the Spoils: Why Learning Must Evolve
Tuesday, June 21, 2011
CLO Webinars start at 2 p.m. Eastern / 11 a.m. Pacific
Register at www.clomedia.com/events Join the CLO Network: http://network.clomedia.com/
#CLOwebinar