Mastercard’s Partnership with Metrics That Matter and Degreed

Solving The Mystery of Informal Learning Evaluation

What Is Informal Learning?

Formal Learning Informal Learning

“Traditional” Training

Instructor Led or structured online course

Objectives determined by instructor or course owner

Self-paced

Bite-size

Online

Just-in-time

Objectives determined by learner

Challenges Measuring Informal Learning• No set schedule

• Different success criteria• Higher volume of training

“Informal Learning fills the gap between the classroom and the workplace”

How the Workforce Really Learns

L&D

Le

dSe

lf-

Dir

ect

ed

Daily Weekly Monthly Quarterly Yearly

Web Search

Peer Interactions

Articles & Blogs

Videos

Books & e-Books

Apps

Online Networks

Webcasts & Webinars

Coaching & Mentoring

ILT’s

e-Learning Courses

Conferences & Trade Shows

Live Classes (outside of work)

Online Courses

Podcasts & Audio Books

In- Person Networking

Degreed – How the workforce learns 2016

Traditional L&D is Not enough for Todays Workforce

How likely are you to recommend your employer’s learning and development opportunities to a colleague?

49% Detractors

33% Passives

18% Promoters

-31% Net Promoter Score

Degreed – How the Workforce Learns (2016)

77%

of workers say they do at least some of their learning on a smartphone or tablet

Hybrid Jobs

Source: “The Hybrid Job Economy: How New Skills are Rewriting the DNA of the Job Market” Burning Glass 2018

ExamplesRequire diverse combination of skills

Combine left brain thinking (creative) and right brain thinking

(analytical)

Faster growing and higher paying than “traditional” jobs

Lack Entry Level Opportunities

What does this mean?

Hybrid Job Traits

Marketing Manager with Data Analysis Skills

Engineer with Sales/Business Acumen

Skills

Security Analyst with User Experience Skills

Hybrid Jobs (continued)

Source: “The Hybrid Job Economy: How New Skills are Rewriting the DNA of the Job Market” Burning Glass 2018

$71,000

$78,000

$75,000

$63,000

$100,000

$87,000

$85,000

$81,000

Average Salary with Hybridizing Skill Average Salary without Hybridizing Skill

General Manager

Ski ll: Data Analysis

Premium: 29%

Project Manager

Ski ll: Tableau

Premium: 13%

Civi l Engineer

Ski ll: People Mgmt.

Premium: 12%

Marketing Manager

Ski ll: SQL

Premium: 41%

Digital Learners Aren’t Just Millennials

Expect Technology in Learning

Learn at Own Pace

Access Information “Just in Time”

57%

55%

54%

57%

55%

52%

Millennial

Baby Boomer

n = 9,071 employees.

Source: CEB 2016 L&D Digital Learner Survey.

Note: Millennials are defined as ages 18–34, and baby boomers are defined as ages 55–70.

Mastercard Learning Modalities

Mastercard Learners

What is a Pathway?

300+

Degreed @ Mastercard 2018

85%

28 19kAverage Logins Per

User

Published Pathways

Penetration

Unique Items Accessed

MTM in Pathways

Challenges measuring informal learning: Lower response rates from pathways with less traffic

Why we did it: Obtain feedback from users on informal learning platform

What we did: Strategically placed survey link in pathways

Traf

fic

Feedback

High

Low

PositiveNegative

Data Collection

52Courses Surveyed

2,152Survey Responses

4.15Average Rating

12Pathways Surveyed

1,214Survey Responses

4.24Average Rating

Total Informal Learning

Insights – Payments 101 Pathway vs ILT

Pathway Instructor-Led Session

0 1 2 3 4 5012345

468 responses

Average Overall Score: 4.26

79 responses

6 sessions

Average Overall Score: 4.36

Less than 0.22 point difference in:

Learning Effectiveness

Job Impact

Business Results

Courseware

Insights – Payments 101 Pathway

Positive/Neutral, 71%

Negative, 29%

Chart Title

Natural Language Processing

Sentiment of Comments

Insights from Comments

Goal

<15% Negative

Comments

• Some content US focused

• More interactive, less reading

• Outdated content

Observations/Lessons Learned

Response rates increase by roughly 10% when survey was placed in the

middle of the pathway

-Faster Feedback

-Broken links removed

-Outdated content identified

-Topics identified for next version

Survey Placement Quick Wins

Q&A