metrics and myths-pikes peak 2016 · metrics and myths about metrics roy atkinson hdi pikes peak...

Metrics and Myths About Metrics

Roy Atkinson HDI Pikes Peak Chapter

January 20, 2016

About •  HDI Senior Writer / Analyst •  White papers, SupportWorld articles,

research reports, HDIConnect •  15 years experience as a practitioner •  HDI and FUSION Conference Faculty •  International Certification Standards

Committee (ICSC) •  Desktop Support Advisory Board (DSAB) •  Advanced Management Strategy

Tulane University Freeman School of Business

Twitter: @HDI_Analyst | @RoyAtkinson

1.  Measure the things that matter

2.  Understand connections 3.  Assign goals properly 4.  Beware of metrics myths

What We’ll Discuss

Start with Why

Start with Why • Inform and communicate the value of the services • Increase customer satisfaction • Understand how well the people, processes and technology are performing

• Provide the basis for measuring regulatory compliance for SOX, HIPAA, PCI DSS, and others

• Achieve certifications (HDI Support Center Certification, ISO 9000, ISO 20000, COBIT)

• Measure progress towards goals and objectives

This Isn’t Easy Things that are easily measured very rarely get us to what we really want.

— Steve Hultquist, in SupportWorld

Your last full metrics review with stakeholders was •  More than 1 year ago •  6 Months to a year ago •  Last quarter •  Last month

Quick show of hands…

When was the last time you made any major changes to either the metrics or the way they are reported? •  1 Year or more •  6 Months – 1 year •  Last quarter •  Last month

Quick show of hands…

What do you think?

Has the world of IT and/or technical support changed at all in the last 2 years? Will change continue?

Pressure

Recent HDI research showed that 87% of support centers are feeling pressure to show value to business.

Flow of Information

Strategic

Tactical

Operational

Business focus

Requirements for

success

Proof of execution

Based on The Definitive Guide to IT Service Metrics by McWhirter and Gaughan

Focused internally

Flow of Information

Key: Deliver the right information to the right people at the right time in the right way.

Communicating

Beware unintended messages. “We handled 4,000 incidents last month, but handled 5,000 this month.” Message: Things broke 1,000 more times this month.

Communicating

Whenever you change the things you measure or the way you measure, you must communicate clearly to your stakeholders what the changes look like.

Changing Metrics

Strategic Business focus

•  What do executives and leaders want to know?

•  When and how often do they want to know it? •  How do they want the information expressed? •  How do they want the information presented?

Let’s Get More Specific

According to the SAB

The HDI Strategic Advisory Board identified the Shift-Left strategy as an increasing trend.

Shift-Left

Shift-Left means pushing more technical work toward the front line, and repetitive work out into self-service.

Shift-Left

•  Q1: Is it the right plan for your organization?

•  Q2: If yes, how do you measure success?

•  Q3: How will your metrics change?

Shift-Left

Critical Shift-Left Metrics:

•  Level 0 Solvable (LZS) •  First Level Resolution (FLR) •  Mean Time to Resolve (MTTR) •  Customer Satisfaction or Net Promoter

Score™ (CSAT / NPS®) •  Customer Effort Score (CES)

What is LZS? by Rick Joslin

Shift-Left

Critical Shift-Left Metrics:

•  Level 0 Solvable (LZS) •  First Level Resolution (FLR) •  Mean Time to Resolve (MTTR) •  Customer Satisfaction or Net Promoter

Score™ (CSAT / NPS®) •  Customer Effort Score (CES)

then

then

If you Shift-Left, what happens to:

•  Average Handle Time •  Speed to Answer •  Cost per Ticket

Have you thought about…

Nothing Lives in a Vacuum

“Spooky action at a distance”

In 2010, cost per ticket for phone was $20. In 2014, cost per ticket for phone was $14.

Have costs decreased?

Increased Number of Tickets

Percentage of support centers that say the number (volume) of tickets (all channels) has increased:

2009 – 70% 2010 – 67% 2011 – 68% 2012 – 66% 2013 – 66% 2014 – 57% 2015 – 63%

Have costs decreased?

Increased Number of Tickets

•  Cost per Ticket is inversely proportional to the number of tickets

•  If self-service knowledge (Tier 0) succeeds, the number of tickets goes down

•  If the number of tickets goes down, Cost per ticket goes up

How Metrics Affect Each Other

Fully-burdened cost per ticket

(Total costs ÷ Total number of tickets) Total costs = $1.3M/yr | Total # tickets = 52,000/yr

Fully burdened cost per ticket = $25 Total costs = $1.4M/yr | Total # tickets = 76,000/yr

Fully burdened cost per ticket = $18.42

Have costs decreased?

How Metrics Affect Each Other

Quantity / Quality

Quantitative metrics: •  Typically related to volume, time, and/or production; easy

to count or measure •  Quantitative metrics are based on data and information

and can be accurately measured and reported •  Examples: number of calls in queue, number of

abandoned calls, average talk time, after contact work

Quantity / Quality

Qualitative metrics: •  How well someone or something is performing towards

meeting defined guidelines or specifications •  Qualitative metrics can be based on perception or

opinion •  Examples: quality scores, customer satisfaction surveys,

employee satisfaction surveys

For Consideration

•  Number of cases escalated •  Δ in MTTR, escalations vs. FLR•  Number of Incidents vs. Number of Requests •  KM: Number of times articles are reused •  MTBF (Mean Time Between Failures) •  MTBSI (Mean Time Between Service Incidents) •  IUM (Interrupted User Minutes)

Change of Focus

Activities Outcomes

Quantitative Qualitative

How many times are we…? How well are we…?

We resolve tickets. We help produce business results.

1. Support business growth 2. Reduce costs 3.  Improve efficiency 4.  Improve customer service 5. Expand services

Top Spending Priorities 2014-15

Source: HDI 2014 Support Center Practices & Salary Report

Support business growth: Ø Efficiency, effectiveness, capacity Reduce costs: Ø Cost per user; IUM; MTBF Ø  Improve efficiency: Ø  AHT, FLRR, MTTR; Quality scores Improve customer service: Ø CSAT, NPS® or CES; MTTR, IUM Expand services: Ø Capacity (volume), utilization, value

Metrics That Make Sense

Real-World Metrics

MTTFS – Mean Time to Find Someone via Carlos Casanova

MTBCA – Mean Time Before CEO Apologizes MTTFCM – Mean Time To Free Credit Monitoring

via Rick Holland / Forrester

Measuring Support Only

•  Are most incidents due to support, or to something else in the IT organization?

•  Why is the support center generally the only group to measure and report CSAT?

New Question

What if a new question were added to your surveys:

Overall, how satisfied are you with the quality and delivery of IT services?

Incidents

Incidents are the measure of IT quality.

•  Measurement and management •  Industry Standards •  Metrics as goals •  FCR

Metrics Myth-Busting

Metrics Myth-Busting

What we think Deming said: “if you can’t measure it, you can’t manage it” What Deming actually said: “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”

W. Edwards Deming, The New Economics – p. 35 (and no, it’s not Peter Drucker, either)

What is an industry standard? “Generally accepted requirements followed by members of an industry.”

- BusinessDictionary.com

Myth: Industry Standards

This is an industry standard: This is not an industry standard:

FCRR = 72.1% (It’s the average FCRR of respondents to our survey.)

Myth: Industry Standards

Myth: Published Metrics Are Goals

•  HDI’s published metrics are averages or medians (and are labeled as such)

•  They are intended as general industry benchmarks

•  They are not intended to be considered goals or best practices

•  The benchmarks that really count are your own metrics, tracked and trended

•  Customers do want their issues addressed quickly

•  More importantly, they want issues fixed correctly

Is the Fast the Enemy of the Correct?

Is the Fast the Enemy of the Correct?

DIRTFT*

Do It Right the First Time

* “dirtfoot”

A Closer Look at FCR

FCR is, in reality, mostly a measure of fixing known and repeated issues .

Contact

Solution

Resolution

A Closer Look at CSAT

•  Send surveys to people who contact you •  Get X% response •  Calculate CSAT (or NPS® or CES)

I do not think this metric means what you think it means.

Inconceivable!

A Closer Look at CSAT

•  Send surveys to ~60% of the user base •  Get 20% response •  Calculate CSAT (or NPS® or CES)

20% x 60% = 12% HDI 2015 Stat: CSAT = 87%

87% x 12% = 10.44% So, <11% are satisfied

Misleading Numbers

Your customer/user base

Those who contact you

Survey respondents

Satisfied customers

Your customer/user base

Satisfied customers

Misleading Numbers

Meaningful numbers

Meaningful numbers

About 70% of the calls that come to the desk are about known issues.

A Closer Look at FCR

About 30% of calls are for password resets

What happens if you say, “We must achieve 75% FCRR next month?”

Myth: We Should Set Goals on Activities

Is It Performance?

Metrics Backfires

“Stores will be rated on whether or not they have all their shelf-talkers displayed.”

Metrics Backfires

“Stores will be rated on the

cleanliness of their

restrooms.”

Creative Commons: David Woo

Quality

•  DIRTFT – True FCR (as close to 0 reopens / repeats as

possible) – Properly categorized and documented

•  Low IUM (Interrupted User Minutes) IUM = Length of Interruption [minutes] × Number of Affected Users

Mileposts

The metric is not the goal. The metric is only a milepost that helps you measure your progress

towards your goal.

“Don’t put goals on activities; put goals on outcomes.”- Phil Verghis

Don’t Forget

roy.atkinson@ubm.com

http://www.thinkhdi.com/topics/library/white-papers.aspx