metrics and myths-pikes peak 2016 · metrics and myths about metrics roy atkinson hdi pikes peak...
TRANSCRIPT
Metrics and Myths About Metrics
Roy Atkinson HDI Pikes Peak Chapter
January 20, 2016
About • HDI Senior Writer / Analyst • White papers, SupportWorld articles,
research reports, HDIConnect • 15 years experience as a practitioner • HDI and FUSION Conference Faculty • International Certification Standards
Committee (ICSC) • Desktop Support Advisory Board (DSAB) • Advanced Management Strategy
Tulane University Freeman School of Business
Twitter: @HDI_Analyst | @RoyAtkinson
1. Measure the things that matter
2. Understand connections 3. Assign goals properly 4. Beware of metrics myths
What We’ll Discuss
Start with Why
Start with Why • Inform and communicate the value of the services • Increase customer satisfaction • Understand how well the people, processes and technology are performing
• Provide the basis for measuring regulatory compliance for SOX, HIPAA, PCI DSS, and others
• Achieve certifications (HDI Support Center Certification, ISO 9000, ISO 20000, COBIT)
• Measure progress towards goals and objectives
This Isn’t Easy Things that are easily measured very rarely get us to what we really want.
— Steve Hultquist, in SupportWorld
Your last full metrics review with stakeholders was • More than 1 year ago • 6 Months to a year ago • Last quarter • Last month
Quick show of hands…
When was the last time you made any major changes to either the metrics or the way they are reported? • 1 Year or more • 6 Months – 1 year • Last quarter • Last month
Quick show of hands…
What do you think?
Has the world of IT and/or technical support changed at all in the last 2 years? Will change continue?
Pressure
Recent HDI research showed that 87% of support centers are feeling pressure to show value to business.
Flow of Information
Strategic
Tactical
Operational
Business focus
Requirements for
success
Proof of execution
Based on The Definitive Guide to IT Service Metrics by McWhirter and Gaughan
Focused internally
Flow of Information
Key: Deliver the right information to the right people at the right time in the right way.
Communicating
Beware unintended messages. “We handled 4,000 incidents last month, but handled 5,000 this month.” Message: Things broke 1,000 more times this month.
Communicating
Whenever you change the things you measure or the way you measure, you must communicate clearly to your stakeholders what the changes look like.
Changing Metrics
Strategic Business focus
• What do executives and leaders want to know?
• When and how often do they want to know it? • How do they want the information expressed? • How do they want the information presented?
Let’s Get More Specific
According to the SAB
The HDI Strategic Advisory Board identified the Shift-Left strategy as an increasing trend.
Shift-Left
Shift-Left means pushing more technical work toward the front line, and repetitive work out into self-service.
Shift-Left
• Q1: Is it the right plan for your organization?
• Q2: If yes, how do you measure success?
• Q3: How will your metrics change?
Shift-Left
Critical Shift-Left Metrics:
• Level 0 Solvable (LZS) • First Level Resolution (FLR) • Mean Time to Resolve (MTTR) • Customer Satisfaction or Net Promoter
Score™ (CSAT / NPS®) • Customer Effort Score (CES)
What is LZS? by Rick Joslin
Shift-Left
Critical Shift-Left Metrics:
• Level 0 Solvable (LZS) • First Level Resolution (FLR) • Mean Time to Resolve (MTTR) • Customer Satisfaction or Net Promoter
Score™ (CSAT / NPS®) • Customer Effort Score (CES)
then
then
If you Shift-Left, what happens to:
• Average Handle Time • Speed to Answer • Cost per Ticket
Have you thought about…
Nothing Lives in a Vacuum
“Spooky action at a distance”
In 2010, cost per ticket for phone was $20. In 2014, cost per ticket for phone was $14.
Have costs decreased?
Increased Number of Tickets
Percentage of support centers that say the number (volume) of tickets (all channels) has increased:
2009 – 70% 2010 – 67% 2011 – 68% 2012 – 66% 2013 – 66% 2014 – 57% 2015 – 63%
Have costs decreased?
Increased Number of Tickets
• Cost per Ticket is inversely proportional to the number of tickets
• If self-service knowledge (Tier 0) succeeds, the number of tickets goes down
• If the number of tickets goes down, Cost per ticket goes up
How Metrics Affect Each Other
Fully-burdened cost per ticket
(Total costs ÷ Total number of tickets) Total costs = $1.3M/yr | Total # tickets = 52,000/yr
Fully burdened cost per ticket = $25 Total costs = $1.4M/yr | Total # tickets = 76,000/yr
Fully burdened cost per ticket = $18.42
Have costs decreased?
How Metrics Affect Each Other
Quantity / Quality
Quantitative metrics: • Typically related to volume, time, and/or production; easy
to count or measure • Quantitative metrics are based on data and information
and can be accurately measured and reported • Examples: number of calls in queue, number of
abandoned calls, average talk time, after contact work
Quantity / Quality
Qualitative metrics: • How well someone or something is performing towards
meeting defined guidelines or specifications • Qualitative metrics can be based on perception or
opinion • Examples: quality scores, customer satisfaction surveys,
employee satisfaction surveys
For Consideration
• Number of cases escalated • Δ in MTTR, escalations vs. FLR• Number of Incidents vs. Number of Requests • KM: Number of times articles are reused • MTBF (Mean Time Between Failures) • MTBSI (Mean Time Between Service Incidents) • IUM (Interrupted User Minutes)
Change of Focus
Activities Outcomes
Quantitative Qualitative
How many times are we…? How well are we…?
We resolve tickets. We help produce business results.
1. Support business growth 2. Reduce costs 3. Improve efficiency 4. Improve customer service 5. Expand services
Top Spending Priorities 2014-15
Source: HDI 2014 Support Center Practices & Salary Report
Support business growth: Ø Efficiency, effectiveness, capacity Reduce costs: Ø Cost per user; IUM; MTBF Ø Improve efficiency: Ø AHT, FLRR, MTTR; Quality scores Improve customer service: Ø CSAT, NPS® or CES; MTTR, IUM Expand services: Ø Capacity (volume), utilization, value
Metrics That Make Sense
Real-World Metrics
MTTFS – Mean Time to Find Someone via Carlos Casanova
MTBCA – Mean Time Before CEO Apologizes MTTFCM – Mean Time To Free Credit Monitoring
via Rick Holland / Forrester
Measuring Support Only
• Are most incidents due to support, or to something else in the IT organization?
• Why is the support center generally the only group to measure and report CSAT?
New Question
What if a new question were added to your surveys:
Overall, how satisfied are you with the quality and delivery of IT services?
Incidents
Incidents are the measure of IT quality.
• Measurement and management • Industry Standards • Metrics as goals • FCR
Metrics Myth-Busting
Metrics Myth-Busting
What we think Deming said: “if you can’t measure it, you can’t manage it” What Deming actually said: “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”
W. Edwards Deming, The New Economics – p. 35 (and no, it’s not Peter Drucker, either)
What is an industry standard? “Generally accepted requirements followed by members of an industry.”
- BusinessDictionary.com
Myth: Industry Standards
This is an industry standard: This is not an industry standard:
FCRR = 72.1% (It’s the average FCRR of respondents to our survey.)
Myth: Industry Standards
Myth: Published Metrics Are Goals
• HDI’s published metrics are averages or medians (and are labeled as such)
• They are intended as general industry benchmarks
• They are not intended to be considered goals or best practices
• The benchmarks that really count are your own metrics, tracked and trended
• Customers do want their issues addressed quickly
• More importantly, they want issues fixed correctly
Is the Fast the Enemy of the Correct?
Is the Fast the Enemy of the Correct?
DIRTFT*
Do It Right the First Time
* “dirtfoot”
A Closer Look at FCR
FCR is, in reality, mostly a measure of fixing known and repeated issues .
Contact
Solution
Resolution
A Closer Look at CSAT
• Send surveys to people who contact you • Get X% response • Calculate CSAT (or NPS® or CES)
I do not think this metric means what you think it means.
Inconceivable!
A Closer Look at CSAT
• Send surveys to ~60% of the user base • Get 20% response • Calculate CSAT (or NPS® or CES)
20% x 60% = 12% HDI 2015 Stat: CSAT = 87%
87% x 12% = 10.44% So, <11% are satisfied
Misleading Numbers
Your customer/user base
Those who contact you
Survey respondents
Satisfied customers
Your customer/user base
Satisfied customers
Misleading Numbers
Meaningful numbers
Meaningful numbers
About 70% of the calls that come to the desk are about known issues.
A Closer Look at FCR
About 30% of calls are for password resets
What happens if you say, “We must achieve 75% FCRR next month?”
Myth: We Should Set Goals on Activities
Is It Performance?
Metrics Backfires
“Stores will be rated on whether or not they have all their shelf-talkers displayed.”
Metrics Backfires
“Stores will be rated on the
cleanliness of their
restrooms.”
Creative Commons: David Woo
Quality
• DIRTFT – True FCR (as close to 0 reopens / repeats as
possible) – Properly categorized and documented
• Low IUM (Interrupted User Minutes) IUM = Length of Interruption [minutes] × Number of Affected Users
Mileposts
The metric is not the goal. The metric is only a milepost that helps you measure your progress
towards your goal.
“Don’t put goals on activities; put goals on outcomes.”- Phil Verghis
Don’t Forget
