Dr. Scott E. Sampson
Dr. Scott Sampson
James M. Passey Professor of
Service Operations
Brigham Young University
Provo, Utah, USA
Performance
Measurement and
Customer Feedback
rev 3/4/14
Performance Measurement and Customer Feedback 1
Dr. Scott E. Sampson
Measurement
“What gets measured,
gets managed.”Is this a good
thing or a bad thing?
Performance Measurement and Customer Feedback 2
Dr. Scott E. Sampson
How do we measure goods quality?
Performance Measurement and Customer Feedback 3
Dr. Scott E. Sampson
How do we measure service quality?
Performance Measurement and Customer Feedback 4
Ch_CF_instrument_examples
Dr. Scott E. Sampson
The risk of measuring people
Performance Measurement and Customer Feedback 5
Dr. Scott E. Sampson
The risk of measuring people
Performance Measurement and Customer Feedback 6
Dr. Scott E. Sampson
The risk of measuring people
Performance Measurement and Customer Feedback 7
Dr. Scott E. Sampson
The risk of measuring people
You get what you measure!
Performance Measurement and Customer Feedback 8
But what do we measure?
Dr. Scott E. Sampson
Measuring productivity
Performance Measurement and Customer Feedback 9
Dr. Scott E. Sampson
Driver
process domain
interaction Independent processing
Toyota
process domain
Independent processing interaction
Measuring productivity?
produce
car
Performance Measurement and Customer Feedback 10
Dr. Scott E. Sampson
Driver
process domain
interaction Independent processing
Toyota
process domain
Independent processing interaction
Measuring productivity?
drive car
Performance Measurement and Customer Feedback 11
Dr. Scott E. Sampson
Driver
process domain
interaction Independent processing
Toyota
process domain
Independent processing interaction
Measuring productivity
produce
carsell car drive car
Performance Measurement and Customer Feedback 12
Dr. Scott E. Sampson
Provider’s Process Domain Customer’s Process Domain
Independent processing Surrogate interaction Direct interaction Direct interaction Surrogate interaction Independent processing
Measuring Productivity
completed
units
meeting
design
standard
met needs
“realizing value”
meeting needs efficiently
creating “value
potential”
Performance Measurement and Customer Feedback 13
Dr. Scott E. Sampson
Provider’s Process Domain Customer’s Process Domain
Independent processing Surrogate interaction Direct interaction Direct interaction Surrogate interaction Independent processing
Measuring Productivity
Performance standards?
• recollection of facts
• ability to follow procedures
• ability to make rule-based
decisions
• “knowledge”
Performance Measurement and Customer Feedback 14
Dr. Scott E. Sampson
Provider’s Process Domain Customer’s Process Domain
Independent processing Surrogate interaction Direct interaction Direct interaction Surrogate interaction Independent processing
Measuring Productivity
Performance standards?
• recollection of facts
• ability to follow procedures
• ability to make rule-based
decisions
• “knowledge”
Performance Measurement and Customer Feedback 15
Questions:
Are we training students to do things that can
be automated? For example…
• Instead of teaching calculus should we be
teaching people how to effectively use
Wolfram Alpha?
• Instead of teaching accounting skills should
we spend our time writing better accounting
software?
• Instead of even teaching should we be
spending our time developing courseware?
Dr. Scott E. Sampson
Provider’s Process Domain Customer’s Process Domain
Independent processing Surrogate interaction Direct interaction Direct interaction Surrogate interaction Independent processing
Measuring Productivity
Alternate performance standards?
• ability to deal with decision ambiguity
• critical thinking
• ability to innovate
• social skills and persuasiveness
• ability to work in teams
Performance Measurement and Customer Feedback 16
Dr. Scott E. Sampson
What gets measured, gets managed
“Teaching”• easy to measure
• easy to automate
“Education”• difficult to measure
• difficult to automate
Performance Measurement and Customer Feedback 17
Performance standards?
• recollection of facts
• ability to follow procedures
• ability to make rule-based
decisions
• “knowledge”
Alternate performance standards?
• ability to deal with decision ambiguity
• critical thinking
• ability to innovate
• social skills and persuasiveness
• ability to work in teams
Do we measure what is convenient
and quantifiable or what we want?
Dr. Scott E. Sampson
review CF-instrument-examples
Challenges with
customer
measurement
Performance Measurement and Customer Feedback 18
Dr. Scott E. Sampson
Six challenges with customer measurement
1. Subjective Rulers
2. Intrusive Measurement
3. Resistance to Measurement
4. The Halo Effect
5. Self-Selected Sampling
6. Interpreting the Interpretations
Performance Measurement and Customer Feedback 19
Dr. Scott E. Sampson
1. Subjective Rulers
• “Customer measures of quality are generally
subjective.”
• “Making a subjective measure numerical does not
make it objective.”
• How do we analyze this?
• Does a mean make sense?
How would you rate our food?
1. poor
2. ok
3. good
4. very good
5. exceptional
Performance Measurement and Customer Feedback 20
Dr. Scott E. Sampson
2. Intrusive Measurement
• “The act of customer-measurement of quality can
influence perceptions.”
• Hawthorn effect.
1. Was there anything about
your stay that we could have
improved?
2. How would rate our service?
Performance Measurement and Customer Feedback 21
Dr. Scott E. Sampson
3. Resistance to Measurement
• “Most customers do not consider quality measurement to
be value adding, therefore resist providing
measurements.”
• “This resistance increases as the customers’ cost of
providing measurement increases.”
Performance Measurement and Customer Feedback 22
Dr. Scott E. Sampson
4. The Halo Effect
• “Customers automatically combine individual components
of quality into an overall quality perception.”
• “Attempted measurements of individual components may
actually have more to do with the overall perception than
the individual components.”
How would you rate his…
1.leadership?
2.intelligence?
3.diplomacy?
4.wit?
5.singing voice?
Performance Measurement and Customer Feedback 23
Dr. Scott E. Sampson
5. Self-Selected Sampling• “Customer-measurement makes it is possible to influence
sampling, but very difficult to control sampling.”
• “When we control sampling we know how survey-responding customers (sampled) compare with customers in general.”
“Strategies to increase
response rates, such
as awards or drawing
for prizes, influence
some types of
customers more than
others.”
“Therefore, it is
important to consider
sample bias.”
Performance Measurement and Customer Feedback 24
Dr. Scott E. Sampson
5. Self-Selected Sampling
• Sample bias
Performance Measurement and Customer Feedback 25
Dr. Scott E. Sampson
6. Interpreting the Interpretations • “Customer measurement requires the customer to interpret both their perceptions and the measurement scale.”
• “Two customers with identical perceptions might interpret the measurement scale differently, resulting in different measurements.”
How did we do?
☐poor
☐ok
☐good
☐excellent
How did we do?
☐poor
☐ok
☐good
☐excellent
Performance Measurement and Customer Feedback 26
Dr. Scott E. Sampson
Performance Measurement and Customer Feedback 27juncture
Dr. Scott E. Sampson
Customer
measurement ROI
Performance Measurement and Customer Feedback 28
Dr. Scott E. Sampson
Typical customer measurement systems
• “…in a study of 22 large ‘customer driven’
companies, [TARP] found that the companies
spent a median of $1 million and employed the
equivalent of 13 full-time professional staff per
year on customer feedback systems.”
“TARP researchers report that
‘many of these companies
have little to show for the
investment’”
Why not?source: Sampson, S. E. 1999, “An Empirically Defined Framework for Designing Customer
Feedback Systems,” Quality Management Journal, Vol. 6, No. 3, pp. 64-80.
Performance Measurement and Customer Feedback 29
Dr. Scott E. Sampson
An inconvenient truth…
Quality customer
feedback requires an
investment, therefore
should produce a return.
Performance Measurement and Customer Feedback 30
Dr. Scott E. Sampson
Developing an
effective customer
feedback system
Performance Measurement and Customer Feedback 31
Dr. Scott E. Sampson
Designing effective customer feedback systems
• Step 1…
Begin with
the end in
mind!
Plan your work
then work your
plan!
Performance Measurement and Customer Feedback 32
Dr. Scott E. Sampson
Designing effective customer feedback systems
• Where?
• Why?
• What?
• How?
• When?
• Who?
Performance Measurement and Customer Feedback 33
Dr. Scott E. Sampson
Designing effective customer feedback systems
• Where?
• Why?
• What?
• How?
• When?
• Who?
Functional target?
• design
• marketing
• operations
• customer support
Feedback goal?
(see CFSD
tables)
Performance Measurement and Customer Feedback 34
Service-
Product
Design
Goals
Marketing
Goals
Operations
Goals
Customer
Support
Goals
Ascertain
Customer
Needs
Identify
Customers
Measure
Specification
Conformance
Evaluate
Customer
Satisfaction
Develop
Service
Product
Promote
Service
Product
Compare
Across
Organization
Assure
Repeat
Business
Benchmark
Versus
Competition
Generate
Sales
Leads
Increase
Employee
Involvement
Provide
Sounding
Board
Dr. Scott E. Sampson
Designing effective customer feedback systems
• Where?
• Why?
• What?
• How?
• When?
• Who?
Functional target?
• design
• marketing
• operations
• customer support
Feedback goal?
(see CFSD
tables)
Objective frame?
• short-term
• long-term
Questioning?
Response type?
• dichotomous
• scale
• list (to check)
• field (fill-in-blank)
• open-ended
Data use?
• quantitative
• qualitative
Bias sensitive?
yes ↔ no
Solicitation type?
active ↔ passive
Extent of questioning?
brief ↔ extensive
Feedback incentive? Visibility?
Temporal frame?
• pre-process
• in process
• post-process
Channeling?
• who collects?
• who reviews?
• who responds?
• who follows up?
Performance Measurement and Customer Feedback 35
Dr. Scott E. Sampson
Designing effective customer feedback systems
• Where?
• Why?
• What?
• How?
• When?
• Who?
Functional target?
• design
• marketing
• operations
• customer support
Feedback goal?
(see CFSD
tables)
Objective frame?
• short-term
• long-term
Questioning?
Response type?
• dichotomous
• scale
• list (to check)
• field (fill-in-blank)
• open-ended
Data use?
• quantitative
• qualitative
Bias sensitive?
yes ↔ no
Solicitation type?
active ↔ passive
Extent of questioning?
brief ↔ extensive
Feedback incentive? Visibility?
Temporal frame?
• pre-process
• in process
• post-process
Channeling?
• who collects?
• who reviews?
• who responds?
• who follows up?
Performance Measurement and Customer Feedback 36
Dr. Scott E. Sampson
Summary
• No royal road to effective service measurement.
• Understand challenges of measuring customers.
• Take a systematic approach.
• Begin with the end in mind!
Performance Measurement and Customer Feedback 37
Dr. Scott E. Sampson
Performance Measurement and Customer Feedback 38end
Dr. Scott E. Sampson
Sample goals of customer feedback
Performance Measurement and Customer Feedback 39
Service-Product
Design Goals
Marketing
Goals
Operations
Goals
Customer
Support Goals
Ascertain
Customer Needs
Identify
Customers
Measure
Specification
Conformance
Evaluate
Customer
Satisfaction
Develop Service
Product
Promote Service
Product
Compare
Across
Organization
Assure
Repeat
Business
Benchmark
Versus
Competition
Generate
Sales
Leads
Increase
Employee
Involvement
Provide
Sounding Board
Dr. Scott E. Sampson
CF Goals relating to Service-Product Design
Performance Measurement and Customer Feedback 40
Goal(S)hort-term and (L)ong-term
Objectives
Survey Question Examples
(showing response types*)
How to Use the Feedback
(sensitive to response bias?)
Ascertain Customer
Needs
(S) Assess: Determine the
strength of various customer
needs being met.
How important are each of the
following product features... (S)
Means and histograms. Use to
help prioritize service product
improvement efforts. (yes)
(L) Probe: Determine important
needs potentially addressed by
the service product.
What features are important for
this type of service product? (L
or F) What other needs we
might serve? (F, O)
Use as a discussion topic for
focus groups with customer
groups. (no)
Develop Service
Product
(S) Check: Assure that the
service product is meeting the
objectives it was designed to
meet.
How would you rate us
according to the following
objectives... (S)
Means and histograms.
Identify service product
weaknesses or fail points. (yes)
(L) Prescribe: Identify ideas for
future features or service
products.
Suggestions? Possible
improvements? (O)
Use as seed ideas in the design
process. (no)
Benchmark Versus
Competition
(S) Compare: Determine how
our company is viewed relative
to the competition.
How would you rate us relative
to the competition... (S)
Means and histograms. Can
help identify where we need to
study the competition. (no)
(L) Borrow: Determine ways in
which the competition provides
a superior service product.
What do you particularly like
about our competitors? (O)
Use as guidance for studying
competitors’ service products.
(no)
*Question response types: Dichotomous (e.g., yes/no), Scale (e.g., 1 to 7), List to check, Field (Name/address/phone), Open ended, Any.
Dr. Scott E. Sampson
CF Goals relating to Marketing
Performance Measurement and Customer Feedback 41
Goal(S)hort-term and (L)ong-term
Objectives
Survey Question Examples
(showing response types*)
How to Use the Feedback
(sensitive to response bias?)
Identify Customers
(S) Describe: Determine what
type of people our customers
are.
Age? Income? (other
demographic questions) (L, F)
Means and histograms. Assist
in targeting and advertising.
(yes)
(L) Trace: Discover why/how
our customers came to be our
customers.
How did you select us? What
features were major selling
points? (L or O)
Analyze effectiveness of
various advertising and sales
efforts. (yes)
Promote Service
Product
(S) Impress: Give the
perception that the company
cares about customers and
quality.
(any questions about perceived
quality and meeting customer
needs) (A)
Respond to feedback, when
possible (i.e. have name and
address or phone). (no)
(L) Publicize: Collect
exceptional service product
reviews for future publicizing.
Comments about our
company? Anything you
consider exceptional? (O)
Publish excerpts in company
and other publications. (no)
Generate
Sales
Leads
(S) Push: Invite customers to
repurchase.
Which of these (service
products) can we send you
more information about? (L, N)
Forward to sales department.
(no)
(L) Enroll: Encourage
customers to join a frequent
purchase club or mailing list.
Would you like to join our
(membership club or mailing
list)? (N)
Forward to sales department.
(no)
*Question response types: Dichotomous (e.g., yes/no), Scale (e.g., 1 to 7), List to check, Field (Name/address/phone), Open ended, Any.
Dr. Scott E. Sampson
CF Goals relating to Operations
Performance Measurement and Customer Feedback 42
Goal(S)hort-term and (L)ong-term
Objectives
Survey Question Examples
(showing response types*)
How to Use the Feedback
(sensitive to response bias?)
Measure Specification
Conformance
(S) Inspect: Determine if the
service product performed to
formal specifications.
Did (such-and-such service
product feature) occur? (D)
Feed failure information back to
employees responsible for
providing that feature. (no)
(L) Track: Track areas of
potential variance to assure the
process remains in control over
time.
Rate the following features we
provide... (S)
Use standard control-chart
analysis to identify if and when
the process goes out of control.
(yes)
Compare
Across
Organization
(S) Focus: Help focus
management on company sites
or departments in need of
attention.
What department/location did
you visit? Comments? (look for
problems) How do they rate...?
(F, O, S)
Summarize data by
department/ location, looking
for patterns, outliers, and high
variance. (no)
(L) Baseline: Develop
performance standards based
on best practices occurring in
the organization.
What department/location did
you visit? (L) How did they do
in these areas... (F, S)
Summarize data and distribute
to each department/location,
highlighting top results. (yes)
Increase Employee
Involvement
(S) Motivate: Provide the basis
for rewarding or punishing
employees for the service
product they provide.
Ratings for... (employees or
their work). (S) Employee to
nominate for recognition?
Why? (F, O)
Recognize or reward employ-
ees for top ratings or significant
nomination by customers. (no)
(L) Empower: Direct
employees in development of
their area of responsibility.
How might we improve...
(specific aspects of the service
product) (O)
Direct the feedback to
employees, who then report
how it is handled. (no)
*Question response types: Dichotomous (e.g., yes/no), Scale (e.g., 1 to 7), List to check, Field (Name/address/phone), Open ended, Any.
Dr. Scott E. Sampson
CF Goals relating to Customer Support
Performance Measurement and Customer Feedback 43
Goal(S)hort-term and (L)ong-term
Objectives
Survey Question Examples
(showing response types*)
How to Use the Feedback
(sensitive to response bias?)
Evaluate Customer
Satisfaction
(S) Reassure: Receive
assurance that customers are
generally satisfied.
Did we meet your expectations?
(D) How do we rate overall? (S)
Means and histograms. Com-
pare with target rating values.
(yes)
(L) Protect: Watch for customer
problems that could potentially
become serious (such as legal
liability).
Complaints? Comments?
(focusing on complaint
responses) (O)
Investigate complaints, espec-
ially reoccurring. Resolve prob-
lems before they turn serious.
(no)
Assure
Repeat
Business
(S) Recover: Attempt to
reconcile and reclaim
dissatisfied customers.
Any problems with...? Were you
satisfied with...? (D, N)
Contact customers when
possible and offer apology and
remuneration. (no)
(L) Retain: Determine reasons
customers defect to prevent
future defections.
What might cause you to
choose the competition? (O)
How likely is that to occur? (S)
Build checks into the service
product to assure defection-
inducing problems are avoided.
(no)
Provide Sounding
Board
(S) Hear: Allow customers the
opportunity to speak their
minds.
Comments? (O) When possible, thank customer
for feedback. Apply other
objectives as appropriate. (no)
(L) Communicate: Provide the
opportunity to interact with
customers.
Questions or comments? (O)
Would you like a response? (D)
Name and address? (N)
Tell the customer how you acted
on the feedback, and possibly
ask for more feedback. (no)
*Question response types: Dichotomous (e.g., yes/no), Scale (e.g., 1 to 7), List to check, Field (Name/address/phone), Open ended, Any.
Dr. Scott E. Sampson
Practice ExerciseBest way to collect student feedback?
Performance Measurement and Customer Feedback 44
Dr. Scott E. Sampson
Best way to collect student feedback?
1. Determine how to improve courses?
2. Help promote courses to future students?
3. Identify the best instructors?
4. Measure student satisfaction?
5. Motivate faculty to be more responsive?
6. Uncover how to make courses more engaging?
7. Help course meet more student needs?
Performance Measurement and Customer Feedback 45
Service-Product Design Goals Marketing Goals Operations Goals Customer Support Goals
Ascertain Customer Needs Identify Customers Measure Specification Conformance Evaluate Customer Satisfaction
Develop Service Product Promote Service Product Compare Across Organization Assure Repeat Business
Benchmark Versus Competition Generate Sales Leads Increase Employee Involvement Provide Sounding Board
Develop Service Product
Promote Service Product
Compare Across Organization
Evaluate Customer Satisfaction
Increase Employee Involvement
Benchmark Versus Competition
Ascertain Customer Needs
Dr. Scott E. Sampson
Best way to collect student feedback?
1. Determine how to improve courses?
2. Help promote courses to future students?
3. Identify the best instructors?
4. Measure student satisfaction?
5. Motivate faculty to be more responsive?
6. Uncover how to make courses more engaging?
7. Help course meet more student needs?
Performance Measurement and Customer Feedback 46
Service-Product Design Goals Marketing Goals Operations Goals Customer Support Goals
Ascertain Customer Needs Identify Customers Measure Specification Conformance Evaluate Customer Satisfaction
Develop Service Product Promote Service Product Compare Across Organization Assure Repeat Business
Benchmark Versus Competition Generate Sales Leads Increase Employee Involvement Provide Sounding Board