measuring whats important to your customers download
TRANSCRIPT
AMA Marketing Effectiveness
Online Seminar Series
Lynette Rowlands
American Marketing Association
Proprietary and Confidential, Maritz Inc. ©
A wealth of information is available for marketing professionals at
www.MarketingPower.com
The #1 marketing site on the web
Proprietary and Confidential, Maritz Inc. ©
Commonly Asked Questions
1. Will I be able to get copies of the slides after the event?
2. Is this web seminar being taped so I or others can view it after the fact?
Proprietary and Confidential, Maritz Inc. ©
Commonly Asked Questions
Yes
1. Will I be able to get copies of the slides after the event?
2. Is this web seminar being taped so I or others can view it after the fact?
Yes
Proprietary and Confidential, Maritz Inc. ©
Introducing Today’s Speaker
Using Importance Measurement to Drive Product Improvements
Keith Chrzan
Vice President, Marketing
Sciences Maritz Research
Proprietary and Confidential, Maritz Inc. ©
Agenda
Introduction Stated Importance Methods Derived Importance Methods Summary
Introduction
Proprietary and Confidential, Maritz Inc. ©
Why Measure Importance?
Products or services can be thought of as bundles of attributes (properties, features, benefits, etc.)
Marketers usually cannot afford to optimize all attributes at once, so they must prioritize
In order to prioritize, marketers must know the relative importance of the attributes, hence the need for importance measurement
May be part of customer satisfaction, image/positioning, brand choice, loyalty, concept testing or other studies
Introduction
Proprietary and Confidential, Maritz Inc. ©
Myers and Alpert’s Clarification
Salience – attribute is easily brought to mind Importance – attribute is important Determinance – attribute is important and
brands differ on it, so that it “determines” preference or choice
Introduction
Proprietary and Confidential, Maritz Inc. ©
Stated Importance
Direct questioning methods e.g. Open-ends, rating, ranking, sorts, etc.
Respondents’ answers directly tell us what attributes are more important than others
Introduction
Proprietary and Confidential, Maritz Inc. ©
Derived Importance
Indirect questions Respondent describes her actual experience Respondent evaluates that experience We infer importance by relating descriptions to the
evaluations We apply statistical predictive models to respondent-
supplied attribute and evaluative judgments and use statistical outputs as measures of importance
Introduction
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods
Unconstrained methods Constrained methods
Derived Importance Methods Summary
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Objectives
As a result of this section, you will be able to Describe two broad classes of stated importance
measures List seven specific kinds of stated importance
measurement Identify the strengths and weaknesses of the various
kinds of stated importance measurement Determine the best stated importance measure for
your project using a decision tree
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods
Unconstrained methods Open-end questions Importance ratings
Constrained methods
Derived Importance Methods Summary
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Open-End Questions
Example: What attributes are important when choosing a <widget>?
Advantages Simple to ask in any survey modality (mail, phone, Web,
etc.) Question does not force response categories on
respondent – researcher may learn about new attributes
Disadvantages Importance and recall are confounded Open-ends may measure salience more than importance
Recommendation: Use for exploratory research, not for importance measurement
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Importance Ratings
Example: When choosing a <widget> how important is—
Not at all ExtremelyImportant Important
Fast service 1 2 3 4 5
Wide variety 1 2 3 4 5Reliability 1 2 3 4 5Ease of use 1 2 3 4 5Country of origin 1 2 3 4 5Package color 1 2 3 4 5
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Importance Ratings
Advantages Easy to ask in any survey modality Respondents are familiar and comfortable answering
importance ratings Clients are familiar and comfortable receiving
importance ratings
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Importance Ratings
Disadvantages Respondents tend to rate most attributes very
positively – this reduces ability to discriminate among them and hampers subsequent analyses
Scale use heterogeneity Some respondents use high part of the scale others use
low part – positional heterogeneity Some respondents use a wider range of the scale than do
others – heteroskedasticity These response effects hampers interpretation and
multivariate analyses and harms cross-cultural comparisons
Recommendation: Use as a last resort
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Importance Ratings
Empirical evidence against importance ratings Discriminant analysis with brand used as dependent
variable and importance ratings as predictors seldom makes sense, and it should, if importances are meaningful
Derived importance works like this: You model some dependent variable (Y) as a linear function of some performance measures (X) and you calculate coefficients that are proxies for importance
If you do the reverse, modeling Y as a function of importance ratings, the coefficients should be measures of performance, but they usually are uncorrelated with actual measures of performance
Importance ratings are next to worthless
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods
Unconstrained methods Constrained methods
Rank order Q-sort Constant sum Method of paired comparisons Maximum difference scaling
Derived Importance Methods Summary
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Rank Ordering
Example: Please rank these 11 features of <widgets>. Give the most important feature a “1,” the next most important feature a “2,” and so on until the least important feature has a rank of “11.”
Advantages Works well on mail, Web or in-person surveys if
attribute list is short Response distribution is constrained so rank
information is standardized for use in cross-cultural comparisons
All respondents’ scores have the same mean All respondents’ scores have the same variance
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Rank Ordering
Disadvantages Difficult to administer in phone surveys Difficult to administer if attribute list is long Non-parametric statistical tests for rank orders are
relatively weak, so less discriminating than even importance ratings
Recommendation: Do not use
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Q-Sort
Example: If 17 attributes are to be evaluated, respondent is instructed to sort them so that– 1 is in a pile for the most important attribute 3 are in a pile for the next most important attributes 1 is in a pile for the least important attribute 3 are in a pile for the next least important attributes 9 are implicitly sorted into the pile of middle
importance
Resulting distribution is 1 : 3 : 9 : 3 : 1
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Q-Sort
Advantages Forced distribution standardizes responses, making
this a viable technique for cross-cultural comparisons Discriminating Works in face-to-face, mail and Web surveys
Disadvantages Will not work in phone interviews Time consuming task if the number of attributes is
large
Recommendation: May use if there are not too many attributes and data collection is not phone
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Constant Sum Allocation
Example: Please assign 11 points to these attributes according to how important they are to you when choosing a <widget>. You can assign some, none, or all 11 points to a given attribute, as long as the total number of points you assign is 11.Fast service _____Wide variety _____Reliability _____Ease of use _____Package color _____Country of origin _____ TOTAL 11
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Constant Sum Allocation
Advantages Ratio measurement of importance Discriminating – trade-off prevents all attributes from
being important
Disadvantages Difficult to do in telephone interviews unless attribute
list is very short Difficult to administer at all if attribute list is long Unknown scale use heterogeneity Unknown ability to standardize cross-cultural studies
Recommendation: Perhaps use with short attribute lists
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Method of Paired Comparisons
From Thurstone (1927) Updated design theory by David (1988) Updated analysis via hierarchical Bayesian
analysis Ask attributes two at a time, forcing respondent
to choose which is more important Ask 1.5 times as many pairs as there are
attributes
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Method of Paired Comparisons
Example: Is ease of use or reliability more important
when choosing a <widget>? Is reliability or package color more important
when choosing a <widget>? Is variety or fast service more important
when choosing a <widget>?
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Method of Paired Comparisons
Advantages Easy to administer in any survey modality VERY discriminating Automatically standardized for cross-cultural
comparisons (no scale-use bias) HB analysis produces individual level importances Ideal for input to needs-based segmentation If well balanced (all attributes occur equally often with
each other) analysis is simple: importance is proportional to percentage of times an attribute is chosen when it is available
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Method of Paired Comparisons
Disadvantages A paired comparison question takes about 50% longer
for respondent to answer in a phone survey, so that a 2 minute importance rating battery becomes a 3 minute paired comparison battery, all else being equal
Recommendation: Good method, use when possible
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Maximum Difference Scaling
Multiple choice extension of MPC 3+ attributes per question; respondent picks
most and least important Example:
Which of the following is least important when you buy a widget and which is most important?
Least Most[ ] Ease of use [ ][ ] Country of origin [ ]
[ ] Package color [ ][ ] Reliability [ ]
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Maximum Difference Scaling
Advantages Handles more attributes with fewer questions than
MPC HB analysis produces individual level importances Even MORE discriminating than MPC Automatically standardized for cross-cultural
comparisons Ideal for input to needs-based segmentation If well balanced, analysis is simple: importance is the
log of the number of times chosen divided by the number of times available
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Maximum Difference Scaling
Disadvantages Requires visual presentation of stimuli (i.e. paper and
pencil or Web survey) May require analysis to produce importances
Recommendation: Good method, use when possible
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Stated Importance Decision Tree
Stated importance
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods Derived Importance Methods Summary
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Objectives
Following this section, you will be able to Identify four types of derived importance models, in
two broad classes Use a decision tree to settle on a derived importance
model for your project Identify the strengths and weaknesses of the various
kinds of derived importance measurement
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Overview - Derived Importance Measures
Respondent evaluates a brand/product using A rating (satisfaction, performance, liking, purchase
intent) A share (market share, share of preference, an
allocation) A choice of one brand/product/therapy over others
Respondent rates product on multiple attributes Predictive statistical model relates attributes to
the evaluation Model yields coefficients as proxies for
importance Correlation and choice-based methods
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods Derived Importance Methods
Correlation-based Methods Correlation Regression True Driver Analysis
Choice-based Methods
Summary
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Variance
Correlation-based methods (correlation, regression, True Driver Analysis) depend on variance patterns Importance just means “shared variance with the
dependent variable” If an attribute doesn’t share variance with a dependent
variable, it can’t be important
Attributes which do not vary, or that vary only randomly, cannot share variance with the dependent variable, and so cannot be important e.g. ‘cost of entry’ attributes like “4 wheels on a car”
or “airline safety”
Proprietary and Confidential, Maritz Inc. ©
Correlation
Description: Bivariate correlations of evaluation with each attribute in turn
Advantages Easy to do Coefficients are unaffected by multicollinearity
Disadvantages Lack of statistical control because attributes are
analyzed in isolation Importance can be double (or triple or more) counted to
the extent attributes are correlated with one another No composition rule for coefficients, so model cannot
support simulation Scale use heterogeneity can spuriously inflate all
correlations
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Correlation
What correlation does
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Multiple Regression Description: All attributes are used simultaneously
to predict the evaluation; coefficients or standardized coefficients are importances
Advantages Accessible technique learned in school that clients are
familiar with Easy to do
Disadvantage - multicollinearity (intercorrelation of independent variables) is omnipresent in survey research and has a pernicious effect on regression coefficients
It can distort the size and even the sign of coefficients Makes coefficients unstable
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Multiple Regression
What regression does
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Multiple Regression Why multicollinearity is bad news for regression
Derived importance
Proprietary and Confidential, Maritz Inc. ©
A Middle Ground
Kruskal came up with a way of computing importance by doing regression analysis and “averaging over orderings”
Theil added an information-theoretic framework for this averaging over orderings
This idea works so well to solve the shortcomings of both correlation and regression that we’ve built it into a family of techniques for importance measurement that we call True Driver Analysis
Derived importance
Proprietary and Confidential, Maritz Inc. ©
True Driver Analysis (TDA)
Advantages Immune to multicollinearity Importances are . . .
Additive Ratio scaled Intrinsically meaningful
Disadvantages Complex programming
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods Derived Importance Methods
Correlation-based Methods Choice-based Methods
MNL
Summary
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Logit vs Regression
In regression, the unit of analysis is a brand and its ratings:
Obs. Overall Att 1 Att 2 Att 3 Att 4
1 4 5 3 2 5
Proprietary and Confidential, Maritz Inc. ©
Logit vs Regression
In MNL, the unit of analysis is a choice, so we collect multiple brands’ ratings:
Obs. Chosen Att 1 Att 2 Att 3 Att 4
1 0 5 3 2 5 1 0 3 2 2 3 1 1 4 5 1 1 1 0 5 4 4 2
Proprietary and Confidential, Maritz Inc. ©
Multinomial Logit (MNL)
Description Evaluation is brand preferred over others (choice or
share) Models this relative preference as a function of
attributes of all competing brands Between-brand differences drive the model (as they
drive choice in the real world) MNL coefficients are significant when differences
between brand ratings relate to brand choice – i.e. it measures determinance
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Multinomial Logit (MNL)
Advantages Model built from between-brand differences in
attribute ratings, so scale use heterogeneity is not a problem
For the same reason, less likely to be affected by multicollinearity
Explicitly includes competitive context Specifically answers the oft-asked question “Why do
customers choose one product over another” (or “competitors’ products over mine,” or “mine over theirs”)
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Multinomial Logit (MNL)
Disadvantages Because we need to ask every respondent about
several brands’ attributes, questionnaire can get long if there’s a lot more in it
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Derived Importance Decision Tree
Proprietary and Confidential, Maritz Inc. ©
Measuring Attribute Importance
Introduction Stated Importance Methods Derived Importance Methods Summary
Derived importance
Proprietary and Confidential, Maritz Inc. ©
Stated Vs. Derived Importance
Fairly often, stated and derived measures give conflicting views of what is “important”
Proprietary and Confidential, Maritz Inc. ©
Back to Myers and Alpert
One reason stated and derived methods may not agree is that they are not even measuring the same thing Open-end measures confound the measurement of
salience and importance Other stated methods may measure importance,
though there is good reason to believe attribute importance ratings do not do so very well
MNL measures determinance, NOT just importance Regression-based derived importance methods
measure a cross-sectional version of determinance
Proprietary and Confidential, Maritz Inc. ©
Stated Vs. Derived Importance
Among stated importance methods— Simpler approaches, like importance ratings and
rankings, have serious shortcomings More complex methods (MPC, MaxDiff) overcome at
least some of the shortcomings MPC for phone surveys MaxDiff for mail, in-person or Web-based surveys
Proprietary and Confidential, Maritz Inc. ©
Stated Vs. Derived Importance
Among derived importance methods— Simple approaches (correlations, regression) just
aren’t good enough More complex methods (TDA, MNL) fix a lot of the
shortcomings TDA for customer satisfaction and concept testing MNL for brand choice/brand preference
Proprietary and Confidential, Maritz Inc. ©
Q & A
Proprietary and Confidential, Maritz Inc. ©
Thanks for your time and participation today!
To replay this webcast (available September 10): go to www.MarketingPower.com/ResearchSeries
For copies of today’s presentation:
www.maritzresearch.com/measurement or (877) 4 MARITZ
To contact today’s speaker:
Keith [email protected]
Questions for AMA: Lynette Rowlands