product testing: methodological issues & design considerations
TRANSCRIPT
© 2014 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not be disclosed or reproduced without the prior written consent of Ipsos.
Product Testing: Methodological Issues & Design Considerations
October 30, 2014
1
Contents
2
What Business Issue to Address
Unbranded vs. Branded Test
Monadic vs. Sequential Monadic Design
Central Location Test vs. In-Home Use Test
2
3
4
What Business Issue to Address
3
4
Fact: Product Testing is critical for all Successful FMCG clients
Consumers need to evaluate the physical product
R&D still needs explicit guidance as to how to develop winning formulations
Clients still need a trusted partner given the associated investment
Maximize Sales & ROI
Launch
Develop & Optimize Product, Pack, Price
Generate &Screen Ideas
Identify & Screen Insights
Develop Brand /Category Strategy
Define Long-Term Brand Vision
MID STAGELATE STAGE
POST LAUNCH FUZZY FRONT ENDDefine Innovation
Platforms
Product Testing
Lifecycle
Evaluate ConceptsFinalize
Proposition
Monitor Performance
Build ConceptsExploration
Guidance Validation
Benchmarking
Cost Savings / QI
The Innovation Process
5
Iden
tify
Obj
ecti
ves
6
Align research objectives not only by stage but by type of product development
Research Goals: Explore & prioritize Identify next generation product
features & benefits
Exploration1
EARLY STAGE
Benefits:Capture consumer input at the earliest stages of development to uncover
critical product features to fill development pipeline
7
Benefits:Working with R&D / Sensory, use efficient designs and Sensory / technical
data to identify drivers of liking to guide next stages of development
Research Goal:Screen & optimize prototypes
2
Exploration
Guidance2
EARLY STAGE
8
Benefits: Fine tuning minor product enhancements Disaster Check; Identify “red flags” Prepares the product for market with realistic sense of success
9
Focuses on: Assessing consumer feedback
for a small set of products Confirming acceptance of an
optimized product (from Early Stage)
Finding the best product Understanding overall fit and /
or shifts in performance
Validation3
Benefits: Track Product Health; early warning system for production Use variation in production as basis for quality control and product improvement Monitor competition, identify points of difference & track internal quality processes
Focuses on:assessing quality control and in-market tracking of product quality over time
Benchmarking
10
4
Consumer complaints have spiked & sales are declining. How can we pinpoint where / when the problem is occurring?
How does my leading sku perform and / or benchmark against the leading in-market competitor?
We have just launched a new sku, how can we track performance post launch to monitor & ensure year 1 success?
What are clients asking at Benchmarking stage?
11
4
Cost Savings / Quality Improvement (QI)
12
5Focuses on:products that are currently in the market which may not need enhancements to improve consumer acceptance but require ingredient or process modification to extend their life in the market
Benefits: Identifying cost savings opportunities An approach to business decision making coupling dollars at risk with
consumer perceptions of preference
What are clients asking at the Cost Savings / Quality Improvement stage?
13
5
Can consumers detect differences when a new ingredient is used in an existing formula?
How can we increase profitability by introducing a cost reduced prototype without causing risk to the franchise?
Is the new formula an equally acceptable replacement for the current product and how can we be sure that it will be successful in-market?
Setting Action Standard
14
• Is consistent with Brand Performance Standard; e.g., Client’s internal protocol
• Is agreed PRIOR to test and aligned with client• Include a benchmark; i.e., current product, main competitor, or
norms (rarely)• Have a defined measure; e.g., mean scores, Top Box, Top 2 Boxes• Have a set of defined attributes; e.g., Overall Liking, Purchase
Intent, Uniqueness, product attributes, Preference • Define Statistical Test and Confidence Level to use• Define Target group; e.g., at total sample
New Product Development• Mean Score of Priced Purchase Intention of the new product is rated at
least parity (at 90% confidence level) with the competitor• Mean Scores of Overall Liking, Uniqueness, and attributes “makes me
feel beautiful” and “makes my skin fairer” of the new product are rated significantly higher (at 95% confidence level) vs. the competitor
Benchmarking• Mean Score of Overall Liking is rated at least parity (at 90% confidence
level) with the competitor
Cost Savings • Overall preference of the new prototype should be at least parity (at
90% confidence level) vs. the current product
Examples of Action Standards
15
16
VS
UNBRANDED
BRANDED
Design Considerations:Unbranded
vs. Branded Test
IPSOS
SOS
IPSOSIPSOS
IPSOSIPSOS
IPSOSIPSOS
IPSOSIPIPSOS
IPSOS
IPSOS
UNBRANDEDUnbranded product evaluations are appropriate at early / Guidance stages of research when the focus is solely on product formulation acceptance and / or for screening prototypes in the absence of brand or positioning.
What’s important ?
Unb
rand
ed v
s.
Bran
ded
BRANDEDBranded product evaluations are used at later / Validation stages of research when the focus is on holistic understanding of product performance, in the context of brand equity and / or new brand / concept positioning.
What’s important ?
Will respondents be made aware of brand(s)?
17
Unbranded Evaluations
18
The evaluation or ratings provided for an object presented to respondents when the object is not identified either through packaging or other labeling
The purpose of the blinding or brand concealment is to remove from respondent consideration the effect of branding on the object’s evaluation
This is an attempt to obtain an evaluation that focuses on object characteristics unaffected or unbiased by the influence of the brand and the image it conveys
Early stage concept or idea screening A manufacturer is exploring
entering a new category and does not want their current brand image to impact perceptions
Early stages of product development
Focus on comparing product formulations Screening prototypes Tuning formulas
Identifying product characteristics that drive key measures
What Why
Branded Evaluations
19
The evaluation or ratings provided for an object presented to respondents when the object is identified or branded
The intent is to allow the branding and its associated imagery to affect or influence perceptions of the object characteristics
The effects of branding can mask or obscure the differences between objects
Specifically, differences that are due to the influence of physical or sensory characteristics, positioning and / or attributes
Later stage evaluations when the product already has a full positioning
Understanding the impact of brand on product evaluations
Cost Reduction
When products cannot be de-branded
Cost prohibitive to blind the product
Consider evaluations among current brand users and non-users to understand impact of the brand
What Why
IPSOS
SOS
IPSOSIPSOS
IPSOSIPSOS
IPSOSIPSOS
IPSOSIPIPSOS
IPSOS
IPSOS
Can we test these products unbranded?
20
Lifebuoy Bar Soap Roma Kelapa
Dove Bar Soap Oreo
ProChiz
21
Design Considerations:Monadic
vs. Sequential Monadic
Design
2222
Respondent tries first product and evaluates it using typical rating scales
Then tries a second product and gives direct comparisons between products
Testing similar products which may have close monadic ratings. Preference ratings can be used as a “tie-breaker”.
Have only a small sample of qualified respondents or incidence is low
Early stagePractical approach
when home use context may be less important
Single product evaluation represents a more realistic & natural environment
Testing a novel or unfamiliar product
When there are carryover effects
When To Use
Respondent tries one product and evaluates it and then tries a second product and evaluates it separately without comparisons
Often CLT
Respondent tries only one product and evaluates
Often in home (IHUT)
What It Is
Protomonadic Sequential MonadicMonadicDesign
Product Testing Designs
2323
Same types of questions as “Monadic” design for first product only
Preference questions on overall performance and attributes after second product trial
Not often recommended
Some clients want to use first position monadic and have preference as a tie breaker
Having same respondent evaluate multiple products may increase statistical precision because there is more control over respondent variability
Each product is evaluated by an independent sample then the results evaluated against each other
Design Characteristics
As per monadic PLUS
Overall preference is sometimes added at end of second evaluation
“Absolute measurements” without any comparison references
HedonicIntensityEfficacyJAR
Type Of Questions
Protomonadic Sequential MonadicMonadicDesign
Product Testing Designs (Cont.)
Product Testing Designs: Practical Considerations
24
Objectives Design Unbranded or Branded?
Innovation Exploration Sequential monadic
Usually Unbranded
Guidance Sequential monadic
Usually Unbranded
Validation Monadic Usually Branded
Renovation Benchmarking MonadicBoth
Unbranded and Branded
Cost Savings / Quality Improvement Protomonadic Branded
25
Design Considerations:Central Location Test
vs. In-Home Use Test
VS
Central Location Test
In-Home Use Test
Venu
e In what environment will product(s) be evaluated?
26
Used for late stage in some F&B & for early stage when the goal is to evaluate in a controlled environment
What’s important ?
CENTRAL LOCATION TEST (CLT) IN-HOME USE TEST (IHUT) An IHUT is appropriate at later /
Validation stages of research when goal is to confirm product acceptance in a realistic setting
Should be considered when client is seeking context around how product is prepared consumed, etc.
What’s important ?
Central Location Test (CLT)
27
A study conducted to evaluate the use and performance of an object in a controlled environment
Respondents are asked to come to a centrally located facility where they are exposed to the object(s) to be tested
Presentation of objects typically follows a sequential monadic design
Respondents will be pre-recruited and invited to a central location
Appropriate for early stage research
When you seek a highly controlled environment
Early stage research such as guidance research, may be more qualitative in nature, with smaller base sizes
Prototypes (limited amount of product)
Sniff Tests
Respondents sniff the fragrance of a variety of products
Products which are not normally used or prepared in-home
Restrictions on shipping
What Why
In-Home Use Test (IHUT)
28
A study conducted to evaluate the use and performance of an object in a setting more consistent with how the object might normally be used by consumers
Appropriate for later stage research
The data obtained from such an evaluation are considered to have good validity given the natural setting in which the object is used
Desire to allow respondents to have a real-life experience with the object
Packaging research
Cost Reduction research
Products which require usage over a more extended period of time
Measuring satiety or other characteristics over time
Understand When, Where, and How consumers use a product on different occasions
What Why
CLT vs. IHUT / Pro’s & Con’s
29
Central Location Test In-Home Use Test
Pro Con Pro Con
Use
Appropriate for prototype testing
where there may be limited supply of
product
May not reflect how consumers would
actually use product
Product used in natural environment;
Consumers control when, how, and how
much they use
No control over variability in how product is used
ExperienceConsumers may have
increased awareness of product differences
Product can be used repeatedly over time; Preparation / serving
according to each individual
Potential liability issues if problems
arise when product is used (e.g., illness)
Control
Can be highly controlled
environment; Can control product preparation and
consumption
Consumers may feel controlled environment is artificial; May not be as tightly controlled as
expected
No strict control over what happens in-home; Products
can be prepared very differently
Evaluations
Can probe on a wide range of product
characteristics; Can ensure consumers understand rating
scales
Product characteristics may impact evaluations
differently than in an IHUT
Can measure satiety and consumption rates;
Appropriate when a consumer needs to
have the products in their hands (packaging)
CLT vs. IHUT: Practical Considerations
30
Go with CLT if ... It’s in the early stage where a strict control is required in
product preparation It’s about screening many prototypes in the early stage in
Foods & Beverages R&D can’t provide enough samples to conduct IHUT It’s about a fragrance sniff test
However, keep in mind that ... CLT captures the short term effect of the product While IHUT captures the long term effect
─ Example: The infamous “New Coke” disaster (April 23, 1985)
Remembering New Coke, The Ultimate Product Introduction Disaster (http://fw.to/GbO886O) - April 24, 2014
31
To Sum Up, Touch Points to Consider When Designing a Product Test Project are ...
Research Objective
Stage of Development
Benchmarks
Action Standard
Unbranded vs. Branded
Design
Venue for Testing
32
What else to consider in designing a product test project?
Sample Definition• What’s the target market?• Whom to interview?
Sample Size• Depends on product tier, based on
financial and strategic importance• Depends on products availability• Depends on planned subgroups
analyses
Begin Your Quest with InnoQuestIpsos in IndonesiaGraha Arda, 3rd FloorJl. H.R. Rasuna Said Kav. B-6KuninganJakarta 12190
© 2014 PT. Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not be disclosed or reproduced without the prior written consent of Ipsos.