sensory panels: set-up, management and reducing bias
TRANSCRIPT
Yasigiworld Ltd
www.yasigiworld.com
Paul Hughes
Sensory panels
Set-up, avoiding bias and enhancing the quality of sensory panel data
www.yasigiworld.com
Introduction
• Sensory data is essential to our business
– Final decisions on product release
– Evolving new product concepts
– Competitor analysis
• Output requires human intervention
• Data (good and bad!) is always forthcoming
Reliable data of required quality enhances
competitiveness and facilitates brand management
www.yasigiworld.com
• Challenges
– Convening and maintaining panels
– Use of scales and (in)appropriate data handling
– Prediction of sensory qualities from analytical data
– Integrating sensory information - holistic
Introduction
Want to address scaling issues, but first, to focus
on the core of sensory analysis: the panel
www.yasigiworld.com
Scope
• Panel pool size paradox
• Correcting for assessors
• Towards predicting sensory performance from analysis
– Magnitude estimation
– The Sensory Unit
• Addressing the holistic challenge
www.yasigiworld.com
Panel pool size paradox
• For a panel of a given size, what size of panel pool do you need?
• Clearly depends on panellist availability
• Can model chance of convening a panel from a pool of n panellists: (readily handled by MS Excel!)
• Where P(A) is the probability that panellist i is available
)1.()!(!
!.)](1[.)]([)( )( eq
ini
nAPAPpanelaConveningP
n
ri
ini
www.yasigiworld.com
Panel pool size paradox
0.0
0.2
0.4
0.6
0.8
1.0
15 20 25 30
Size of panel pool
P(c
on
ven
ing
a p
an
el
of
8)
P = 0.5
P = 0.6
P = 0.7
P = 0.8
P = 0.9
P = 0.95
www.yasigiworld.com
• So if the probability of a panellist being available is 0.5, then a panel convenor needs well over 20 trained people to choose from to have a good chance of running a given panel
• This level of availability is not atypical for some staff…
• So whilst training is important, so is reliable attendance
• The performance of
Panel pool size paradox
www.yasigiworld.com
Panel pool size paradox
• So, assuming that a pool of 20 is maintained for panels of 8, there are around 126,000 possible panel compositions
• We asked 20 tasters to assess the bitterness of two commercial lager beers
• Then, we computed the mean of each possible panel combination for panel sizes of 6, 8 and 10….
www.yasigiworld.com
Panel pool size paradox
0
2
4
6
8
10
12
14
16
18
20
20 25 30 35 40 45
Panel mean
Pro
ba
bilit
y o
f a
tta
inin
g p
an
el m
ea
n (
%)
6 from 20
8 from 20
10 from 20A
0
5
10
15
20
25
30
35
40
45
20 25 30 35 40 45
Panel mean
Pro
ba
bilit
y o
f a
tta
inin
g p
an
el m
ea
n (
%)
6 from 20
8 from 20
10 from 20B
www.yasigiworld.com
Panel pool size paradox
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30 35 40 45
Panel mean
Pro
ba
bil
ity
of
ac
hie
vin
g p
an
el
me
an
(%
)
6 from 20
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30 35 40 45
Panel mean
Pro
ba
bil
ity
of
ac
hie
vin
g p
an
el
me
an
(%
)
8 from 20
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30 35 40 45
Panel mean
Pro
ba
bil
ity
of
ac
hie
vin
g p
an
el
me
an
(%
)
10 from 20
Resolution between products improves with panel size
www.yasigiworld.com
Panel pool size paradox
Panel size Data set Mean (standard
deviation)
A > B/%
(Error rate/%)
6 A 30.6 (3.16)
83 (17) B 26.7 (1.34)
8 A 30.6 (2.54)
89 (11) B 26.7 (1.09)
10 A 30.5 (2.08)
93 (7) B 26.7 (0.90)
Here, 10 panellists rather than six more than halves error rate
www.yasigiworld.com
Panel pool size paradox
• The paradox is the balancing of costs and effort of panel maintenance with quality of resulting information
• As tools such as profile analysis are being used to go beyond pass/fail to distinguish between products of similar quality, need better resolution from sensory testing
• This has implications for the way in which we collect and analyse profile data…
www.yasigiworld.com
• It is common in production companies for assessors to evaluate a relatively small subset of products (compared with, say, a research facility)
• Individuals tend to scale consistently within themselves but not between themselves
– Panel means represent few if any individuals
– Adds a lot of scatter to the data
Correcting for assessors
www.yasigiworld.com
Correcting for assessors
0
20
40
60
Estery
Hoppy
Sulphury
Sweet
Bitter Sour
Astringent
Acetaldehyde
Diacetyl
Taster 1
Taster 2
www.yasigiworld.com
Correcting for assessors
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12
Tast
e s
core
Taster code
Max Min Median
Six assessments of the same beer batch over a two week period. Note internal consistency between tasters
www.yasigiworld.com
• Could we filter out individual variances first?
– Yes, mean-centering each individual panelist
• Step 1: Create reference beer set
Correcting for assessors
Panelist-descriptor matrix of median
score values
The “beer samples” are presentations of the same beer batch six times.
Result is a panelist-descriptor matrix
www.yasigiworld.com
• Step 2: Correct test sample using panelist correction
Correcting for assessors
Reference panelist- descriptor matrix
Test sample panelist- descriptor matrix
Panelist-corrected test sample panelist-
descriptor matrix
Panelists
Descri
pto
rs
- =
Final vector contains the panelist-corrected scores for the test sample
www.yasigiworld.com
• Tested out using beer bitterness as an example. Used proprietary algorithm to create panel means for all panels of nine from a pool of 12 assessors
Correcting for assessors
Reference Test
0
5
10
15
20
25
30
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Fre
qu
en
cy
Panel scores
Reference Test
Some suggestion that the test is less bitter
than the reference. Not very convincing though!
Uncorrected panelist data
www.yasigiworld.com
• Corrected data shows that the test is unambiguously less bitter than the reference
Correcting for assessors
0
5
10
15
20
25
30
-2.7
1
-2.5
6
-2.4
1
-2.2
6
-2.1
1
-1.9
6
-1.8
1
-1.6
6
-1.5
1
-1.3
6
-1.2
1
-1.0
6
-0.9
1
-0.7
6
-0.6
1
-0.4
6
-0.3
1
-0.1
6
-0.0
1
0.1
4
Fre
qu
en
cy
Reference-corrected panel scores
Reference sample
Distribution of test sample
www.yasigiworld.com
• Mean-centering correction of assessors, together with creating exhaustive combinations of panel membership from a pool of assessors substantially enhances panel resolution
• Particularly useful in situations where panelists assess relatively few products, but become expert in the assessment of those products
• Critically, no change of the sensory experiment is required, reducing chance of bias and risking the opportunity to track historical data
Correcting for assessors
www.yasigiworld.com
Towards predicting sensory performance from analysis
• Better prediction of sensory quality from analytical data can give more cost-effective product monitoring and NPD
• Opportunities to get:
– The same information for less investment
– More information for the same investment
• Enhanced competitiveness
www.yasigiworld.com
Towards predicting sensory performance from analysis
• Scales are often labelled arbitrarily, eg
• This has been shown to be erroneous. Labels often better satisfy the scale below:
This is due to non-linearity of sensory responses…
www.yasigiworld.com
Towards predicting sensory performance from analysis
0
5
10
15
20
25
0 10 20 30 40 50
[Bitterness] (mg/l)
Nu
mb
er
of
JN
Ds
Flavour threshold
www.yasigiworld.com
Towards predicting sensory performance from analysis
0
10
20
30
40
50
60
0 10 20 30 40 50 60
[Hop acids] / mg/l
JN
Ds
1.05 1.075 1.10 1.15
www.yasigiworld.com
Towards predicting sensory performance from analysis
• Two-step procedure for relating sensory and analytical data
1. Convert analytical data into Flavour Units
2. Apply non-linear correction to Flavour Units to derive the Sensory Unit. Expression of the form:
)2.(][
)( eqthresholdFlavour
AnalyteFUUnitsFlavour
)3.()ln( eqbFUaSU
www.yasigiworld.com
Towards predicting sensory performance from analysis
0
1
2
3
4
5C
arb
on d
ioxid
e
Hop a
cid
s
Eth
anol
Isoam
yl a
ceta
te
DM
S
Eth
yl a
ceta
te
Meth
aneth
iol
Dim
eth
yl t
risulp
hid
e
Eth
yl t
hio
aceta
te
MB
T
Gly
cero
l
Sulp
hite
Meth
yl t
hio
aceta
te
Dia
cety
l
Phosphate
Chlo
ride
Hydro
gen s
ulp
hid
e
Malto
se
Pota
ssiu
m
Aceta
ldehyde
Malto
trio
se
Sulp
hate
Magnesiu
m
Fla
vo
ur
Un
its
(F
U)
0
1
2
3
4
5
0 1 2 3 4 5
Expected FU's
Ob
se
rved
FU
's
MBT(50 vs 20 ppt)
CO24.2 vs 4.5 g/l)
Hop acids(23 vs 21 ppm)
DMS(40 vs 55 ppb)
Isoamyl acetate(2.2 vs 2.0 ppm)
Off-diag
onal
– out o
f “perfe
ct” sp
ecifica
tion
0
1
2
3
4
5
0 1 2 3 4 5
Expected FU's
Ob
se
rved
FU
's
MBT(50 vs 20 ppt)
CO24.2 vs 4.5 g/l)
Hop acids(23 vs 21 ppm)
DMS(40 vs 55 ppb)
Isoamyl acetate(2.2 vs 2.0 ppm)
0
1
2
3
4
5
0 1 2 3 4 5
Expected FU's
Ob
se
rved
FU
's
MBT(50 vs 20 ppt)
CO24.2 vs 4.5 g/l)
Hop acids(23 vs 21 ppm)
DMS(40 vs 55 ppb)
Isoamyl acetate(2.2 vs 2.0 ppm)
Off-diag
onal
– out o
f “perfe
ct” sp
ecifica
tion
Step 1
www.yasigiworld.com
Towards predicting sensory performance from analysis
-70
-60
-50
-40
-30
-20
-10
0
10
20
30
-70 -60 -50 -40 -30 -20 -10 0 10 20 30
Expected SU's
Observed SU'sAbove threshold, when
spec is below threshold
Observed and expected
are below threshold
Below threshold, when
spec is above threshold
MBT
DMS
Hop acids
Ethyl thioacetate
-70
-60
-50
-40
-30
-20
-10
0
10
20
30
-70 -60 -50 -40 -30 -20 -10 0 10 20 30
Expected SU's
Observed SU'sAbove threshold, when
spec is below threshold
Observed and expected
are below threshold
Below threshold, when
spec is above threshold
MBT
DMS
Hop acids
Ethyl thioacetate
www.yasigiworld.com
Towards predicting sensory performance from analysis
• Such an approach requires validation in a commercial environment
• Certain missing data is essential, such as the magnitudes of the JND steps, but this can be derived by experiment
• Moves us further on, but still assumes a 1-to-1 mapping of analytes to flavours…
www.yasigiworld.com
Integrating sensory information
• “Holistic” is a common term
• Implies interconnectedness
• To a first approximation, can ignore minor variables
• For more accurate information, need to bring in more and more parameters
• Today, merely want to set the scene
www.yasigiworld.com
Holistic beer quality
Uncontrolled image Controlled image Intrinsic liquid properties
Flavour Visual Internet
Mouthfeel
CO2/N2
Saliva
pH
Viscosity
Polyphenols
Mass media
TV
Newspapers
Magazines
Dispense, packaging
Glasses
Fonts
Home systems
Theatre of pour
Bottle/can
Health/physiology
Taste
Sweet
Salt
Sour
Bitter
Umami
Aroma
Hop
Malt
Fermentation
Maturation
Age-related
Off-aromas
Individual postings
Formal media
Marketing
Product
Promotion
Price
Place
Colour Clarity Foam Integrity Well-being
Chemical Biochemical
Physical
Microbiological
GMO
Radioactivity
Nutrition
Morning-after
Psychological
Allergens
Integrating sensory information
Integration required at various levels…
Intrinsic liquid properties
Flavour Visual
Mouthfeel
CO2/N2
Saliva
pH
Viscosity
Polyphenols
Health/physiology
Taste
Sweet
Salt
Sour
Bitter
Umami
Aroma
Hop
Malt
Fermentation
Maturation
Age-related
Off-aromas
Colour Clarity Foam Integrity Well-being
Chemical Biochemical
Physical
Microbiological
GMO
Radioactivity
Nutrition
Morning-after
Psychological
Allergens
On-the-spot experiences
Delayed responses
www.yasigiworld.com
Integrating sensory information
• Challenge is our classically reductionist view of both sensory and chemical analysis
• How to integrate? First need to understand the activities of specific flavours and how they interact with the matrix…
www.yasigiworld.com
Integrating sensory information
Sophistication
Accuracy (correlation to sensory score)
Free diacetyl concentration
Weighted sum of VDK levels and
intermediates. Matrix compensation
Sum of free diacetyl and pentanedione
concentrations
Weighted sum of free diacetyl and
pentanedione concentrations
www.yasigiworld.com
Summary
• We ask more of our sensory analysis today – Finer resolution between products – NPD of food and drinks that push traditional product
envelope
• My argument is that we need to ensure that we get the very best from our sensory panels, by – Taking heed of already well-established scientific
observations and statistical doctrine – Applying some simple post-processing data analysis
tricks to improve panel resolution – Moving towards more holistic measures of sensory
attributes
• A challenge, but a competitive opportunity to those that do it well!
www.yasigiworld.com
• Yasigiworld Ltd was set up with a few aims in mind, not least to provide on-line educational resources and cost-effective texts
• If you have any comments of queries contact me, Paul Hughes, either at
– [email protected], or
– Connect up via LinkedIn
• Coming soon to ourYoutube channel, our 100seconds on… series on alcohol-related subjects
About us