winery laboratory quality control - ucanr
TRANSCRIPT
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 1
Winery Laboratory Quality Control
Pat HoweFermentation Readiness:
Harvest Decisions from the Vineyard to the Winery
UC Davis, July 29, 2019
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 2
Winery Laboratory Quality Control
…or…
Pat HoweFermentation Readiness:
Harvest Decisions from the Vineyard to the Winery
UC Davis, July 29, 2019
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 3
How to Successfully Fudge Analytical Results
Pat HoweFermentation Readiness:
Harvest Decisions from the Vineyard to the Winery
UC Davis, July 29, 2019
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 4
Fudge: Definition
• [noun] soft creamy candy
• [verb] fake or falsify; “fudge the figures"; "cook the books"; "falsify the data"Synonyms: manipulate, fake, falsify, cook, wangle, misrepresent
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 5
Disclaimer
• This talk begins with a humorous and ironical presentation of how to fudge analytical results
• I do not advocate fudging of lab results.
• The purpose of this ironical and humorous approach is to two‐fold: • To show that a typical laboratory without a quality control program will to use the same techniques that a fudger would use
• To show that a basic quality control program will provide tools to defend or substantiate both expected and unexpected analytical results
• If you are Ironically Challenged, see footnotes on each slide as a guide to know when I am being humorous.
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 6
Reasons to fudge‐ what motivate a fudger?
• Time constraints
• Inadequate physical resources
• Inadequate training
• Laziness
• Boredom
• Long scientific tradition, the “dry‐lab”
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 7
Keys to successful fudging
• Understand the winemaking process and why the analysis is being requested
• Understand the analytical methods and equipment used
• Understand “expected” results
• Cover your tracks
• Anticipate questions
• Divert suspicion
Successful fudging is not easy
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 8
How is the analysis being used?
• Actionable analyses
• Establishing baselines
• Required analyses
• Curiosity/research
• Longstanding tradition
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 9
If your results are questioned…
• Making up results is only half the art of fudging
• Successfully defending your fudged result is critical to your success
• Six standard strategies
How would you react if you were being called out for fudging?
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 10
Common strategies if results are questioned
• Bad Sample
• Blame the analyst
• Error in reporting
• Lousy equipment
• Lousy procedure
• Blame the previous results
Let’s go through these techniques
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 11
1: Bad Sample
• Wrong sample • Wrong wine, wrong tank, wrong barrels, etc
• Improperly taken• Tank not mixed, bad berry sampling, bottom sample instead of top, before pump over not after
• Improperly stored• Refrigerated/frozen or left in hot car
• Improperly processed• Filtered or not, decolorized or not, centrifuged or not
Humorous and Ironical
This is also a true issue even with valid analytical methods
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 12
2: Blame the analyst
• Works best in multiple shift environment• Incompetent graveyard shift
• Works great in high staff turnover environments (and can contribute to high staff turnover, too!)
• The new guy is incompetent.
• Backfires if you are the analyst
• Backfires if the analysts gets wind of this and waits for you after work
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 13
3: Error in reporting
• The result was good, but:• Math error
• Transcription error
• Digits reversed
• Decimal place missed
Humorous and Ironical
This is also a true issue even with valid analytical methods; human error can’t be underestimated
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 14
4: Lousy equipment
• The equipment is too high tech and you can’t figure it out
• Might get you a trip to the training center
• The equipment is too old and you need to get new stuff
• Works good when your budget is always being cut
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 15
5: Lousy procedure
• Has never worked and is always suspect
• Fraught with all sorts of problems• No standard reagents, the equipment is weird
• You’ve tried all sorts of improvements: • Change sample size, time of processing, temperature, new reagents, different equipment, etc
• “Everyone” has problems with it• You are constantly discussing this with other wineries and trying to solve the problem
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 16
6: It was the previous result
• The current result is correct!
• The earlier value was wrong!
• Works for baseline and addition analyses
• Works great if staff, procedures, or equipment has recently changed
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 17
Amateur Fudging
• How many of those fudging techniques sound strangely familiar?
• Are you fudging even though you are performing analysis? You may be “fudging” without even knowing that you are doing it!
• Can you prove you are not just making up a number? How can you determine if you are getting valid results?
• What is a valid analytical result?
• What is the difference between:• Fudging the results vs Running a procedure without validating and checking?
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 18
What is a valid analytical result?
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 19
What Winemakers Mean by “Good” Results
• Knowledge of situation (expectations)
• Acceptable ranges (“reasonable values”)
• Historical perspective (previous values)
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 20
What lab technicians mean by “Good” results
Winemaker doesn’t complain
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 21
How the Lab can be “Good”
• Anticipate expectations
• Only report “reasonable values”
• Check historical values
Common Winery Laboratory approach
Also the techniques used by a good fudger!
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 22
The dangers of “good” results
• There are four possible situations when evaluating analytical outcomes
• The true value of the sample could be either• Expected, within specification, routine, usual or normal
• Unexpected, out of spec, bizarre, unusual or abnormal
• The lab results could be either• The true value or
• Not the true value
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 23
Example: Bottling and Winery A
Lab Result= True Value
Lab Result= Not True Value
30 ppmRoutine True
Value
30 ppm (expected)Standard routine
5 ppm added = 35
20 ppm (unexpected) Result is questioned, and
rechecked 5 ppm added = 35
20 ppmUnusual True
Value
20 ppm (unexpected)Result is questioned
but sustained. 15 ppm added = 35
30 ppm (expected) NOT rechecked
5 ppm added=25
Deadly Serious
Bottling Tank SO2 checksUsually first result is 30 ppm and than a 5 ppm addition
for a final approval at 35 ppm
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 24
Example: Barrel Checks at Winery B
Lab Result= True Value
Lab Result= Not True Value
0.8 g/LExpected True
Value
0.8 g/L (expected)Standard routine
Barrels combined
1.4 g/L (unexpected) Result is questioned,
and rechecked Barrels combined
1.4 g/LUnusual True
Value
1.4 g/L (unexpected) Result is questioned
but sustained. Barrels
quarantined
0.8 g/L (expected) NOT rechecked
Barrels combined and infection spread
Deadly Serious
Checking barrel lots for VA prior to rack to tank and return.Last check of the lots showed range of 0.7 to 0.9 g/L VA
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 25
The Lab’s View
Lab Results = True Value
Lab Result= Not True Value
Situation ARoutine or
expected value
NO COMPLAINTS
COMPLAINTS
Situation BUnusual or problematic
value
COMPLAINTSPrevented problems
NO COMPLAINTSProblems missed
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 26
Which is the worst situation?
Lab Results = True Value
Lab Result= Not True Value
Situation ARoutine or
expected value
NO COMPLAINTSThe “perfect” lab
COMPLAINTSBad reputation for
laboratory, questioning results
Situation BUnusual or
problematic value
COMPLAINTSPrevented problemsA strong qc program needed to report and support these results-but wine is protected
NO COMPLAINTSProblems missed
If the problem is not attributed to the lab,
then NO negative feedback The lab is
still “perfect”-but wine is
compromised
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 27
Types of Errors‐the Lab’s View
• Wineries may be missing a lot of problems because their lab might be more comfortable reporting expected values
• If the only “good” result is one that is expected, why waste time running analyses?
In other words…
THE LAB MIGHT AS WELL FUDGE!
Humorous and Ironical
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 28
Worst Case is a “good” (expected) number…
• The worst case is when the lab gets the “expected” value when it is not true
• When the lab reports an expected value that is incorrect no extra lab work is required, and wine quality is sacrificed
• But: Many wineries train employees to only question values that are “unexpected”
All values should be questioned
All values should be substantiated
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 29
Traditional Winery Laboratory Teachings
Actual Quotes from some Wine Analysis Texts may inadvertently promote fudging philosophies:
“Finally, we frequently fail to ask ourselves, “is this value reasonable?”
“Looking at one’s analytical results from a perspective of past values and common‐sense expectations of possible ranges or magnitudes of values can save some embarrassment.”
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 30
Let’s review: How did we get here?
• An analysis technique which always gets a “good” result will cause the least amount of trouble
• Any procedure which consistently gives “good” results will rarely be questioned
• A procedure can be horribly inappropriate for the situation, but if the results are always “good” it will be used
• The results may be “good” but they may also be inaccurate and imprecise. That is, a “good” result may be “WRONG”!
How do we fix this problem?
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 31
Anti‐fudge: A basic QC program
• All procedures are written, followed, and reviewed regularly and are fit for the task.
• Written procedures include performance criteria such as precision, accuracy and upper and lower limits as found in the lab
• All employees are trained and signed off on each procedure they perform. Competence standards are part of training.
• Laboratory Control Samples are run routinely, and there is an action plan for out‐of‐control results.
• The lab participates in performance testing, either in‐house, external, or both
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 32
Anti‐fudge: A good procedure
• Is accurate and precise based on your needs.
• The procedure (even a standard method from a book) has been validated in your lab with your equipment and personnel
• Accuracy and precision are proven, tracked, and recorded
• Upper and lower levels are known
• Has an associated written procedure that is followed
• Is reviewed for performance and read routinely by employees to update and improve
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 33
Accuracy and Precision
• Is the procedure Accurate and Precise?
• How can we assure ourselves that the result is accurate and precise?
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 34
Is this Procedure Accurate?
• Accuracy is the degree of agreement of individual measurements with some true or reference value
• It is an indication of how close to the true value the result is
• Determined by running proper standards• Not determined by comparing to other labs, despite frequent occurrence…as many wineries are unable to prepare proper standards
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 35
Is this Procedure Precise?
• Precision is the degree of mutual agreement among individual measurements
• It is an indication of how close multiple results are to one another
• Terms such as replicable, repeatable, and reproducible are used in reference to precision
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 36
Sources of Variation in Types of Precision
Source of Variability
Replicable Repeatable Reproducible
Sample Same Same Same
Sub sample Same or different
Same or different
Most likely different
Analyst SameAt least one of these must be
different
Different
Apparatus Same Different
Day Same Same or different
Laboratory Same Same Different
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 37
Common Ways to Test these Variations
Replicable Repeatable ReproducibleSame person runs a duplicate sample in the same run.
A different person runs the same
sample
Another lab runs the same sample
Same person run the same sample later in the day
A different piece of equipment is
used
Different people, equipment, sub sample,
day, or lab
Minimizes the variation,
should have the “best” precision
Maximizes the variation within
a lab, more realistic
Would expect the lowest precision: this
is also sometimes called “robustness”
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 38
Laboratory Control Samples & Control Charts
• A laboratory control sample (LCS) is simply a sample which is run frequently.
• Typically at start up, at shut down, and spaced between every x number of samples.
• It must be treated exactly as a normal sample
• It may contain a known amount of substance (a reference control) or not.
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 39
Getting started with an LCS & Control Charts
• Locate wines which are analytically identical to use for your LCS
• Treat the LCS exactly as any normal wine sample
• Perform analysis and record the results • Generate about 12 or 15 points for each analyte
• Calculate mean and standard deviation for each analyte
• Plot and chart the points and note fluctuation around the mean
• Integrate the analysis of controls into your daily runs
• Rules• > 3 sigma is out of control; >2 sigma is “warning”, more complicated rules for rigorous compliance
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 40
Controls and Control Charts
• Good analysis will be randomly distributed • 68 % of results are within 1 sigma of the mean
• 95.5% are within 2 sigma
• 99.7 % are within 3 sigma
151050
14.5
14.0
13.5
13.0
Sample Number
Sam
ple
Mea
n
X-bar Chart for Alcohol
X=13.59
2.0SL=14.05
3.0SL=14.29
-2.0SL=13.13
-3.0SL=12.90
Deadly Serious
Gau
ssia
n cu
rve
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 41
Controls and Control Charts
• Good analysis will be randomly distributed • 68 % of results are within 1 sigma of the mean
• 95.5% are within 2 sigma
• 99.7 % are within 3 sigma
151050
14.5
14.0
13.5
13.0
Sample Number
Sam
ple
Mea
n
X-bar Chart for Alcohol
X=13.59
2.0SL=14.05
3.0SL=14.29
-2.0SL=13.13
-3.0SL=12.90
Deadly Serious
Gau
ssia
n cu
rve
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 42
Simple Control Charting
• Set up in Excel• Need good excel skill set
• Contact me if you want a copy of my old sheet
• Freeware out on the web
• Record on paper and physically chart‐• Quick and easy, but someone still must crunch data
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 43
Control Charting Excel
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 44
MedLabQC (freeware)
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 45
Are the results valid? Or is the lab unknowingly Fudging?
• Use a standard method and VALIDATE the method as it is used in YOUR laboratory
• Determine the optimum accuracy and precision
• Determine linearity, minimum and maximum values
• Run blanks, standards, and controls routinely• Determine the working precision (and accuracy if known)
• Empower the technicians with the means to determine when they are doing a good job
• Control charts should show variations in results appropriate for the test‐ an occasional “out of control” result and a corrective action is the sign of a healthy laboratory environment.
Deadly Serious
How to Successfully Fudge Analytical Results
copyright 2008 Patricia Howe www.patriciahowewines.com 46
THANK YOU!
Questions?