test it in service
DESCRIPTION
The 17th Annual International Deming Research Seminar. Test it in Service. a.k.a. Voice of the Customer. What We Plan to Cover (AIM). Why Test in Service? Why Good Sample Design? Accuracy versus Precision Some Statistical issues Some Methodological Issues Some Data Issues - PowerPoint PPT PresentationTRANSCRIPT
Test it in Servicea.k.a. Voice of the Customer
The 17th Annual International Deming Research Seminar
The 17th Annual International Deming Research Seminar 2
What We Plan to Cover (AIM)
23 March 2011
Why Test in Service?
Why Good Sample Design?
Accuracy versus Precision
Some Statistical issues
Some Methodological Issues
Some Data Issues
Some Questionnaire Issues
The 17th Annual International Deming Research Seminar 3
Deming Lecture Japan 1950
23 March 2011
4
4. Test in Service
3. Sell
5. Redesign
(a. Old Way)
(b. Original Shewhart 1939) (c. Deming 1950 Version)
Evolution of Shewhart CycleSpecification
Design
Production
Make
Inspection
Sell
1. Design
2. Make
The 17th Annual International Deming Research Seminar 5
Production Viewed as a System
A
B
C
D
Suppliers ofMaterials andEquipment
Receipt and test of materials
Tests of Processes, machines, methods, costs
Production, assembly, inspectionConsumerConsumer
ConsumerConsumer
ConsumerConsumer
Consumer
ConsumerResearch
DesignandRedesign
23 March 2011
The 17th Annual International Deming Research Seminar
Process
6
Input Conversion Output
Voice of the Process
Voice of the Customer
GAP
23 March 2011
The 17th Annual International Deming Research Seminar
How do we get the Voice of the Customer?
7
Input Conversion Output
Voice of the Customer
Test it in Service
Consumer Research
23 March 2011
The 17th Annual International Deming Research Seminar 8
Test it in Service
Thought
Data
Knowledge23 March 2011
The 17th Annual International Deming Research Seminar 9
Survey Impact on Management
Wrong Design
Wrong Information
Wrong Decision
23 March 2011
The 17th Annual International Deming Research Seminar 10
What is the aim of the survey?
Who determines the aim? The Subject Matter Expert
Why is the aim important? Without an aim there is no system
23 March 2011
The 17th Annual International Deming Research Seminar 11
Who is involved in Surveys
Subject Matter Expert
Statistician
Respondents
23 March 2011
The 17th Annual International Deming Research Seminar 12
The Subject Matter Expert
■ Commissions the survey■ Determines the aim■ Can act on the results■ Has budget authority
23 March 2011
The 17th Annual International Deming Research Seminar 13
Aspects of Sample Design
■ Statistical Sample size Variance Accuracy
■ Methodological Sample selection Computation
23 March 2011
The 17th Annual International Deming Research Seminar 14
What is a sample
■ Complete coverage gives answer■ A subset of the interest group
Draw Sample
23 March 2011
The 17th Annual International Deming Research Seminar 15
The Frame
■ The Frame is a list of sample units■ Frames are not always complete
The Gap
23 March 2011
The 17th Annual International Deming Research Seminar 16
Data vs. Information
The sample yields data
Results Data
Information
AnalysisThe Subject Matter Expert needs Information
23 March 2011
The 17th Annual International Deming Research Seminar 17
Information contains
■ Result
■ Measure of uncertainty
23 March 2011
18
Which is more important
Sample size?
or
Sample method?
Bias
Sampling Error
Total Error
23 March 2011 The 17th Annual International Deming Research Seminar
Sample size?or
Sample method?
The 17th Annual International Deming Research Seminar 19
Accuracy and Precision
Not Accurate Accurate
Not Precise
Precise
Sample Size
Sample Method
23 March 2011
The 17th Annual International Deming Research Seminar 20
Sample Size (n)
Expected result percentage
Desired precisionWhere is the actual result
t the number of standard deviations under the Normal Curve
Then
p
/D p p p p
2
2
(1 )t p pn
D
tArea for Mean ±ts
1 68%
2 95%
3 99.73%23 March 2011
The 17th Annual International Deming Research Seminar 21
Sample Size Example
Q. Was the Conference useful?
Please check one: Yes □ or No □
If one wants 95% reliability that the actual responses fall within 10% of the response of 50% then .0.5, 0.10, 2p D t
2 2
2 2
(1 ) 2 0.5 (1 0.5)100
0.1
t p pn
D
23 March 2011
The 17th Annual International Deming Research Seminar 2223 March 2011
2
2
(1 )t p pn
D
Why = 50% estimate?p
2 2
2 2
2'
t tf p p
D D
2
2
2
2
1' 0.5
2 2
tDf p p
tD
2
2
2"
tf p
D 0.5 is maximum
2 2
2
t p pf p
D
The 17th Annual International Deming Research Seminar 23
23 March 2011
Because it is the maximum sample size
Why = 50% estimate?p
The 17th Annual International Deming Research Seminar 24
Sample Method
23 March 2011
■ Clear Aim■ Random Sampling■ Unbiased Questions
The 17th Annual International Deming Research Seminar 25
Clear Aim
23 March 2011
What does the owner wish to accomplish with the data?
How will the data be used in practice?
Is the data measureableCause
Effect
Is the data useable?
e.g. measure engagement with an Association
e.g. measure how often contacted by Association
The 17th Annual International Deming Research Seminar 26
Random sample
23 March 2011
■ Equal probability of selection Random numbers Random Start nth Sample Replicated design with random start per zone Deming's Replicated Design:
Total sample cut into zoneslike sliced bread
Total sample size
One zone
The 17th Annual International Deming Research Seminar 27
Reduce Bias in Questionnaire
23 March 2011
If questionnaire…
…is for interviewing then train interviewer toAvoid loaded questions
Avoid verbal bias
Avoid body language bias
…is for respondent's completionAvoid loaded questions
Avoid question overload
Open Ended Questions
28
An open-ended question is designed to encourage a full, meaningful answer using the subject's own knowledge and/or feelings.
Opposite of closed-ended question, which encourages a short or single-word answer.
Open-ended questions also tend to be more objective and less leading than closed-ended questions.
Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE
Open-Ended Question Examples
29
Closed-Ended Questions Open-Ended Questions
Do you get on well with your boss?
Tell me about your relationship with your boss.
Who will you vote for this election?
What do you think about the two candidates in this election?
Is that a photograph of your children?
Tell me about the children in this photograph.
Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE
Prevent Response Bias
30
Avoid Ambiguity: Questions not focused enough to obtain the needed information.
Poor question: How often do you buy fast food?
Better question:
The last 10 times you’ve eaten lunch out, how often have you purchased each of the following types of food?
Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE
Prevent Response Bias
31
Avoid questions that are too general.
Poor question: Do you like orange juice?
(Do you mean taste? texture? price?)
Better question - more specificity:
Do you like the taste of fresh-squeezed orange juice?
Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE
Prevent Response Bias
32
Avoid loaded questions: use of language likely to bias response:
Do you actually support Proposition X?
Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE
Prevent Response Bias
33
Avoid leading questions: Questions that lead respondents toward a particular answer.
Isn’t it true that women are more likely to talk on the phone while driving?
Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE
The 17th Annual International Deming Research Seminar 34
What We Covered
23 March 2011
Why Test in Service?
Why Good Sample Design?
Accuracy versus Precision
Some Statistical issues
Some Methodological Issues
Some Data Issues
Some Questionnaire Issues
The 17th Annual International Deming Research Seminar 35
Questions?
23 March 2011