introduction to survey analysis -...

Introduction to Survey Analysis!

Professor Ron Fricker!Naval Postgraduate School!

Monterey, California!

3/26/13 1

Reading:!None!

Goals for this Lecture!

•  Introduction to analysis for surveys!–  Assuming simple random sampling (SRS)!–  With and without finite population correction (fpc)!

•  Basic methods!–  Confidence intervals!

•  Binary questions!•  Simultaneous CIs for Likert scale responses!

–  Hypothesis tests!•  Talk a bit about how to display survey data

and results in a briefing!

3/26/13 2

Important Question: What Methods Appropriate for Survey Data Analysis?!

•  Sub-question: When can we use sample mean or proportion to estimate population mean or proportion?!–  Answer: Depends on the response scale!

•  For this lecture, we will start by assuming SRS and sample size less than 5 percent of the population!–  So standard methods apply!–  Then we’ll add in corrections for fpc!

3/26/13 3

Remember the Central Limit Theorem!

•  Analysts often a bit confused by the (typically) discrete nature of survey data!–  Most methods in “Stat 101” classes based on

Normal distribution, which is not discrete!•  But remember the Central Limit Theorem: !Sampling distributions of sums and averages of iid data are approximately normally distributed!

•  So, under simple random sampling (SRS), with large enough samples, Likert scale and other types of discrete survey data may not be a problem!

3/26/13 4

Naïve Analyses!

•  Naïve analyses just present sample statistics for the means and/or proportions!–  Perhaps some intuitive sense that the sample

statistics are a measure of the population!–  But often don’t account for sample design!

•  Further, “point estimates” provide no information about sample uncertainty!–  If you did another survey, how much might its

results differ from the current results?!–  Leaves it up to the reader to guess at the precision

– not a good idea!

3/26/13 5

Illustrative Example!

•  Consider a binary yes/no question in survey with a population of N=100,000 people:!–  For a sample of n1= 4 people, if 3 say “yes” our

point estimate is!–  For a sample of n2= 500 people, if 375 say “yes”

our point estimate is !

•  But clearly uncertainty of is greater than :!

vs. !

3/26/13 6

1p̂ 2p̂

1ˆ 3 / 4 0.75p = =

2ˆ 375 / 500 0.75p = =

( ) ( )1

0.75 1 0.75ˆ. . 0.217

4s e p

−= = ( ) ( )

2

0.75 1 0.75ˆ. . 0.019

500s e p

−= =

Applying Continuous Methods to Binary Survey Questions!

•  In surveys, often have binary questions, where desire to infer proportion of population in one category or the other!

•  Code binary question responses as 1/0 variable and for large n appeal to the CLT!–  Confidence interval for the mean is a CI on the

proportion of “1”s!–  t-test for the mean is a hypothesis test on the

proportion of “1”s!

3/26/13 7

So, for Binary Questions!

•  The (1- )100% confidence interval is!

where!

•  Similarly, one can do a hypothesis test, just treat as the mean and the standard error is!

3/26/13 8

( ) ( )( )/2 /2ˆ ˆ ˆ ˆ ˆ ˆ1 / , 1 /p z p p n p z p p nα α− − + −

# of successes/agrees/yesesˆ# of respondents

p =

p̂

( )ˆ ˆ1 /p p n−

α

When the FPC Applies!

•  The (1- )100% confidence interval is!

! where!

•  Similarly, one can do a hypothesis test, just treat as the mean and the standard error is!

3/26/13 9

( ) ( )/2 /2

ˆ ˆ ˆ ˆ1 1ˆ ˆ1 , 1

p p p pn np z p zN n N nα α

⎛ ⎞− −⎛ ⎞ ⎛ ⎞⎛ ⎞ ⎛ ⎞⎜ ⎟− − + −⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟⎝ ⎠ ⎝ ⎠⎝ ⎠ ⎝ ⎠⎝ ⎠

# of successes/agrees/yesesˆ# of respondents

p =

p̂( )ˆ ˆ1

1p pn

N n−⎛ ⎞⎛ ⎞− ⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠

α

Example: CI for Fraction of DL Students Who Have Received a Library Briefing!

•  Point estimate: or 62.4% •  Standard error (with fpc):!

•  (Approximate) 95% CI:!

•  So, we’re 95% confident between 60.2% and 64.6% of DL students briefed by library !

•  Major assumption: Respondents are random sample of all DL students !

3/26/13 10

ˆ 408 654 0.624p = =

* Data from 2011 survey of DL students for NPS Library

( ) ( )0.624 1 0.624654ˆ. . 1 0.0111031 654

s e p−⎛ ⎞⎛ ⎞= − =⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠

( ) ( )0.624 1.96 0.011,0.624 1.96 0.011 0.602,0.646− × + × =

Solution Using the R survey Package!

3/26/13 11

Example Continued: Hypothesis Testing!

•  Test the hypothesis that the proportion of DL students in the population who have received a briefing is greater than 60%!

•  We want to know !

•  In R, we have!

•  p-value < , so we can conclude that the true proportion is greater than 60% !

3/26/13 12

p-value = Pr X > 0.624 |µ = 0.60,s = 0.011( )

α = 0.05

CIs For Likert Scale Questions with More than Two Levels!

•  Need to calculate simultaneous confidence intervals for k Likert scale levels!

•  Use!

where!–  ni = number of respondents choosing the ith level!

–  !

–  B is the upper ( )100th percentile of the distribution with 1 degree of freedom!

3/26/13 13

ni + B / 2n+ B

±B2 / 4+ Bni 1− ni / n( )

n+ B( )2

n = ni

i=1

k

∑

Goodman, L.A. (1965). On Simultaneous Confidence Intervals for Multinomial Proportions, Technometrics, 7, 247-254.

χ2

α k

Example from Library Survey !

•  Survey results:!

•  For a 95% CI, , so with k = 6:!

•  Then for the percent saying strongly agree!

3/26/13 14

Strongly Agree!

Agree! Neutral! Disagree! Strongly Disagree!

N/A!

“Library Research is a Crucial Part of My NPS DL Studies”!

95!(15.0%)!

218!(34.4%)!

196!(31.0%)!

86!(13.6%)!

27!(4.3%)!

11!(1.7%)!

( )( )

2

2

6.96 / 4 6.96 95 1 95 / 63395 6.96 / 2 0.154 0.037633 6.96 633 6.96

+ × −+ ± = ±+ +

* Data from 2011 survey of DL students for NPS Library

α = 0.05

Graphically!

3/26/13 15

0!

5!

10!

15!

20!

25!

30!

35!

40!

45!

Strongly Agree! Agree! Neutral! Disagree! Strongly Disagree!

Perc

ent!

Applying Continuous Methods to Likert Scale Survey Data!

•  Likert scale data is inherently categorical!•  If willing to make assumption that “distance”

between categories is equal, then can code with integers and appeal to CLT!

!Strongly agree!!Agree!!Neutral!!Disagree!!Strongly disagree!

3/26/13 16

1 2 3 4 5

An Example of Inference for the Mean of a Likert Scale!

•  Consider the new student survey conducted by a previous class!

•  Question 1 asked, “In general how do you rate the satisfaction or dissatisfaction with the IN-PROCESSING procedures at NPS?”!–  Response scale was a 5-point Likert from Very

Satisfied (5) to Very Dissatisfied (1)!•  We might be interested in the mean response

for the population !–  Is it near neutral, in the positive range, or the

negative?!

3/26/13 17 17

* Data from 2008 survey of NPS new students

Example, continued!

•  A confidence interval is appropriate to address this question!–  Assume the respondents were a SRS from a

population of size N=183!•  Data:!

3/26/13 18 18


Example, continued!!

•  To calculate:!

•  The 95% confidence interval: !!

3/26/13 19 19

2

/2 1 n sx zN nα

⎛ ⎞⎛ ⎞± − ⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠


Example, continued!

3/26/13 20

!

•  Using the R survey package:!

•  With 95% confidence, the mean response for the population is in the range (3.79, 3.86)!–  Average person in the population is on the

satisfied side of the Likert scale!

Example, continued!

•  Perhaps we want to test the hypothesis that the population mean response is less than 4 !–  I.e., the average person is not quite

“satisfied” (i.e., a 4) on the Likert scale!–  First, assume a SRS from a large population !

•  Then!

!3/26/13 21 21


Now, Accounting for the FPC!

•  Note that we’ve already proven the alternative hypothesis that the mean is less than 4!–  Using the fpc means the standard errors will be

smaller, so we’ll just reject more strongly!•  But, if we want the exact p-value anyway:!

•  It’s really, really small: We can be sure that, while the population feels positively about NPS check-in, the average person is not quite “satisfied” as defined on the Likert scale !

3/26/13 22


To Summarize!

•  To analyze mean responses when (1) SRS used, (2) fpc does not apply, and (3) coding scale appropriate, proceed with “Stat 101” analytical tools !–  With fpc, just need to adjust the standard errors!–  With complex sampling, more complicated

analytical methods or software required!•  Can also use categorical data methods, such

as chi-squared tests!–  More on this in upcoming lecture!

3/26/13 23 23

A Bit About Briefing Survey Results!

•  When briefing survey results:!–  DON’T just present the data question-by-question

in the order asked in the survey – boring!!–  DO tell a story!

•  Focus on the questions that!–  Answer the survey objective!– Give results interesting to the “client”!

»  Sometimes it’s the outliers or tails…!•  Order the presentation of results so that it’s

logical and interesting to the listener!

3/26/13 24

A Good Briefing Outline!

•  Survey objective(s)!•  Outline of the survey instrument!

–  Perhaps a brief discussion of design development !•  Fielding methods and details!

–  Response rate(s)!•  Comparison of sample to population !!

–  Demonstrate how representative (or not) sample is!•  Results (see next slide)!•  Conclusions & discussion!

3/26/13 25

Displaying Survey Results!

•  For results slides, use a small number of standardized formats!–  Put the “take away” summary in slide header!–  Give actual survey question verbiage and number

who answered the question!•  When giving percentages, show the n as well,

and vice versa!•  As appropriate, display uncertainty due to

sampling (i.e., the margin of error)!–  Use actual quotes (e.g., from open-ended

questions) to reinforce graphs and plots!

3/26/13 26

Almost 50% of DL Students (313 of 633) Agree Library Critical to Their Studies!

27

n=95! n=218! n=196! n=86! n=27!0!

5!

10!

15!

20!

25!

30!

35!

40!

45!


Perc

ent!

“How much do you agree or disagree with the following statement: ��‘Library research is a critical part of my NPS Distance Learning Studies.’ ”

•  “I believe that the NPS library is an essential tool that should always be available to DL students.”!

•  “The NPS Library is a valuable resource. Not every class requires need for use, however, some do.”!

•  “I have used the NPS Library for a good deal of my day job work as well. It has provided me an easy avenue to periodicals and journals that are often difficult to get access to through local research streams.”!

3/26/13

n=95! n=218! n=196! n=86! n=27!0!

5!

10!

15!

20!

25!

30!

35!

40!

45!


Perc

ent!

Program! Percent!

SE (Nuclear) – Electrical (n=3)! 100!

SE (Nuclear) – Mechanical (n=1)! 100!

Systems Analysis (n=27)! 42!

Cost Estimating and Analysis (n=12)! 27!

Electronic Sys. Engineering (n=47)! 23!

MSES (n=47)! 23!

EMBA (mil & civ) (n=148)! 21!

Systems Engineering (n=230)! 16!

Program Management (n=68)! 10!

Systems Eng. Management (n=31)! 3!

Those Who Disagreed Were Largely Engineering and OR Disciplines!

28


3/26/13

n=95! n=218! n=196! n=86! n=27!0!

5!

10!

15!

20!

25!

30!

35!

40!

45!


Perc

ent!

Program! Percent!

SE (Nuclear) – Electrical (n=3)! 100!

SE (Nuclear) – Mechanical (n=1)! 100!



Electronic Sys. Engineering (n=47)! 23!

MSES (n=47)! 23!

EMBA (mil & civ) (n=148)! 21!




Some Majors Did Not Disagree at All, But Small Numbers Not Definitive!

29


Program! Percent!

Contract Management (n=3)! 0!

HSI (N=9)! 0!

Software Engineering (n=4)! 0!

Space Systems Operations (n=6)! 0!

3/26/13

n=95! n=218! n=196! n=86! n=27!0!

5!

10!

15!

20!

25!

30!

35!

40!

45!


Perc

ent!

Neutrals Are Largely Outside (Hard) Engineering Disciplines!

30





Program! Percent!

MSES (n=47)! 51!


HSI (n=9)! 37!

EMBA (mil & civ) (n=148)! 35!



Electronic Systems Engineering (n=47)! 21!




3/26/13

n=95! n=218! n=196! n=86! n=27!0!

5!

10!

15!

20!

25!

30!

35!

40!

45!


Perc

ent!

Most Disciplines Had Substantial Percentage of Students Who Agreed!

31





Program! Percent!

Space Systems (n=6)! 100!


Space Systems Operations (n=6)! 83!

HSI (n=9)! 63!



Electronic Systems Engineering (n=47)! 53!



EMBA (mil & civ) (n=148)! 43!


MSES (n=47)! 22!

3/26/13

On Displaying Margins of Error!

•  “Do I have to display margins of error on every plot?”!–  No, sometimes it’s overkill and/or distracting!–  But they should be communicated somehow !

•  If not included on every plot and table, give the reader/audience some general guidelines: !!“For analyses of the entire DL student population, the margins of error in this survey are approximately!–  two percent for questions with a binary scale (e.g., yes/no), !–  five percent for questions with a Likert scale (e.g., strongly

agree, agree, neutral, disagree, strongly disagree).!!When analyzing smaller groups the margins of error will be larger, perhaps substantially.”!

3/26/13 32

Barcharts and Histograms Not Optimal for Comparing Between Groups or Subsets!

•  Neither plot particularly good at allowing visual comparison between groups!

3/26/13 33

Married and collocated with family Single Married and a geographic bachelor Single with dependents Other

Very SatisfiedSatisfiedNeutralUnsatisfiedVery Unsatisfied

Presidio Health Clinic Rating

Count

0100

200

300

400

500

600

700


Very UnsatisfiedUnsatisfiedNeutralSatisfiedVery Satisfied


Count

050

100

150

200

250

300

Barcharts and Histograms Not Optimal for Comparing Between Groups or Subsets!

•  Converting to percentages does not really help:!

3/26/13 34



Fraction

0.0

0.2

0.4

0.6

0.8

1.0



Fraction

0.0

0.1

0.2

0.3

0.4

0.5

Likert-scale Data:Diverging Stacked Bar Charts!

3/26/13 35

Source: Robbins, N.B., and R.M. Heiberger, Plotting Likert and Other Rating Scales, Proceedings of the 2011 Joint Statistical Meetings, 1058-1066.

Compare to Traditional Bar Charts!

•  Much harder to distinguish differences:!–  Divided bar chart:!

–  Side-by-side bar chart!

3/26/13 36

Source: Robbins, N.B., and R.M. Heiberger, Plotting Likert and Other Rating Scales, Proceedings of the 2011 Joint Statistical Meetings, 1058-1066.

Diverging Stacked Bar Charts!

3/26/13 37

Diverging Stacked Bar Charts!

3/26/13 38

Other Thoughts on Survey Briefings!

•  Goal is to communicate to decision maker what the data say about the survey objective!–  Don’t make it a data dump!

•  Focus on effective graphical communication!–  Use graphics that effectively communicate the quantitative

results!•  See Cleveland (1994, 1993) and Tufte (1990, 2001)!

–  Save the mathematics, modeling, and technical details for the back-up slides / report appendix!

•  But do communicate the necessary details to convince the audience that the survey was done effectively and rigorously!–  Response rate (presumably high), margin(s) of error, etc.!

3/26/13 39

References on Good Graphics!

•  Cleveland, W.S. (1993). Visualizing Data, Hobart Press.!

•  Cleveland, W.S. (1994). The Elements of Graphing Data, Hobart Press.!

•  Tufte, E.R. (2001). The Visual Display of Quantitative Information, 2nd ed., Graphics Press.!

•  Tufte, E.R. (1990). Envisioning Information, Graphics Press.!

3/26/13 40

What We Have Covered!

•  Introduction to analysis for surveys!–  Assuming simple random sampling (SRS)!–  With and without finite population correction (fpc)!

•  Basic methods!–  Confidence intervals!

•  Binary questions!•  Simultaneous CIs for Likert scale responses!

–  Hypothesis tests!•  Discussed a bit about how to display survey

data and results in a briefing!

3/26/13 41

introduction to survey analysis -...

Documents