C1, L2, S1

Design Method of Data Collection

Surveys and Polls

Experimentation Observational

Studies

C1, L2, S2

NonsamplingErrors

Sampling/Chance/Random Errors

Selection bias Interviewer effects

Non-response bias Behavioural considerations

Self selection Transfer findings

Question effects Survey-format effects

Sampling

C1, L2, S3

Sources of Nonsampling Errors

Selection bias

Population sampled is not exactly the population of interest.

e.g. KARE 11 poll, telephone interviews

population

sample

C1, L2, S4

Sources of Nonsampling Errors

Non-response bias

People who have been targeted to be surveyed do not respond.

Non-respondents tend to behave differently to respondents with respect to the question being asked.

C1, L2, S5

1936 U.S. Election

• Country struggling to recover from the Great Depression

• 9 million unemployed

• 1929-1933 real income dropped by 1/3

C1, L2, S6

1936 U.S. Election

• Candidates:

– Albert Landon (Republican)

“The spenders must go!”

– Franklin D Roosevelt (Democrat)Deficit financing - “Balance the budget of the people before balancing the budget of the Nation”

This man has never been President.

C1, L2, S7

1936 U.S. Election• Roosevelt’s percentage

–Digest prediction of the election result

–Gallup’s prediction of the Digest prediction

–Gallup’s prediction of the election result

–Actual election result

43%

44%

56%

62%

• Digest sent out 10 million questionnaires to people on club membership lists, telephone directories etc.

– received 2.4 million responses• Gallup Poll used another sample of 50,000• Gallup used a random sample of 3,000 from the Digest lists to predict Digest outcome

C1, L2, S8

Sources of Nonsampling Errors

Self-selection bias

People decide themselves whether to be surveyed or not.

Much behavioral research can only use volunteers.

C1, L2, S9

Sources of Nonsampling Errors

C1, L2, S10

Sources of Nonsampling Errors

This poll is not scientific and reflects the opinions of only those Internet users who have chosen to participate. The results cannot be assumed to represent the opinions of Internet users in general, nor the public as a whole. The QuickVote sponsor is not responsible for poll content, functionality or the opinions expressed therein.

C1, L2, S13

Sources of Nonsampling Errors

Question effects

Subtle variations in wording can have an effect on responses.

e.g. “Should euthanasia be legal?”vs. “Should voluntary euthanasia be legal?”

C1, L2, S14

18 August 1980 New York Times/CBS News Poll

“Do you think there should be an amendment to the constitution prohibiting abortions?”

Yes 29% No 62%

Later the same people were asked:

“Do you think there should be an amendment to the constitution protecting the life of the unborn child?”

Yes 50% No 39%

C1, L2, S15

Ethnicity 1986 1991 1996 Single Ethnicity

94.6 94.3 81.0

European 81.2 78.1 65.8 Maori 9.1 9.6 7.6Two Ethnicities 4.0 4.5 11.2 European & Maori 2.9 2.7 4.7Two European gps 0.0 0.6 4.5

Question Effects in the NZ Census1986: “What is your ethnic origin? (Tick the box or boxes which apply to you.)

1991: “Which ethnic group do you belong to?” (Tick the box or boxes which apply to you.)

1996: “Tick as many circles as you need to show which ethnic group(s) you belong to.”

C1, L2, S16

Sources of Nonsampling Errors

Interviewer effects

Different interviewers asking the same question can obtain different results.

e.g. the sex, race, religion of the interviewer

C1, L2, S17

Interviewer Effects in Racial Questions

In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked:

“Do you personally feel that you trust most white people, some white people or none at all?”

• White interviewer: 35% answered “most”

• Black interviewer: 7% answered “most”

C1, L2, S18

Sources of Nonsampling Errors

Behavioral considerationsPeople tend to answer questions in a way they consider to be socially desirable.

e.g. pregnant women being asked about their drinking habits

C1, L2, S19

Behavioral Considerations in Election

• Official vote counts show that 86.5 million people voted in the 1980 U.S. presidential elections.

• A census bureau survey of 64,000 households some weeks later estimated 93.1 million people voted.

C1, L2, S20

Sources of Nonsampling Errors

Transferring findings

Taking the data from one population and transferring the results to another.

e.g. Minneapolis opinions may not be a good indication of Winonan opinions.

Minneapolis

sample

Winona

C1, L2, S21

Sources of Nonsampling Errors

Survey-format effects

e.g. question order, survey layout, interviewed by phone or in- person or mail.

C1, L2, S22

NonsamplingErrors

Sampling/Chance/Random Errors

Selection bias Interviewer effects

Non-response bias Behavioural considerations

Self selection Transfer findings

Question effects Survey-format effects

Sampling

C1, L2, S23

1999 New Zealand Newspaper Graphic

C1, L2, S24

Nonsampling Errors

What nonsampling errors could be present if the results of the survey were used as representative of the views held by New Zealanders in general?

Discuss in groups for 1 minute.

C1, L2, S25

Nonsampling Errors

• Selection bias

– only those who read this paper

– only those that can afford to pay for the call

• Question effects

– how do you answer the question??

• Self-selection bias

C1, L2, S26

SurveyErrors

Sampling/Chance/ Random Errors

NonsamplingErrors

C1, L2, S27

Sampling / Chance / Random Errors

• errors caused by the act of taking a sample

• have the potential to be bigger in smaller samples than in larger ones

• possible to determine how large they can be

• unavoidable (price of sampling)

C1, L2, S28

Nonsampling Errors

• can be much larger than sampling errors

• are always present

• can be virtually impossible to correct for after the completion of survey

• virtually impossible to determine how badly they will affect the result

• must try to minimize in design of survey (use a pilot survey etc.)

C1, L2, S29

Surveys / Polls

A pilot survey is a small survey that is carried out before the main survey and is often used to identify any problems with the survey design (such as potential sources of non-sampling errors).

C1, L2, S30

Surveys / Polls

A report on a sample survey/poll should include:– target population (population of interest)– sample selection method– the sample size and the margin of error– the date of the survey– the exact question(s) being asked