can$official$records$correct errors$in$turnoutself6reports?$
Post on 25-Apr-2022
14 Views
Preview:
TRANSCRIPT
Can Official Records Correct Errors in Turnout Self-‐reports?
Ma:hew K. Berent, Stanford University
Jon A. Krosnick, Stanford University
Arthur Lupia, University of Michigan
AssumpJons
• IF: EffecJve sampling.
• PLUS: EffecJve survey design.
• THEN: Accurate data.
• TO: Understand the populaJon.
Theory vs. PracJce
Apparent Self-‐report Errors
EsJmates from sample data do not match known populaJon values.
Reasons for Discrepancies
QuesJon: Did you vote? • Non-‐response.
• The quesJon is not answered. • Survey effects.
• ParJcipaJon affects behavior. • MisreporJng.
• Answers are factually incorrect.
Types of MisreporJng
QuesJon: Did you vote? • MisinterpretaJon.
• Interpreted: Is poliJcs important to you? • Misremembering.
• Recalled: I wanted to vote so I must have. • IntenJonal lying.
• Reported: (I didn’t, but saying that would not look good so I’ll say) “Yes”.
When Voters Lie
“It’s a given that people fib in surveys.”
The Wall Street Journal, August 2, 2008
Apparent problem: Respondents lie.
Proposed soluJon: Use official records.
What to Do
Using Official Records
• Obtain official records of turnout histories.
• Match each respondent to his or her official record.
• Determine the “correct” turnout for each respondent.
Two Ways Thought to IdenJfy Lying
A respondent reported one thing
BUT
• An official record shows another
OR
• No official record is found
Fail-to-match
A Case Study 2008-‐2009 ANES Panel Study
Total Sample
• N=4,206 respondents recruited via RDD sampling.
• US ciJzen • VoJng age • Landline households • Completed monthly online surveys
Turnout
Which one of the following best describes what you did in this elecJon?
__ Definitely did not vote __ Definitely voted in person at a polling place on elec2on day __ Definitely voted in person at a polling place before elec2on day __ Definitely voted by mailing a ballot to elec2ons officials before elec2on day __ Definitely voted in some other way __ Not completely sure whether you voted or not
The QuesJon
ANES Self-‐report Turnout Rates
ANES Self-‐report Turnout Rates
ANES Self-‐report Turnout Rates
ValidaJng Turnout Self-‐reports
The ValidaJon Task • Match government records to survey respondents.
• Determine “correct” turnout for each respondent.
Two problems • 200,000,000+ people eligible to vote in the 2008.
• Some government records are difficult to obtain.
Commercial Vendors
• Several are available for records matching.
• Maintain naJonal databases of government records.
• Use records from mulJple sources to increase data accuracy.
• Developed technology for records matching.
Our Requirements
• Full disclosure about the vendor’s records. • When they were obtained.
• From whom were they obtained.
• How the records are cleaned.
• Full disclosure about the matching process.
• Details about their matching algorithm.
• Their criteria for determining a match.
What We Learned
• Vendors use names, addresses, and dates of birth to match records.
• Some vendors are more transparent than others.
• Some vendors can match records at different levels of confidence.
• No vendor was willing to provide complete details about their matching algorithm.
In-‐house Records Matching
• Obtain government registraJon and turnout records from a sample of states.
• Use a publicly available computer applicaJon to match respondents to government records.
• LinkPlus – Developed by the CDC
2008-‐2009 ANES Panel Study Our Subsample (N=761)
Respondents with a residenJal address in: • California • Florida • New York • North Carolina • Ohio • Pennsylvania
Weighted to reflect within-‐state landline populaJons:
• Age • Sex • Race • EducaJon • Income • Marital status
Matching Respondents to Records
Three methods of matching
Matching Respondents to Records
Last name First name Address DOB
Strict Iden2cal Iden2cal Iden2cal Iden2cal
Three methods of matching
Matching Respondents to Records
Last name First name Address DOB
Strict Iden2cal Iden2cal Iden2cal Iden2cal
Moderate Same or similar
Same or similar
Same or similar
Iden2cal
Three methods of matching
Matching Respondents to Records
Last name First name Address DOB
Strict Iden2cal Iden2cal Iden2cal Iden2cal
Moderate Same or similar
Same or similar
Same or similar
Iden2cal
Least Same or similar
Same or similar
Same or similar
Same or similar
Same or similar
Same or similar
Iden2cal
Iden2cal Iden2cal Iden2cal
Three methods of matching
Strict Moderate Least All target states
43% 59% 69%
Turnout Rates According to Government Records
Strict Moderate Least Popula2on All target states
43% 59% 69% 62%
“Correct” Turnout Turnout Rates According to Government Records
Strict Moderate Least Popula2on All target states
43% 59% 69% 62%
Using Government Records
If lying causes turnout over-‐repor2ng
• Moderate matching is the most valid
• Least matching is somewhat valid
• Strict matching is not valid
• Self-‐reports cannot be trusted
Turnout Rates According to Government Records
Problem Solved?
Problem: Respondents Lie
SoluJon: Use government records
Proof: Sample turnout=Popula2on turnout
DerivaJve: Sample registra2on=Popula2on registra2on
“Correct” registraJon
Popula2on All target states 83%
• PopulaJon registraJon rate based on staJsJcs published by states.
• RegistraJon rate among voJng age US ciJzens in landline households.
• Adjusted for deadwood.
RegistraJon Rates According to Government StaJsJcs
Popula2on Strict All target states 83% 49%
• Strict matching underesJmates populaJon registraJon rate.
“Correct” registraJon RegistraJon Rates According to Government StaJsJcs
Popula2on Strict Moderate All target states 83% 49% 66%
• Moderate matching underesJmates populaJon registraJon by 17%.
• Moderate matching appears invalid.
“Correct” registraJon RegistraJon Rates According to Government StaJsJcs
Popula2on Strict Moderate Least All target states 83% 49% 66% 78%
• Even Least matching underesJmates populaJon registraJon.
“Correct” registraJon RegistraJon Rates According to Government StaJsJcs
Compared to the PopulaJon
Strict Moderate Least Turnout Way Too Low Closest Too High RegistraJon Way Too Low Way Too Low Closest
What is going on?
CompeJng Biases
• Downward bias – Using government records incorrectly idenJfies some respondents as having not turned out.
• Upward bias – People who parJcipate in surveys are more likely to vote.
Evidence of an upward bias?
Turnout among the Registered
Among voJng-‐age US ciJzens in landline households WHO ARE REGISTERED:
Turnout=75%
Based on staJsJcs published by states, adjusted for deadwood
Turnout among the Registered
Among voJng-‐age US ciJzens in landline households WHO ARE REGISTERED:
Turnout=75%
Strict Moderate Least Turnout 88%
Strict matching suggests higher turnout among respondents
Turnout among the Registered
Among voJng-‐age US ciJzens in landline households WHO ARE REGISTERED:
Turnout=75%
Strict Moderate Least Turnout 88% 90%
Moderate matching suggests higher turnout among respondents
Turnout among the Registered
Among voJng-‐age US ciJzens in landline households WHO ARE REGISTERED:
Turnout=75%
Strict Moderate Least Turnout 88% 90% 88%
Least matching suggests higher turnout among respondents
Why the downward bias?
Self-‐Reported Voters
Truthful Record shows no turnout
Fail-‐to-‐match
Strict 50% 3% 47% Moderate 68% 4% 27% Least 80% 5% 15%
Self-report confirmed by government records
Self-‐Reported Voters
Truthful Record shows no turnout
Fail-‐to-‐match
Strict 50% 3% 47% Moderate 68% 4% 27% Least 80% 5% 15%
Self-report disconfirmed by government records
Self-‐Reported Voters
Truthful Record shows no turnout
Fail-‐to-‐match
Strict 50% 3% 47% Moderate 68% 4% 27% Least 80% 5% 15%
Self-report cannot be confirmed by government records
Problem: Respondents lie.
Problem: Sample turnout > PopulaJon turnout.
SoluJon: Use government records.
Using government records create an illusion of accuracy.
SoluJon: Use self-‐reports.
Turnout self-‐reports appear extremely accurate.
Sample Turnout ≠ PopulaJon Turnout
Thank You!
top related