The Journal of

Public Opinion Poll Question Databases:An Evaluation

by Stephen Woods

Available online 19 December 2006

This paper evaluates five polling resource:iPOLL, Polling the Nations, Gallup Brain, PublicOpinion Poll Question Database, and Polls and

Surveys. Content was evaluated on disclosurestandards from major polling organizations,

scope on a model for public opinion polls, andpresentation on a flow chart discussing search

limitations and usability.

Stephen Woods,Pennsylvania State University

bswoods@psu.eduN.

Academic Librarianship, Volume 33, Number 1, pages 41–46

INTRODUCTION

Graham Walden’s excellent guide to accessing the literature ofopinion polls was written in a time when sophisticated tools forsearching and retrieving polling information were just emerg-ing or under exploration.1 One type of tool that has emerged isthe electronic database that allows library users to search forterms in the questions of surveys or the public opinion pollquestion database.

The genius of this type of tool is that it allows library usersand professionals to search beyond the abstracts and controlledvocabulary used in traditional sources and provides opportu-nities to discover hidden variables often untapped in research.For the experienced researcher, it also offers new methods fordiscovering potential research data and literature. For thenovice, these tools supply generous access to summarystatistics and in some cases professional reports.

However, the accessibility of these variables also increases thepossibility of library users as well as reference librarians to makeegregious errors in selecting the appropriate and reliable statisticsfor their research.2 For example, a user might select a variablethat contains summary statistics about assisted suicide, but only400 of the 1200 surveyed answered that particular question. Thisis a serious problem and means that the quality of the statistic anddata related to this particular variable is questionable. Thesetypes of databases need to provide some level of evaluativeinformation that allows users to identify deficiencies.

Equally challenging is the increasing need for these data-bases to provide users and the reference librarian with a clearcontext for the information that is being presented. Forexample, many of these databases link the question recordsto copies of the questionnaire, codebook, the raw data, andspecial reports. This can be particularly confusing for thenovice user who is unaware of the relationship between thesedifferent types of resources.

BRIEF HISTORY OF POLLING DATABASES

The Congressional Research Service of the Library ofCongress awarded a contract in 1978 to the Survey ResearchCenter at the University of California Berkeley to develop aninformation retrieval system for survey data.3 They created oneof the first known computerized databases for public opinionpolls called POLL (Public Opinion Legislative Locator). This

January 2007 41

Figure 1

database functioned much like a libraries bibliographic onlinecatalog containing brief descriptive records from the results ofpublic opinion reported in news releases or printed reportsconducted by major public opinion polling organizations.

In 1983, the staff and directors of the Roper Center forPublic Opinion Research began the construction of a morecomplex computer-based retrieval system that would allowusers to search terms in the questions of public opinion pollsrather then brief abstracts.4 By 1986, the Roper Center beganproviding access to this databank, also call POLL, containing acollection of all substantive questions from surveys taken in theUnited States. Initially, State and Foreign samples were notincluded. The criteria for inclusion in the databank included:the survey must be systematic and scientific, data must berepresentative of a national sample, and the Roper Center mustbe given copies of complete question and wording responses.5

The Odum Institute in the early 1980s won a grant from theNational Science Foundation to look at repeated questions in121 Harris national surveys from 1963 to 1976.6 The text of thequestions and the responses were entered into a mainframedatabase system created by the Institute known as the Biblio-graphic Processing System. The results from this project wereeventually published in ‘‘Sourcebook of Harris NationalSurveys: Repeated Questions 1963–1976’’ by ElizabethMartin, Diana McDuffee, and Stanley Presser of the Institute.By the late 1980s and early 90s, the Institute decided to expandthis database from its early beginnings and bring it up to dateby adding other surveys.7 They added all of the Harris surveysand later questions from USA Today polls, Atlanta JournalConstitution polls, Southern Focus polls, Carolina Poll, andpolls from the National Network of State Polls.

The problem with mainframe technology was that it wasonly accessible to an elite few. ORS Publishing beganpromoting an electronic product in CD-ROM format calledPolling the Nations, 1986–1997. The database was originallycreated from the American Public Opinion Index and surveyquestions from their microfiche product American PublicOpinion Data. It was expanded to include surveys from morethen sixty other countries in Europe, Canada, Mexico, Africa,and Asia. Another publisher, Scholarly Resources, also beganselling its own electronic product in CD-ROM format calledThe Gallup poll public opinion, 1935–1997. This product wasprimarily an index to literature published by the GallupOrganization, but it also included a small sample of 60,000questions taken from surveys.

The advent of the Internet allowed all of these organizationsand publishers to explore ways to make their products moreaccessible to a much broader audience. Lexis/Nexis beganmarketing its own version based on the data it was able toobtain from the Roper Center as part of it reference suite calledPolls and Surveys. Later, the Roper Center marketed it ownversion, iPOLL, integrating it with the Centers bibliographicdata catalog. ORS Publishing mounted its data on the Internetin a product similarly called Polling the Nations. The OdumInstitute introduced it own version the Opinion Poll QuestionDatabase and integrated it with the Institutes bibliographicdatabase description of data. Recently, the Gallup Organizationintroduced its own product Gallup Brain that includesquestions and responses from people interviewed by the GallupPoll as well as summary reports from Gallup Poll TuesdayBriefings back to 1997 and Gallup Management Journal backto 1999.

42 The Journal of Academic Librarianship

The accessibility of summary statistics provided by thesepolling resources necessitates a better understanding by usersand reference librarians of the limitations of the content ofthese resources. This article is an evaluation of the content,scope and presentation of five polling Web-based resourcedatabases: iPOLL, Polling the Nations, Gallup Brain, PublicOpinion Poll Question Database, and Polls and Surveys.

METHOD OF EVALUATION

The scope, context, and usability of each product will beevaluated based on criteria developed from Dodd’s article whichidentifies five major contextual stages in the evolution of apublic opinion poll (see Fig. 1): the idea, development of aquestionnaire, identification of the sample or population,collection of the data, and finally the report (article, statistics,tables). The evaluation of the scope of each product will be basedon the extent access is provided to four products of this process:the questionnaire, codebook, raw data, and special reports.

A flow chart was created to represent each Web page that auser will potentially encounter in a session to assess theusability. Using Dodd’s five contextual stages for publicopinion polls, an assessment of contextual clarity will beprovided as well as a discussion of the searching capabilitiesand limitations of each product.

Finally, a set of core content questions for each product wereidentified by analyzing codes and standards for publishingreports or summaries from public opinion polls. These includepublications of the National Council on Public Polls,8 Councilof American Survey Research Organizations,9 World Associ-ation of Opinion and Marketing Research,10 AmericanAssociation for Public Opinion Research, 11 and the EuropeanSociety of Marketing and Research:12

! Code of Professional Ethics and Standards13

! Best Practices for Survey and Public Opinion Research14

! Code of Standards and Ethics for Survey Research15

! CASRO Guidelines for Survey Research Quality16

! Code of Professional Ethics and Practices17

! Principles of Disclosure18

Figure 3

! 20 Questions A Journalist Should Ask About Poll Results19

! ESOMAR/WAPOR Guide to Opinion Polls20

! ICC/ESOMAR International Code of Marketing and SocialResearch Practice21

From these documents, nine content questions wereidentified and used as a standard for evaluating whether ornot the product provides the necessary information in theirresponses:

! What is the purpose of the poll?

! Who is the sponsor of the poll?

! Who is the organization that conducted the poll?

! What questions were asked?

! In what order were the questions asked?

! Who was polled?

! How were the interviews conducted?

! When was the poll done?

! What statistics are offered to substantiate accuracy?

EVALUATION

Polling the Nations

The scope of this product does not include direct access tothe questionnaires, raw data, codebooks, or special reports. It isprimarily a database of selected questions from questionnaires.However, each record does provide contact information wheretheoretically the user can request copies of the questionnaire andraw data. The problem with this approach is often the companyor individuals responsible for the data did not archive the data.

The limited scope of Polling the Nations results in a productthat is fairly easy to navigate, but as we will see provides somechallenges that significantly affect a user’s ability to evaluatethe content. As the flowchart indicates (see Fig. 2), the usernavigates from the home page to a search screen, a result list,and ultimately to a question record. However, from the homepage, there is a glossary as well as a link to information aboutpolling that provides the user with some contextual informationabout the processes and limits of information associated withpublic opinion polls.

From the search screen, users can search by: topic, question,universe (geographic area covered by the survey), date, source(organization responsible for survey), or results. The controlled

Figure 2

vocabulary used in the topic field search is fairly comprehen-sive and can be useful as controlled vocabulary. However, thequestion and result field search can be somewhat confusing forthe user unless they understand what it is they are searching.The user has to be aware that they might miss some relevantquestion if they fail to search both of these fields separately.For example, if you wanted to know what percentage ofchildren play baseball after school compared to other sportsand you only searched in the question field, you couldpotentially miss questions where baseball was a possibleresponse. Finally, this product provides a nice export featurethat allows users to export large sets of questions for analysis.

This product provides information for five of our contentquestions: the sponsor, organization who conducted the poll,how the interviews were conducted, who was polled, and whenthe poll was done. However, the user should be cautioned thatthe notes field will often provide added information regardingthe sample. For example, the sample size for the questionnairemay be 1000, but sample size for that particular question maybe only 394 self-described baseball fans.

One of our questions is only partially addressed by thisproduct: what other questions were asked. A button erro-neously labeled ‘‘view all questions in this poll’’ is provided atthe question record level. This button in reality only allows theuser to see other questions from the survey that happened to bein the database. A related content question omitted from thisproduct is the order of the questions. This can be a significantproblem if a user is trying to evaluate the response particularlyif there are leading questions. For example, if a user is lookingat a question pertaining to the privatization of social securityand the question before it is about the Enron scandal, then usershould be asking some serious question about the design of thesurvey. Finally, two other content questions not addressed bythis product are the purpose of the poll and statistics tosubstantiate accuracy. Ironically, this product provides adetailed discussion from their home page on margin of errorand confidence levels without providing the user with thisinformation in the question record.

Lexis/Nexis’ Polls and Surveys

Questionnaires, raw data, codebooks, or special reports arenot included in the scope of this product as well. The recordsdo provide contact information for obtaining the raw data andquestionnaires, but the same problem mentioned above makesthis a tenuous solution at best. The content for this productcomes delayed from the Roper Center for Public OpinionResearch and is not as up-to-date as Roper’s iPOLL product.

The limited scope of this product also allows for a fairlysimple navigation (see Fig. 3). However, a noted disadvantageto this resource is that it is buried within a suite of resourcesforcing the user to navigate through three screens beforefinding the search interface.

January 2007 43

Figure 4

From the search screen, the user can search by keyword orquestion. The user can also narrow their search by popula-tion (U.S. voters and U.S. adults), survey sponsors, surveyorganization, source document, number of participants, anddate. The keyword search is a nice feature that allows theuser to avoid the confusion of searching for the term only inthe question or the response. Although this product providesa field for limiting a search to survey sponsors andorganizations, it would have been helpful to provide a listof source documents, sponsors, and survey organizations forthe user to select. There is also a serious problem with thefeature that allows the user to limit by number of participants.This product does not take into consideration that somequestions are actually responses of a subpopulation of thetotal population found in the ‘‘number of participants’’ fieldof the record. Who was polled is only partially addressed bythis product, because it neglected to add the notes forinformation regarding the numbers for sub-populations forcertain questions.

Four of our content questions are provided by this product:the sponsor, organization who conducted the poll, how theinterviews were conducted, when the poll was done. Who waspolled is only partially answered. One of the content questionspartially addressed by this product and difficult to discover is:what other questions were asked. By searching the first part ofthe field question ID (i.e., USGALLUP.05FEB25); a list ofsome of the other questions can be teased out of the resource.However, the user must be aware that this is not acomprehensive list of only those questions from the surveythat happened to be added. A related content question, questionorder, is also omitted from this resource limiting the user’spotential for evaluation. Finally, this product provides noinformation about our content questions pertaining to thepurpose of the poll and provides no statistics (beyondfrequency) to assist the user.

Gallup Brain

The scope of this product provides the users to specialreports back to 1997 published by the Gallup Organization,but does not provide access to the questionnaires, codebooks,or the raw data. Although the product is exclusively questionsfrom surveys and polls conducted by the Gallup Organization,the raw data and questionnaires can be purchased directlyfrom the Gallup Organization.

Navigation within Gallup Brain is fairly complicated,particularly if the user is not familiar with the relationshipand differences between the questionnaires and special reports(see Fig. 4). Adding to the confusion, this product providestwo different types of reports: special reports and trend reportanalysis. Trend reports provide the user with analysis on atopic over time, whereas the special reports provide the userwith analysis of a single poll at a specific point in time.

From the home screen, the user can select a tab fordocuments, topics and trends, and searching. The documentstab allows the user to browse the latest special reports andtrend reports published in Tuesday Briefings and the GallupManagement Journal. Also, the user can browse a list ofthe latest questionnaires with a link to the questionnaireprofile record. The topics and trends tab provides the userwith a list of subjects the user can browse with links tospecial reports or trends trend reports in the documentrecord.

44 The Journal of Academic Librarianship

From the search screen, the user can do a keyword searchfor documents (special or trend reports) or questionnaires.There is an advanced search screen that allows a user to searchby date and phrases. Also, from the search screen, the user canbrowse a nifty chronology of selected polls by decades. If theuser searches for documents, the results list will contain bothspecial and trend reports. If the user searches the question-naires, the results list will contain a list of questions fromvarious polls. One of the limitations users will need to be awareof is that this product does not search the question responsesonly the questions. From the question record, by clicking onthe link in the questionnaire field, the user can see moreevaluative information about the questionnaire as well as a listof some of the other questions from the survey. Finally, theRoper Center is beginning to experiment with providing exportfeatures for users who want to download large sets of questionsfor analysis.

An evaluation of the content of this product is difficult,because the product is somewhat complex providing evaluativeinformation for questions on the question profile and ques-tionnaire profile. Trend and special reports must be evaluatedseparately, because it is difficult to trace the relationshipbetween the questions and the reports.

For the questions, five of our content questions are answeredin the question and questionnaire profile: the sponsor,organization who conducted the poll, how the interviews wereconducted, who was polled, and when the poll was done. Thisproduct does supply a field for sub-populations whennecessary. The questions that were asked and the order ofquestions are only partially provided by this product. A mean isalso provided on the question record when applicable.

The special and trend reports in this product are all verydifferent. There is no standard set of information that isprovided in every case. Some reports provide information forall nine content questions including confidence values andmargins of error. Other reports particularly trend reportsprovide very little evaluative information. The purpose of thepoll is not provided for special reports, trend reports, orquestions.

Figure 6

Ipoll

What makes the scope of this product unique is itsintegration with the Roper Center’s Catalog of Holdings andaccess to the raw data, questionnaires, and codebooks througha complementary product RoperExpress. However, what canbe frustrating for the user is the fact that many of the questionsin the iPOLL database are taken from polls and surveys that theRoper Center does not own or have access to. Finally, neitherproduct provides access to special reports.

The navigational structure of iPOLL and integration withRoperExpress are fairly straightforward given the complexityof the resources (see Fig. 5). If the user has access toRoperExpress, there will be a link provided from the questionrecord to the Catalog of Holdings record, data set, andcodebooks.

From the search screen, the user can do a keyword search orbrowse from a pull-down menu of a list of topics. The user canalso browse a list of polling organizations to limit their search.This is a great addition for advanced searchers who may beaware of the focus and reputation of certain organizations, butmight not be helpful for the novice. A description of theorganizations and types of surveys they conduct would behelpful. The search screen also allows the user to limit theirsearch and sort by date. A significant searching limitation ofthis product is that it only searches the questions and not thequestion responses.

iPOLL provides information for five of our content questions:the sponsor, organization who conducted the poll, how theinterviews were conducted, who was polled, and when the pollwas done. To find other questions that were asked on the survey,iPOLL provides a link to other questions in the survey. However,the user must be aware that this is not a comprehensive list of allthe questions nor are they in order. To look a list of all thequestions on a particular survey as well as the order, the user willhave to look at the codebook provided through RoperExpress.Consequently, the content provide for evaluating all thequestions on the survey and their order is only partially providedfor users that have access to RoperExpress and not available atall other users.

Frequency statistics are provided for the questions as well asinformation about non-respondents. However, iPOLL does notprovide statistics such as confidence values, means, or marginsof errors. The Roper Center is beginning to experiment with

Figure 5

online analysis tools. Finally, there is no information providedabout the purpose of the survey or poll.

Public Opinion Poll Question Database

What makes this product different is the fact that all of thequestions in the database are linked to their correspondingcodebooks, questionnaires, and the raw data. It does notprovide the users with related special reports. The problem withthis approach is that given the complexity of mounting the data,codebooks, and raw data and the fact that the resources are freethe questions can be a little dated. For example, the newestquestion in the database on baseball is 2003.

This is a fairly complex product that can pose somenavigational challenges for the user (see Fig. 6). This isprimarily due to the fact that in this product the Odum Instituteis integrating two products: question database (bold text) andthe catalog of data holdings (italicized text).

From the search screen, users can do a keyword search andlimit their search to words in the question text, surveyinformation, or anywhere. There is also capability for limitingto certain polls: Carolina polls, Louis and Harris, Southernpolls, State polls, and Knight Foundation Community polls.Finally, a user can limit their search to polls since certain dates.This product also allows the user to sort their results by mostcurrent or oldest. Users can also download large sets (500questions) of questions at a time for analysis as well.

This product provides access to eight of our nine contentquestions: the sponsor, organization who conducted the poll,how the interviews were conducted, who was polled, when thepoll was done, what questions were asked, order of thequestions, and the purpose of the poll.Frequencies are providedon the question detail. There is no information provided forconfidence values or margin of error. However, this productdoes provide users with the ability to run simple crosstabulations with weighting through a Web browser.

CONCLUSION

Academic Librarians are facing increasing demands for accessto electronic resources along with diminishing budgets. One ofthe important outcomes from this evaluation is to providelibrarians with a better understanding of the scope, context, andlimitations of these polling resources to allow them to makeinformed decisions with respect to their own collection

January 2007 45

development efforts. Equally important is to provide librarianswithout a statistical background with objective tools to measurethe quality of the content of future resources or revisions tothese existing resources. Our users need to be able to makeinformed decisions with respect to the content of statisticalinformation that these resources provide.

My sincere hope is that this evaluation will provide bench-marks for publishers with respects to the content of pollinginformation they are providing. Furthermore, publishers need tobegin to consider new ways to provide the users with thecontext of the information they are providing (see Fig. 1).

NOTES AND REFERENCES

1. Graham Walden, ‘‘Public Opinion Polls: A Guide to Accessing theLiterature,’’ Reference Services Review 4 (1988): 65–74.

2. Stephen Woods, ‘‘Essential Questions for Evaluating PublicOpinion Polls,’’ Reference Services Review (in press);Herbert Asher, Polling and the Public: What Every Citizen ShouldKnow (5th Edition CQ Press, Washington, DC, 2001);Shelden Gawiser and Evan G. Witt, A Journalist Guide to PublicOpinion Polls (Praeger Publishers, Westport, Connecticut, 1994);See also, American Statistical Association, ASA Series: What is aSurvey? http://www.amstat.org/sections/SRMS/whatsurvey.html(accessed July 3, 2006).

3. Dodd, Sue Dodd, ‘‘Characteristics and Sources of Public OpinionPolls in the United States,’’ Numeric Databases. (Norwood, NewJersey: Ablex Publishing Corporation, 1984), 153–187.

4. The Roper Center which was established in 1946 at WilliamsCollege by Elmo Roper. It was Ropers contention that opinion polldata would be valuable for historians in the future. By 1977 theRoper Center had outgrown Williams College and moved to theUniversity of Connecticut in order to utilize the resources of amajor university. See also ‘‘The Roper Public Opinion ResearchCenter,’’ in Reader in Machine-Readable Social Data (Engle-wood, Colorado: Information Handling Services, 1977), 130–140.

5. John Barry, ‘‘The Roper Center: The World’s Largest Archive ofSurvey Data,’’ Reference Services Review 1–2 (1988): 41–50;E. Martin, D. McDuffee and S. Presser, Sourcebook of Harrisnational surveys: repeated questions 1963–1976 (Institute forResearch in Social Science, University of North Carolina at ChapelHill, 1981).

6. The Odum Institute affiliated with the University of North Carolinawas founded in 1924 by Howard W. Odum to support research andtraining of social science faculty and graduate students. Theinstitute maintains one of the oldest archives of polling data in theUnited States and is the exclusive repository for Louis Harrispublic opinion data. http://www2.irss.unc.edu/irss/home.asp(accessed July 3, 2006).

7. (David Sheaves, personal communication, June 28, 2005).8. The National Council on Public Polls was founded in 1969primarily to assist journalist in understanding, interpreting andreporting polls. In 2000, the NCCP created a Polling ReviewBoard (PRB) to monitor the conduct and reporting of polls anddetailed procedures to determine whether or not surveys meet thedisclosure agreements. http://www.ncpp.org (accessed July 3,2006).

9. The Council of American Survey Research Organizations startedin 1975 as a trade association of survey research organizations and

46 The Journal of Academic Librarianship

currently represents over 250 companies and research operations inthe United States and abroad, http://www.casro.org (accessed July3, 2006).

10. The World Association for Public Opinion Research foundedin 1947 with the objective of providing a worldwide meetingground for those working in survey research with particularemphasis on uniting those working in academic settings withthose doing similar work in the private sector. It is aprofessional association of individuals, rather than organiza-tions or institutions, http://www.wapor.org (accessed July 3,2006).

11. The American Association for Public Opinion Research wasstarted in 1947 by a group of researchers interested in theadvancement of public opinion research. The AAPOR is currentlycomprised of 1600 representing academic institutions, commercialfirms, government agencies, and non-profit groups as producersand users of survey data, http://www.aapor.org (accessed July 3,2006).

12. Founded in 1948 as the European Society for Opinion andMarketing Research, ESOMAR now unites 4000 members, bothclients and providers of market research, in over 100 countries,making it the only organization that unites market researchprofessionals on a worldwide level, http://www.esomar.org(accessed July 3, 2006).

13. American Association for Public Opinion Research. Code ofProfessional Ethics and Standards. http://www.aapor.org/pdfs/best_pra.pdf (accessed July 3, 2006).

14. American Association for Public Opinion Research, Best Practicesfor Survey and Public Opinion Research http://www.aapor.org/default.asp?page=survey_methods/standards_and_best_practices/code_for_professional_ethics_and_practices (accessed July 3,2006).

15. Council of American Survey Research Organizations, Code ofStandards and Ethics for Survey Research. http://www.casro.org/codeofstandards.cfm (accessed July 3, 2006).

16. Council of American Survey Research Organizations, Guidelinesfor Survey Research Quality. http://www.casro.org/guidelines.cfm(accessed July 3, 2006).

17. World Association of Opinion and Marketing Research, Code ofProfessional Ethics and Practices. http://www.unl.edu/WAPOR/ethics.html (accessed July 3, 2006).

18. National Council on Public Polls, Principles of Disclosure.http://www.ncpp.org/docs/Principles%20Disclosure%200409.pdf(accessed July 3, 2006).

19. Shelden Gawiser and Evan G. Witt, 20 Questions a Journalistshould Ask about Poll Results http://www.ncpp.org/qajsa.htm(accessed July 3, 2006).

20. European Society of Marketing Research and World Associationof Opinion and Marketing Research, Guide to Opinion Polls.http://www.esomar.org/esomar/show/id=103897 (accessed July 3,2006).

21. International Chamber of Commerce and Society of Market-ing Research, ICC/ESOMAR International Code of Marketingand Social Research Practice. http://www.iccwbo.org/home/statements_rules/rules/1995/esomcod.asp (accessed July 3,2006);The ICC World Council is the equivalent of the generalassembly of a major intergovernmental organization. The bigdifference is that the delegates are business executives andnot government officials. http://www.iccwbo.org/index.asp.