applied opinion research training workshop
DESCRIPTION
Applied Opinion Research Training Workshop. Sharon Felzer , Public Opinion Research Specialist, Development Communications Division (DevComm) Mary McIntosh , Princeton Research Survey. Why do Attitudes Matter?. Mary McIntosh. Why do attitudes matter?. - PowerPoint PPT PresentationTRANSCRIPT
Applied Opinion Research Training Workshop
Sharon Felzer, Public Opinion Research Specialist, Development Communications Division (DevComm)
Mary McIntosh, Princeton Research Survey
Why do Attitudes Matter?
Mary McIntosh
Why do attitudes matter?
Attitude research is a valuable resource for the Bank to– Identify perceptions, underlying motives,
and beliefs of specific audiences that influence behavior and shape attitudes toward policies, programs, institutions, organizations, etc.
Why do attitudes matter?• Perceptions of the Bank’s effectiveness• Where people believe the Bank needs to focus its
efforts• Areas in which people feel that the Bank has failed
in its objectives• Areas in which people feel that the Bank has not
met their expectations• People’s perceptions of a Bank program’s impact
Why do attitudes matter?
We can differentiate people’s attitudes as a function of:– Standing in the country (e.g., elite, media,
beneficiary)– Familiarity/experience with the Bank– Geographic location– Other attitudes, such as opinions on
national government, privatization
Why do attitudes matter?
To distinguish between reality and perception:– Our attitudes color our perceptions of the world and
it is our perceptions, not reality, that count• e.g., stereotypes, self-concept• e.g., one’s understanding of poverty reduction
plays a large role in determining how effective they regard the Bank’s poverty reduction efforts
Thus, attitudes are a critical element of evaluation
Why do attitudes matter?
To complement other evaluative approaches:– Attitude research is not programmatic– Attitude research provides a different kind of
information than economic data– Attitude research is only as good as the instruments
used and the data collection effort – The findings of attitude research are highly
dependent on the sample used
Thus, attitude research complements other sources of research
Why do attitudes matter?
Uses of attitude research:– Findings from attitude research can be used
to inform country team strategies, program initiatives, project designs, etc.
– Findings can also be used to improve an evaluation by making it more comprehensive and informed (potentially increasing the impact of the evaluation)
Using Attitude Research in Evaluations
Sharon Felzer
Using Attitude Research
Why use attitude research when conducting an evaluation?– To understand the paradigm of perception
vs. reality– As an assessment rather than in program/project preparation
(ex-post rather than ex-ante)– To understand gaps in knowledge– To expose barriers to behavior/attitude change of Bank
programs and services– To identify more effective ways to reach beneficiaries– To track attitudes, opinions, and behaviors
over time
Using Attitude Research
Qualitative– Focus groups– In-depth interviews
(e.g., elites, sensitive issues)
– Brainstorming sessions
– Ethnographies
Quantitative– Polls– Attitude surveys– Tracking studies
Research Methodologies
Using Attitude Research
Quantitative– Measures level of
occurrence/distribution
– Asks how many? how often? why? with statistical analysis
– Provides evidence for hypotheses
– Objective
Qualitative– Provides depth of
understanding
– Asks why? with follow-up
• Enables people to speak in their own words describing how they frame issues
– Enables hypothesis development
– Subjective
Using Attitude Research
Common pitfalls of research efforts:– Poorly designed Terms of Reference– Poorly designed instruments– Poorly designed sample, poorly recruited focus
group/interviewees– Unskilled moderators/interviewers– Cost overrun– No buy-in
Using Attitude Research
Tapping into sources of research– Budget issues exist– Use available resources to acquaint yourselves with
your specific audiences. For example:• Government commissioned population studies• Global/regional polls done by research firms like
PSRA, Environics, Eurobarometer, Gallup, and Latinobarometer. Access may cost a fee, but less than commissioning research yourselves.
Make sure the research is relevant!• Add questions to omnibus surveys
Using Attitude ResearchUsing attitude research tools in limited efforts– Budget issues exist– Even when there is no budget for research, it is wise to
incorporate research approaches into intelligence/ information gathering exercises. For example:
• Carefully designed & recruited qualitative research efforts are appropriate for particular audiences in particular times.
• These efforts require well designed guidelines (questionnaires for discussion) that ensure a consistent and systematic conversation followed by research-based analysis.
• Focus groups/guided discussions with a carefully selected sample of the general population tease out important issues related to the Bank’s efforts. Less likely with opinion leaders.
Using Attitude ResearchEthics
Attitude research is guided by stringent ethical rules laid out by organizations such as AAPOR, ESOMAR, and APA
o The confidentiality of respondents’ responses must be maintained
o The anonymity of respondents to the ultimate client must be maintained
o Results should be reported clearly and appropriately, always within the limitations of the research
o All precautions should be taken to reduce researcher/interviewer bias
Workshop Program Details
Exploration and hands-on work, including extensive use of case studies developed from a range of sources – The following areas will be covered:
• Defining objectives and identifying potential hypotheses (Day 1)
• Choosing a methodology and sampling (Day 1)• Designing quantitative instruments and qualitative instruments
(Day 2)• Preparing to conduct research (Day 3)• Analyzing and reporting results (Day 3)
Defining Objectives and Identifying Potential Hypotheses
Mary McIntosh
Objectives and HypothesesDefining objectives– Before starting any research program, you must first
identify the objectives of the research – Why is it important to define your objectives?
• Helps you to fully visualize the research project– Enables you to answer who? what? why? how?
• Helps identify potential hypotheses/assumptions• Helps to determine a budget• Helps to ensure that the Terms of Reference clearly lay
out the scope of the project
Objectives and HypothesesDefining objectives– In defining your objectives, you might want to
consider the following questions:• Why does the Bank want the research done?• What are the goals of the research?• How does the Bank intend to use the findings?• Is the research to be part of a larger project?• What other research has been done that can
complement your efforts?
Objectives and HypothesesDefining objectives– First, you should determine which questions need to
be answered– These questions can be specific or broad depending
on the research objectives• For instance:
– What are benefits of debt-relief in the context of general foreign assistance, as perceived by client governments ?
– How successful was a health project perceived by health professionals in a village?
– Value of OED reports to Board staff?
Objectives and Hypotheses
Defining objectives– Finding the right balance in your
research questions between being too specific or too broad is key
– Don’t use attitude research to learn what you can determine from other sources
• e.g., how may households in a community have electricity?
Objectives and Hypotheses
Defining objectives– You need to understand how your research
objectives are different from the intended objectives of the program, services, or other Bank initiative that you are evaluating
– To do this, you will want to speak to the task team leader, country team, etc.
• Do the research objectives seem appropriate to the task at hand?
Objectives and Hypotheses
Defining objectives– Finally, you will want to provide a
rationale for your proposed objectives• How will the research meet the
objectives you have laid out?
Objectives and Hypotheses
Defining objectives allows you to:
• Define the parameters of the research• Inform the design of the instrument• Organize the analysis
Logical hypotheses will then follow.
Choosing a Methodology and Sampling
Mary McIntosh
Choosing a Methodology
Qualitative Research– What is it?
• Open-ended: Questions are asked such that respondents are able to answer in any manner, at any length
– How is it conducted?• Focus groups, in-depth interviews,
brainstorming sessions, ethnographies• Guidelines (questionnaires for discussion)• Key sub-populations are recruited
Choosing a Methodology
Qualitative Research– Why use it?
• Provides depth of understanding• Answers why? how? with follow-up• Can be used to develop hypotheses • Often used to inform quantitative research
– Drawbacks:• Subjective• Results may not generalize• Language/translation nuances
Choosing a MethodologyQuantitative Research– What is it?
• Close-ended: Respondents are given specific responses to choose from in order to answer questions
– How is it conducted?• Polls, surveys, tracking studies • Fixed questionnaires • Random sample/quota sample
Choosing a MethodologyQuantitative Research– Why use it?
• Provides breadth of understanding• Answers how many? how often?
why? with statistical analysis• Measures level of occurrence/distribution• Results can be generalized to population• Objective• Systematic (can be replicated)
– Drawbacks:• May not capture full picture
Choosing a MethodologyQualitative vs. Quantitative
Sample Questions:– Qualitative: Why do you think corruption is
the most important development challenge in your country?
– Quantitative: Which one of the following do you think is the most important development challenge in your country? Poverty, Education, Corruption, etc.
Choosing a Methodology
Qualitative vs. Quantitative
Sample Questions:– Qualitative: How do you feel about the future
of the next generation in your country?– Quantitative: How do you feel about the
future of the next generation in your country? Very pessimistic, somewhat pessimistic, somewhat optimistic, very optimistic
Choosing a Methodology
Qualitative vs. Quantitative When to Use which Approach:– Question you want to answer – Participants’ ability to complete surveys– Type of data/responses desired– Sensitivity of issues– Budget
Choosing a Methodology
Qualitative vs. Quantitative When to Use which Approach:– You want to explore why people think the
Bank has certain problems within a programQualitative
– You want to find out how prevalent a certain problem is in Bank programsQuantitative
Choosing a Methodology
Qualitative vs. Quantitative When to Use which Approach:– You want to find out how a few key
decision makers feel about the BankQualitative
– You want to find out how a representative sample of key decision makers feels about the BankQuantitative
Sampling
Census (Full Population Representation)– The collection of data from all people in a given
population
Sample– A subset of people from the population
Any time data is collected from less than the whole population,
a sample has been taken
Sampling
Random Sample– A sample in which people have been selected
from the population by a process involving the use of random numbers. All people within a population have an equal probability of being selected
Non-Random Sample– A sample in which people have been selected by
means other than random selection
Sampling
Random Sample– Used to make generalizations about a population
or subpopulation– Reduces the possibility of sampling bias– Only type of sample for which inferential
statistics can appropriately be computed– Does not guarantee that the sample will closely
represent the population
SamplingSimple Random Sample– Most common form of random sampling– Used when objective is to draw inferences
about the whole population– Well-suited to drawing moderate-sized
samples (50-500) from relatively homogeneous populations or larger samples (500+) from heterogeneous populations
Sampling
Sampling is as much of an art as it is a science– Often the size of the population is unknown
• e.g, how many key decision makers are there in a country?
– In qualitative research especially, it is often unclear how large of a sample is necessary
• e.g., how many focus groups are necessary in order to understanding the perceptions of a Bank program?
Sampling
Response Rates– Percent of the intended sample from
which at least partial data is actually collected
– Low response rates pose the risk of substantial response bias
Sampling
Response Rates– How to maximize your response rate:
1. Clearly indicate how the requested information will contribute to something the respondents value
2. Provide credible promises of confidentiality 3. Use influential people to help solicit participation4. Minimize the inconvenience and burden on
respondents5. Offer incentives (e.g., a report)6. Be persistent (e.g., call back)
Hands-on-Work:Choosing a Methodology