applied opinion research training workshop sharon felzer, public opinion research specialist,...

Applied Opinion Research Training Workshop

Sharon Felzer, Public Opinion Research Specialist, Development Communications Division (DevComm)

Mary McIntosh, Princeton Research Survey

Why do Attitudes Matter?

Mary McIntosh

Why do attitudes matter?

Attitude research is a valuable resource for the Bank to– Identify perceptions, underlying motives,

and beliefs of specific audiences that influence behavior and shape attitudes toward policies, programs, institutions, organizations, etc.

Why do attitudes matter?• Perceptions of the Bank’s effectiveness• Where people believe the Bank needs to focus its

efforts• Areas in which people feel that the Bank has failed

in its objectives• Areas in which people feel that the Bank has not

met their expectations• People’s perceptions of a Bank program’s impact


We can differentiate people’s attitudes as a function of:– Standing in the country (e.g., elite, media,

beneficiary)– Familiarity/experience with the Bank– Geographic location– Other attitudes, such as opinions on

national government, privatization


To distinguish between reality and perception:– Our attitudes color our perceptions of the world and

it is our perceptions, not reality, that count• e.g., stereotypes, self-concept• e.g., one’s understanding of poverty reduction

plays a large role in determining how effective they regard the Bank’s poverty reduction efforts

Thus, attitudes are a critical element of evaluation


To complement other evaluative approaches:– Attitude research is not programmatic– Attitude research provides a different kind of

information than economic data– Attitude research is only as good as the instruments

used and the data collection effort – The findings of attitude research are highly

dependent on the sample used

Thus, attitude research complements other sources of research


Uses of attitude research:– Findings from attitude research can be used

to inform country team strategies, program initiatives, project designs, etc.

– Findings can also be used to improve an evaluation by making it more comprehensive and informed (potentially increasing the impact of the evaluation)

Using Attitude Research in Evaluations

Sharon Felzer

Using Attitude Research

Why use attitude research when conducting an evaluation?– To understand the paradigm of perception

vs. reality– As an assessment rather than in program/project preparation

(ex-post rather than ex-ante)– To understand gaps in knowledge– To expose barriers to behavior/attitude change of Bank

programs and services– To identify more effective ways to reach beneficiaries– To track attitudes, opinions, and behaviors

over time


Qualitative– Focus groups– In-depth interviews

(e.g., elites, sensitive issues)

– Brainstorming sessions

– Ethnographies

Quantitative– Polls– Attitude surveys– Tracking studies

Research Methodologies


Quantitative– Measures level of

occurrence/distribution

– Asks how many? how often? why? with statistical analysis

– Provides evidence for hypotheses

– Objective

Qualitative– Provides depth of

understanding

– Asks why? with follow-up

• Enables people to speak in their own words describing how they frame issues

– Enables hypothesis development

– Subjective


Common pitfalls of research efforts:– Poorly designed Terms of Reference– Poorly designed instruments– Poorly designed sample, poorly recruited focus

group/interviewees– Unskilled moderators/interviewers– Cost overrun– No buy-in


Tapping into sources of research– Budget issues exist– Use available resources to acquaint yourselves with

your specific audiences. For example:• Government commissioned population studies• Global/regional polls done by research firms like

PSRA, Environics, Eurobarometer, Gallup, and Latinobarometer. Access may cost a fee, but less than commissioning research yourselves.

Make sure the research is relevant!• Add questions to omnibus surveys

Using Attitude ResearchUsing attitude research tools in limited efforts– Budget issues exist– Even when there is no budget for research, it is wise to

incorporate research approaches into intelligence/ information gathering exercises. For example:

• Carefully designed & recruited qualitative research efforts are appropriate for particular audiences in particular times.

• These efforts require well designed guidelines (questionnaires for discussion) that ensure a consistent and systematic conversation followed by research-based analysis.

• Focus groups/guided discussions with a carefully selected sample of the general population tease out important issues related to the Bank’s efforts. Less likely with opinion leaders.

Using Attitude ResearchEthics

Attitude research is guided by stringent ethical rules laid out by organizations such as AAPOR, ESOMAR, and APA

o The confidentiality of respondents’ responses must be maintained

o The anonymity of respondents to the ultimate client must be maintained

o Results should be reported clearly and appropriately, always within the limitations of the research

o All precautions should be taken to reduce researcher/interviewer bias

Workshop Program Details

Exploration and hands-on work, including extensive use of case studies developed from a range of sources – The following areas will be covered:

• Defining objectives and identifying potential hypotheses (Day 1)

• Choosing a methodology and sampling (Day 1)• Designing quantitative instruments and qualitative instruments

(Day 2)• Preparing to conduct research (Day 3)• Analyzing and reporting results (Day 3)

Defining Objectives and Identifying Potential Hypotheses

Mary McIntosh

Objectives and HypothesesDefining objectives– Before starting any research program, you must first

identify the objectives of the research – Why is it important to define your objectives?

• Helps you to fully visualize the research project– Enables you to answer who? what? why? how?

• Helps identify potential hypotheses/assumptions• Helps to determine a budget• Helps to ensure that the Terms of Reference clearly lay

out the scope of the project

Objectives and HypothesesDefining objectives– In defining your objectives, you might want to

consider the following questions:• Why does the Bank want the research done?• What are the goals of the research?• How does the Bank intend to use the findings?• Is the research to be part of a larger project?• What other research has been done that can

complement your efforts?

Objectives and HypothesesDefining objectives– First, you should determine which questions need to

be answered– These questions can be specific or broad depending

on the research objectives• For instance:

– What are benefits of debt-relief in the context of general foreign assistance, as perceived by client governments ?

– How successful was a health project perceived by health professionals in a village?

– Value of OED reports to Board staff?

Objectives and Hypotheses

Defining objectives– Finding the right balance in your

research questions between being too specific or too broad is key

– Don’t use attitude research to learn what you can determine from other sources

• e.g., how may households in a community have electricity?


Defining objectives– You need to understand how your research

objectives are different from the intended objectives of the program, services, or other Bank initiative that you are evaluating

– To do this, you will want to speak to the task team leader, country team, etc.

• Do the research objectives seem appropriate to the task at hand?


Defining objectives– Finally, you will want to provide a

rationale for your proposed objectives• How will the research meet the

objectives you have laid out?


Defining objectives allows you to:

• Define the parameters of the research• Inform the design of the instrument• Organize the analysis

Logical hypotheses will then follow.

Choosing a Methodology and Sampling

Mary McIntosh

Choosing a Methodology

Qualitative Research– What is it?

• Open-ended: Questions are asked such that respondents are able to answer in any manner, at any length

– How is it conducted?• Focus groups, in-depth interviews,

brainstorming sessions, ethnographies• Guidelines (questionnaires for discussion)• Key sub-populations are recruited


Qualitative Research– Why use it?

• Provides depth of understanding• Answers why? how? with follow-up• Can be used to develop hypotheses • Often used to inform quantitative research

– Drawbacks:• Subjective• Results may not generalize• Language/translation nuances

Choosing a MethodologyQuantitative Research– What is it?

• Close-ended: Respondents are given specific responses to choose from in order to answer questions

– How is it conducted?• Polls, surveys, tracking studies • Fixed questionnaires • Random sample/quota sample

Choosing a MethodologyQuantitative Research– Why use it?

• Provides breadth of understanding• Answers how many? how often?

why? with statistical analysis• Measures level of occurrence/distribution• Results can be generalized to population• Objective• Systematic (can be replicated)

– Drawbacks:• May not capture full picture

Choosing a MethodologyQualitative vs. Quantitative

Sample Questions:– Qualitative: Why do you think corruption is

the most important development challenge in your country?

– Quantitative: Which one of the following do you think is the most important development challenge in your country? Poverty, Education, Corruption, etc.


Qualitative vs. Quantitative

Sample Questions:– Qualitative: How do you feel about the future

of the next generation in your country?– Quantitative: How do you feel about the

future of the next generation in your country? Very pessimistic, somewhat pessimistic, somewhat optimistic, very optimistic


Qualitative vs. Quantitative When to Use which Approach:– Question you want to answer – Participants’ ability to complete surveys– Type of data/responses desired– Sensitivity of issues– Budget


Qualitative vs. Quantitative When to Use which Approach:– You want to explore why people think the

Bank has certain problems within a programQualitative

– You want to find out how prevalent a certain problem is in Bank programsQuantitative


Qualitative vs. Quantitative When to Use which Approach:– You want to find out how a few key

decision makers feel about the BankQualitative

– You want to find out how a representative sample of key decision makers feels about the BankQuantitative

Sampling

Census (Full Population Representation)– The collection of data from all people in a given

population

Sample– A subset of people from the population

Any time data is collected from less than the whole population,

a sample has been taken

Sampling

Random Sample– A sample in which people have been selected

from the population by a process involving the use of random numbers. All people within a population have an equal probability of being selected

Non-Random Sample– A sample in which people have been selected by

means other than random selection

Sampling

Random Sample– Used to make generalizations about a population

or subpopulation– Reduces the possibility of sampling bias– Only type of sample for which inferential

statistics can appropriately be computed– Does not guarantee that the sample will closely

represent the population

SamplingSimple Random Sample– Most common form of random sampling– Used when objective is to draw inferences

about the whole population– Well-suited to drawing moderate-sized

samples (50-500) from relatively homogeneous populations or larger samples (500+) from heterogeneous populations

Sampling

Sampling is as much of an art as it is a science– Often the size of the population is unknown

• e.g, how many key decision makers are there in a country?

– In qualitative research especially, it is often unclear how large of a sample is necessary

• e.g., how many focus groups are necessary in order to understanding the perceptions of a Bank program?

Sampling

Response Rates– Percent of the intended sample from

which at least partial data is actually collected

– Low response rates pose the risk of substantial response bias

Sampling

Response Rates– How to maximize your response rate:

1. Clearly indicate how the requested information will contribute to something the respondents value

2. Provide credible promises of confidentiality 3. Use influential people to help solicit participation4. Minimize the inconvenience and burden on

respondents5. Offer incentives (e.g., a report)6. Be persistent (e.g., call back)

Hands-on-Work:Choosing a Methodology

applied opinion research training workshop sharon felzer, public opinion research specialist,...

Documents