Measurement

• The hardest part of doing research? You’ll see when we begin operationalizing concepts May seem easy/trivial/even boring, but it is crucial• Most important part of research? Fancy statistics on poor measurements are a problem.

The Measurement Process: “Operationalization”

--------------------------------------------------------

Concept

↓

Conceptual Definition

↓

Operational Definition

↓

Variable

Concepts are vague

Empirical political research analyzes concepts and the relationship between them but what is

– Education?– Feminism?– Globalization?– Liberalism?– Democracy?

• Even “easier” concepts may be hard to

define:

Partisanship of voters

Number of political parties in a country

Political tolerance

Conceptual Definition:properties and subjects

Must communicate three things:

1. The variation within a characteristic

2. The subject or groups to which the concept applies

3. How the characteristic is to be measured

E.g.: The concept of ______ is defined as the extent to which _____ exhibits the characteristic of ______.

Try: tolerance, democracy, capitalism, liberalism, etc.

Operational Definition: How does one measure the concept?

• Critical/necessary step for analysis to be possible

• Toughest part

One needs to be very specific

• Easiest to criticize

Almost always problems/exceptions

Need to defend measures thoroughly

OperationalizationA simple example

• Education (how well individuals are educated)

How might we measure it?

• Problems with possible definitions?

• What operationalization is actually used?

• Advantages?

Simple to use

Seems right in most instances

Almost impossible to think of a better

measure

• Disadvantages

Some examples are problematic

OperationalizationA more difficult example

• People’s political partisanship

Conceptual definition: how people feel about

the Democratic v. the Republican party (or

loyalty to the parties, or party attachments)

How might we measure it?• Problems with possible definitions?• What operationalization is actually used?

• Advantages?

Applies to voters and nonvoters alike

Avoids problems of which elections to

use, etc.

Notion of deviating from ID is useful

As often asked, provides strength of ID

as well as direction

• Disadvantages? The leaner problem (text, p. 17) It doesn’t travel well.• A point about its use You see it a lot in the media E.g., did Bush win over Dem’s? How men and women differ on ID? Has the % of ind’s increased?

OperationalizationA deceptively hard example

• Number of political parties in a country

Appears easy: any problems with it?

• What operationalization is actually used?

• Advantages? The way it deals with small parties• Disadvantages Some examples are problematic• A point about its use How good it is may depend on what it is used for A conceptual question again

Reliability and validity

• How well does an operationalization work?

• Begin (see text, p. 14) by defining

Measurement = Intended characteristic

+ Systematic error + Random error.

• Usually judged by assessing

Validity

Reliability

Validity

• Definition is easy:

Does a measure gauge (or, measure)

the intended characteristic and only

that characteristic• But it is difficult to apply:

How do we know what is being measured?• Refers to problems of systematic error

But saying that doesn’t help a whole lot

Validity tests

• Face validity: does the measure look like it measures what it’s supposed to?

Occasionally useful—at least if a measure does not pass this test. Usually no explicit tests are made to determine face validity, but the term is used loosely (Shull & Vanderleeuw)

• Construct validity: concerned with the relationship of a given measure with other measures—e.g., is the SAT a good predictor of success in college?

Useful to a degree But how strong a relationship is required?• Other, related tests (content, criterion-

related validity) are similar

An aside on Hawthorne effects

• Effects that are a result of individuals’ awareness that they are being tested

Origin in an industrial study• Very important in experiments Disguising the purpose of an experiment helps• Analogous impact in psc is in surveys E.g., survey on elections makes people more attentive to them, more likely to vote

Reliability

• A measure is reliable to the extent that it is consistent—i.e., there is no random error

Scale, or guns, are good examples Note: Reliability ≠ Validity• Random Error (noise), never without Unlike with validity, there are tests of reliability

Evaluating Reliability

• Four methods (two mentioned in text) Test-retest method. Problem: learning effect Alternative forms. Problem: equivalent forms? Split-half method. Problem: multiple halves Internal consistency. Generalization of split half. Best; most often used

• Reliability methods

All rely on correlations (later in course)

Best internal consistency method

averages all split-half correlations

This method is called alpha. Simple

formula you can learn if you need to

(Varies between 0 and 1.)

• Validity/reliability concepts apply not just to tests or survey items. Think about:

Profit as measure of CEO ability Gun registrations as measure of gun ownership Reported crimes as a measure of the crime rate• Even “hard” data can be invalid/unreliable

A real-world example

• Interesting, important concept: support for democracy

• Conceptual definition: how much people in various countries say they support (or prefer, or would like) a democratic government.

• Operationalization (survey): Agree or disagree: “Democracy has its problems, but it’s better than any other form of government.”

• Surveys have often found high levels of support for democracy using this kind of measure

• Question: is this a valid measure of support for democracy?

Variables

• Actual measurement of the concept

Variable name v. variable’s values

As long as you remember this

distinction, you shouldn’t have a

problem

• Examples:

Religion (Protestant, Catholic, Jewish, etc.)

Height (values in feet and inches)

Variables (cont.)

• Residual categories--a small, but often nagging point

Cases (respondents, counties,

countries, etc.) for which the data is

missing

• We’ll deal these later—just note the

problem here

Levels of Measurement

• Nominal (least precise): categorical– E.g. Protestant, Catholic, Jewish, Atheist

• Ordinal: relative difference (higher/lower; for/against)– E.g. support, neutral, oppose

• Interval (most precise): exact difference in units– Common in Aggregate Data: turnout, budget, GDP,

numbers of members, deaths in war– Less common in individual level data.

• non-quantifiable (religion, region, etc.)• no agreed-upon scale (happiness, tolerance)

Levels (cont.)

• In practice, the distinction is not always observed.

• We’ll see that later on.

Note that level of measurement and

reliability are not the same thing

• Interval-level data can be unreliable and invalid (crime rates?)

Unit of analysis

• The entity we are describing Individual—we mean individual people Aggregate—any grouping of individuals• Often, a single concept can be studied at

multiple levels Example: professionalization of state legislators

Unit of analysis (cont.)

• May want to measure and explain why some individual legislators show more signs of professionalization

• May want to measure and explain why legislatures in some states are more professionalized

Unit of analysis

• Unit of analysis:– individual or aggregate?

• Ecological fallacy: inference about individuals based on

aggregate data

• E.g., concluding from aggregate data here that religious ind’s are tolerant

Individual Religious? Tolerant?

A Yes Yes

B Yes No

C No Yes

D No No

Aggregate 2 Y;2 N 2 Y;2 N

Identify the unit of analysis and level of measurement

• Gender (Individual 1: F; Individual 2: M)

• Budget (County 1: $3.2 million; County 2: $58.1 million)

• Tolerance (Individual 1: highly intolerant; Indiv 2: neutral)

• Support for Gay Marriage (Sweden: 67%; Spain: 29%)

• Electoral system (country 1: PR; country 2: Plurality)