Defining and Measuring Defining and Measuring VariablesVariables

Chapter 3Chapter 3

Dusana RybarovaDusana RybarovaPsyc 290B Psyc 290B

May 17 2006May 17 2006

Outline:Outline:

1.1. An overview of measurementAn overview of measurement

2.2. Constructs and operational definitionsConstructs and operational definitions

3.3. Validity and reliability of Validity and reliability of measurementmeasurement

4.4. Scales of measurementScales of measurement

5.5. Modalities of MeasurementModalities of Measurement

6.6. Other Aspects of MeasurementOther Aspects of Measurement

1. An overview of 1. An overview of measurementmeasurement

• two aspects of measurement are particularly two aspects of measurement are particularly important in planning a research study or important in planning a research study or reading a research report:reading a research report:– often there is not a one-to-one relationship often there is not a one-to-one relationship

between the variable measured and the between the variable measured and the measurement obtained (knowledge, performance measurement obtained (knowledge, performance and exam grade)and exam grade)

– there are usually several different options for there are usually several different options for measuring any particular variable (types of measuring any particular variable (types of exams and questions on exams)exams and questions on exams)

– Direct measurement (height, weight) vs indirect Direct measurement (height, weight) vs indirect measurement (motivation, knowledge, memory, measurement (motivation, knowledge, memory, marital satisfaction)marital satisfaction)

2. Constructs and operational 2. Constructs and operational definitionsdefinitions

• TheoriesTheories summarize our observations, summarize our observations, explainexplain mechanisms underlying a particular behavior mechanisms underlying a particular behavior and make and make predictionspredictions about the behavior. about the behavior.

• many research variables, particularly variables many research variables, particularly variables of interest to behavioral scientists, are of interest to behavioral scientists, are hypothetical attributes or mechanisms hypothetical attributes or mechanisms explaining and predicting some behavior in a explaining and predicting some behavior in a theory are called theory are called constructsconstructs

external external stimulus stimulus constructconstruct behaviorbehaviorfactorfactor

rewardreward motivationmotivation performanceperformance

• constructs can not be directly observed or measuredconstructs can not be directly observed or measured• however, researchers can measure external, however, researchers can measure external,

observable events as an indirect method of observable events as an indirect method of measuring the construct itselfmeasuring the construct itself

• operational definitionoperational definition– is a procedure for measuring and defining a construct, is a procedure for measuring and defining a construct,

indirect method of measuring something that can not be indirect method of measuring something that can not be measured directlymeasured directly

– an operational definition specifies a measurement procedure an operational definition specifies a measurement procedure for measuring an external, observable behavior and uses the for measuring an external, observable behavior and uses the resulting measurements as a definition and a measurement resulting measurements as a definition and a measurement of the hypothetical constructof the hypothetical construct

– e.g. IQ test is an operational definition for the construct e.g. IQ test is an operational definition for the construct intelligenceintelligence

– - - provide and example of a theoretical construct and its provide and example of a theoretical construct and its operational definitionoperational definition

– You don’t always have to come up with your own You don’t always have to come up with your own operational definition of the construct, you can use operational definition of the construct, you can use some conventional measurement procedure from some conventional measurement procedure from previous studiesprevious studies

3. Validity and reliability of 3. Validity and reliability of measurementmeasurement

• How do you decide which method of How do you decide which method of measurement (operational definition measurement (operational definition of a construct) is the best?of a construct) is the best?

• there are two general criteria for there are two general criteria for evaluating the quality of any evaluating the quality of any measurement proceduremeasurement procedure– validity validity – reliabilityreliability

Validity of measurementValidity of measurement

• Validity of measurementValidity of measurement– concerns the “truth” of the concerns the “truth” of the

measurementmeasurement– it is the degree to which the it is the degree to which the

measurement process measures the measurement process measures the variable it claims to measurevariable it claims to measure

– Is the IQ score truly measuring Is the IQ score truly measuring intelligence? What about size of the intelligence? What about size of the brain and bumps on the scull?brain and bumps on the scull?

Different kinds of validityDifferent kinds of validity– face validityface validity

• the simplest and least scientific definition of validitythe simplest and least scientific definition of validity

• it is demonstrated when a measure superficially it is demonstrated when a measure superficially appears to measure what it claims to measureappears to measure what it claims to measure

• Based on subjective judgment and difficult to quantifyBased on subjective judgment and difficult to quantify

• e.g. intelligence and reasoning questions on the IQ e.g. intelligence and reasoning questions on the IQ testtest

• Problem - participants can use the face validity to Problem - participants can use the face validity to change their answerschange their answers

– concurrent validity (criterion validity)concurrent validity (criterion validity)• is demonstrated when scores obtained from a new is demonstrated when scores obtained from a new

measure are directly related to scores obtained from measure are directly related to scores obtained from a more established measure of the same variablea more established measure of the same variable

• e.g. new IQ test correlates with an older IQ teste.g. new IQ test correlates with an older IQ test

Different kinds of validity Different kinds of validity (cont.)(cont.)

• Different kinds of validityDifferent kinds of validity– predictive validitypredictive validity

• when scores obtained from a measure accurately when scores obtained from a measure accurately predict behavior according to a theorypredict behavior according to a theory

• e.g. high scores on need for achievement test predict e.g. high scores on need for achievement test predict competitive behavior in children (ring toss game)competitive behavior in children (ring toss game)

– construct validityconstruct validity• is demonstrated when scores obtained from a is demonstrated when scores obtained from a

measure are directly related to the variable itselfmeasure are directly related to the variable itself• Reflects how close the measure relates to the Reflects how close the measure relates to the

construct (height and weight example)construct (height and weight example)• in one sense, construct validity is achieved by in one sense, construct validity is achieved by

repeatedly demonstrating every other type of validityrepeatedly demonstrating every other type of validity

Different kinds of validity Different kinds of validity (cont.)(cont.)• Different kinds of validityDifferent kinds of validity

– convergent validityconvergent validity• is demonstrated by a strong relationship between the is demonstrated by a strong relationship between the

scores obtained from two different methods of measuring scores obtained from two different methods of measuring the same constructthe same construct

• e.g. an experimenter observing aggressive behavior in e.g. an experimenter observing aggressive behavior in children correlated with teacher’s ratings of their children correlated with teacher’s ratings of their behaviorbehavior

– divergent validitydivergent validity• is demonstrated by using two different methods to is demonstrated by using two different methods to

measure two different constructsmeasure two different constructs• convergent validity must be shown for each of the two convergent validity must be shown for each of the two

constructs and little or no relationship exists between the constructs and little or no relationship exists between the scores obtained from the two different constructs when scores obtained from the two different constructs when they are measured by the same methodthey are measured by the same method

• e.g. aggressive behavior and general activity level in e.g. aggressive behavior and general activity level in childrenchildren

Convergent validity, divergent Convergent validity, divergent validity and construct validityvalidity and construct validity

• By demonstrating strong convergent validity for two By demonstrating strong convergent validity for two different constructs and then showing divergent different constructs and then showing divergent validity between the two constructs, you obtain validity between the two constructs, you obtain strong construct validity of the two constructsstrong construct validity of the two constructs

Aggressive Aggressive behaviorbehavior

Teacher’s Teacher’s ratingsratings

Experimenter’s Experimenter’s observationobservation

Active Active behaviorbehavior

Teacher’s Teacher’s ratingsratings

Experimenter’s Experimenter’s observationobservation

High convergent validity

Related scores

High Divergent Vali

dityUnrelated

scores

High convergent validity

Related scores

High Divergent Vali

dityUnrelated

scores

Reliability of measurementReliability of measurement• Reliability of measurementReliability of measurement

– a measurement procedure is said to be a measurement procedure is said to be reliable if repeated measurements of the same reliable if repeated measurements of the same individual under the same conditions produce individual under the same conditions produce identical (or nearly identical) valuesidentical (or nearly identical) values

– reliability is the stability or the consistency of reliability is the stability or the consistency of measurementmeasurement

measured score = true score + errormeasured score = true score + error

IQ score = true IQ score + mood, fatigue etc.IQ score = true IQ score + mood, fatigue etc.

Reliability and error of Reliability and error of measurementmeasurement

• Inconsistency (lack of reliability) of measurement Inconsistency (lack of reliability) of measurement comes from errorcomes from error

• The higher the error the more unreliable the The higher the error the more unreliable the measurementmeasurement

• Sources of error Sources of error • observer errorobserver error

– the individual who makes the measurements can introduce the individual who makes the measurements can introduce simple human error into the measurement processsimple human error into the measurement process

• environmental changesenvironmental changes– small changes in the environment from one measurement to small changes in the environment from one measurement to

another (e.g. time of the day, distraction in the room, lighting)another (e.g. time of the day, distraction in the room, lighting)• participant changesparticipant changes

– participants change between measurements (mood, hunger, participants change between measurements (mood, hunger, motivation)motivation)

Types and measures of Types and measures of reliabilityreliability

• successive measurementssuccessive measurements• Obtaining scores from two successive measurements and Obtaining scores from two successive measurements and

calculating a correlation between themcalculating a correlation between them• the same group, the same measurement at two different the same group, the same measurement at two different

timestimes• test-retest reliabilitytest-retest reliability

• simultaneous measurementssimultaneous measurements• obtained by direct observation of behaviors (two or more obtained by direct observation of behaviors (two or more

separate observers at the same time), consistency across separate observers at the same time), consistency across ratersraters

• inter-rater reliabilityinter-rater reliability

• internal consistencyinternal consistency• degree of consistency of scores from separate items on a test degree of consistency of scores from separate items on a test

or questionnaire consisting of multiple itemsor questionnaire consisting of multiple items• you want all the items or groups of items tapping the same you want all the items or groups of items tapping the same

processesprocesses• researchers commonly split the set of items in half, compute researchers commonly split the set of items in half, compute

a separate score of each half, and then evaluate the degree a separate score of each half, and then evaluate the degree of agreement between the two scoresof agreement between the two scores

• split-half reliabilitysplit-half reliability

The relationship between The relationship between reliability and validityreliability and validity

– they are partially related and partially they are partially related and partially independent independent

– reliability is a prerequisite for validity reliability is a prerequisite for validity (measurement procedure can not be (measurement procedure can not be valid unless it is reliable – e.g. IQ, huge valid unless it is reliable – e.g. IQ, huge variance of repeated measurements is variance of repeated measurements is impossible if we are truly measuring impossible if we are truly measuring intelligence)intelligence)

– it is not necessary for a measurement to it is not necessary for a measurement to be valid for it to be reliable (e.g. height be valid for it to be reliable (e.g. height as a measure of intelligence)as a measure of intelligence)

4. Scales of measurement4. Scales of measurement• Scales define the type categories we use in Scales define the type categories we use in

measurement and the selection of a scale has direct measurement and the selection of a scale has direct impact on our ability to describe relationships between impact on our ability to describe relationships between variablesvariables

• the nominal scalethe nominal scale– simply represents qualitative difference in the variable simply represents qualitative difference in the variable

measuredmeasured– can onlycan only tell us thattell us that a difference exists without the possibility a difference exists without the possibility

telling the direction or magnitude of the differencetelling the direction or magnitude of the difference– e.g. majors in college, race, gender, occupatione.g. majors in college, race, gender, occupation

• the ordinal scalethe ordinal scale– the categories that make up an ordinal scale form an ordered the categories that make up an ordinal scale form an ordered

sequencesequence– can can tell ustell us the direction of the difference but not the the direction of the difference but not the

magnitudemagnitude– e.g. coffee cup sizes, socioeconomic class, T-shirt sizes, food e.g. coffee cup sizes, socioeconomic class, T-shirt sizes, food

preferencespreferences

Scales of measurement Scales of measurement (cont.)(cont.)

• the interval scalethe interval scale– categories on an interval scale are organized categories on an interval scale are organized

sequentially, and all categories are the same sequentially, and all categories are the same sizesize

– we we can determine the direction and the can determine the direction and the magnitude of a differencemagnitude of a difference

– May have an arbitrary zero (convenient point of May have an arbitrary zero (convenient point of reference) reference)

– e.g. temperature in Farenheit, time in secondse.g. temperature in Farenheit, time in seconds• the ratio scalethe ratio scale

– consists of equal, ordered categories anchored consists of equal, ordered categories anchored by a zero point that is not arbitrary but by a zero point that is not arbitrary but meaningful (representing absence of a variablemeaningful (representing absence of a variable

– allows us to determine the direction, the allows us to determine the direction, the magnitude, and the ratio of the differencemagnitude, and the ratio of the difference

– e.g. reaction time, number of errors on a teste.g. reaction time, number of errors on a test

5. Modalities of 5. Modalities of measurementmeasurement

• One can measure a construct by One can measure a construct by selecting a measure from three main selecting a measure from three main categoriescategories

• There are three basic modalities of There are three basic modalities of measurement:measurement:– self-reportself-report– physiological measurementphysiological measurement– behavioral measurementbehavioral measurement

•behavioral observationbehavioral observation

•content analysis and archival researchcontent analysis and archival research

Self-report measuresSelf-report measures– you ask a participant to describe his you ask a participant to describe his

behavior, to express his opinion or behavior, to express his opinion or characterize his experience in an interview characterize his experience in an interview or by using a questionnaire with ratingsor by using a questionnaire with ratings

– Positive aspectsPositive aspects•Only the individual has direct access to Only the individual has direct access to

information about his state of mindinformation about his state of mind•More direct measureMore direct measure

– Negative aspectsNegative aspects•Participants may distort the responses to create Participants may distort the responses to create

a better self-image or to please the a better self-image or to please the experimenterexperimenter

•The response can also be influenced by The response can also be influenced by wording of the questions and other aspects of wording of the questions and other aspects of the situationthe situation

Physiological measuresPhysiological measures– Physiological manifestations of the underlying Physiological manifestations of the underlying

constructconstruct– e.g. EEG, EKG, galvanic skin response, perspiration, e.g. EEG, EKG, galvanic skin response, perspiration,

PET, fMRIPET, fMRI– advantages advantages

•provides accurate, reliable, and well-defined provides accurate, reliable, and well-defined measurements that are not dependent on subjective measurements that are not dependent on subjective interpretation interpretation

– disadvantagesdisadvantages•equipment is usually expensive or unavailableequipment is usually expensive or unavailable•Presence of monitoring devices may create unnatural Presence of monitoring devices may create unnatural

situationsituation•question: Are these procedures a valid measure of the question: Are these procedures a valid measure of the

construct (e.g. increase in heart rate to fear, arousal)construct (e.g. increase in heart rate to fear, arousal)

Behavioral measuresBehavioral measures

– behaviors that can be observed and measured behaviors that can be observed and measured (e.g. reaction time, reading speed, focus of (e.g. reaction time, reading speed, focus of attention, disruptive behavior, number of attention, disruptive behavior, number of words recalled on a memory test)words recalled on a memory test)

– How to select the right behavioral measure?How to select the right behavioral measure?• Depends on the purpose of the studyDepends on the purpose of the study

– In clinical setting the same disorder can reveal itself In clinical setting the same disorder can reveal itself through different symptoms through different symptoms

– In studying memory we want to have the same In studying memory we want to have the same measure for all subjects to be able to compare themmeasure for all subjects to be able to compare them

– Beware of situational changes in behavior (e.g. Beware of situational changes in behavior (e.g. disruptive behavior in school vs when disruptive behavior in school vs when observed) and different behavioral indicators of observed) and different behavioral indicators of a constructa construct

6. Other aspects of 6. Other aspects of measurementmeasurement

• multiple measuresmultiple measures– sometimes you can use two (or more) different sometimes you can use two (or more) different

procedures to measure the same variable (e.g. procedures to measure the same variable (e.g. heart rate and questionnaire as a measure of fear)heart rate and questionnaire as a measure of fear)

– problems (the two variables may not behave in the problems (the two variables may not behave in the same way)same way)• e.g. a specific therapy for treating fear may have large e.g. a specific therapy for treating fear may have large

effect on behavior but no effect on heart rateeffect on behavior but no effect on heart rate

– the lack of agreement between two measures is the lack of agreement between two measures is called desynchronycalled desynchrony• One measure can be more sensitive than otherOne measure can be more sensitive than other

• Different measures may indicate different dimensions of Different measures may indicate different dimensions of the variable and change at different times during the the variable and change at different times during the treatmenttreatment

Sensitivity and range effectsSensitivity and range effects

– are the measures sensitive enough to respond are the measures sensitive enough to respond to the type and magnitude of the changes that to the type and magnitude of the changes that are expected? (e.g. seconds vs. milliseconds, are expected? (e.g. seconds vs. milliseconds, difficult or easy exams)difficult or easy exams)

– range effectsrange effects• a ceiling effect (the clustering of scores at the high a ceiling effect (the clustering of scores at the high

end of a measurement scale, allowing little or no end of a measurement scale, allowing little or no possibility of increases in value, e.g. test that is too possibility of increases in value, e.g. test that is too easy)easy)

• a floor effect (the clustering of scores at the low end a floor effect (the clustering of scores at the low end of a measurement scale, allowing little or no of a measurement scale, allowing little or no possibility of decreases in value, e.g. test that is too possibility of decreases in value, e.g. test that is too difficult)difficult)

• Range effects are usually a consequence of using a Range effects are usually a consequence of using a measure that is inappropriate for a particular group measure that is inappropriate for a particular group (e.g. 4-grade test for college students)(e.g. 4-grade test for college students)

Participant reactivity and Participant reactivity and experimenter biasexperimenter bias

– participant reactivity is the way how participant participant reactivity is the way how participant reacts to the experimental situation (e.g. overly reacts to the experimental situation (e.g. overly cooperative, overly defensive, or hostile)cooperative, overly defensive, or hostile)• To avoid these problems one can try to disguise the true To avoid these problems one can try to disguise the true

purpose of the experiment or observe individuals without purpose of the experiment or observe individuals without their awareness (beware ethical issues)their awareness (beware ethical issues)

– experimenter bias is the way experimenter experimenter bias is the way experimenter influences results (e.g. by being warm and friendly influences results (e.g. by being warm and friendly with one group of participants vs. cold and stern with with one group of participants vs. cold and stern with other group)other group)

– to avoid participant reactivity and experimenter bias to avoid participant reactivity and experimenter bias we use:we use:• standardized procedures (e.g. instructions recorded on a standardized procedures (e.g. instructions recorded on a

tape)tape)• a research study is a research study is single blindsingle blind if the researcher does not if the researcher does not

know the predicted outcomeknow the predicted outcome• a research study is a research study is double blinddouble blind if both the researcher and if both the researcher and

the participants are unaware of the predicted outcomethe participants are unaware of the predicted outcome

Participant reactivity and Participant reactivity and experimenter biasexperimenter bias

– to avoid participant reactivity and to avoid participant reactivity and experimenter bias we use ‘blind’ experimenter bias we use ‘blind’ experimentsexperiments•a research study is a research study is single blindsingle blind if the if the

researcher does not know the predicted researcher does not know the predicted outcomeoutcome

•a research study is a research study is double blinddouble blind if both the if both the researcher and the participants are unaware researcher and the participants are unaware of the predicted outcomeof the predicted outcome