hawthorne effect

28
Hawthorne Effect Hawthorne Effect When your experimental effect is due to the experiment itself: the subject is at the center of attention. Can manifest itself as a spurt or elevation in performance or physical phenomenon measured. More of a problem when it operates differently in different cells of the experiment. Solution: Add a control group to the experiment. Have them go through the same experimental procedure, but administer a placebo instead of the treatment. Example: Testing a new design tool. Bring in two groups into the lab, tell them both you have an exciting new tool. Use your real tool with one group, use the old tool with the placebo group.

Upload: creda

Post on 21-Mar-2016

115 views

Category:

Documents


2 download

DESCRIPTION

Hawthorne Effect. When your experimental effect is due to the experiment itself: the subject is at the center of attention. Can manifest itself as a spurt or elevation in performance or physical phenomenon measured. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Hawthorne Effect

Hawthorne EffectHawthorne Effect When your experimental effect is due to the experiment

itself: the subject is at the center of attention. Can manifest itself as a spurt or elevation in performance or

physical phenomenon measured. More of a problem when it operates differently in different

cells of the experiment. Solution: Add a control group to the experiment. Have them

go through the same experimental procedure, but administer a placebo instead of the treatment.

Example: Testing a new design tool. Bring in two groups into the lab, tell them both you have an exciting new tool. Use your real tool with one group, use the old tool with the placebo group.

Page 2: Hawthorne Effect

Blind and Double Blind Blind and Double Blind ProceduresProcedures

• Medical TerminologyMedical Terminology• Blind AdministrationBlind Administration: When the subjects : When the subjects

does not know if he/she is in the does not know if he/she is in the experimental / control conditionexperimental / control condition

• Double Blind AdministrationDouble Blind Administration: When the : When the above is true, and also the experimenter above is true, and also the experimenter does not know which condition the subject does not know which condition the subject is in (Controls for expectancy effects)is in (Controls for expectancy effects)

Page 3: Hawthorne Effect

Experimental terminology Experimental terminology in Multifactor experimentsin Multifactor experiments

• Factors / Independent Variable / Treatment Factors / Independent Variable / Treatment Condition:Condition: Is directly manipulated in real experiments, is selected in Is directly manipulated in real experiments, is selected in

quasi experiments.quasi experiments.• Levels of the IV: Each specific variation of the Levels of the IV: Each specific variation of the

factor. E.g. the different font sizesfactor. E.g. the different font sizes• Main Effect: The difference in the DV between the Main Effect: The difference in the DV between the

different levels of the IVdifferent levels of the IV• Interaction: Does one independent variable effect Interaction: Does one independent variable effect

the other. Do they interact? the other. Do they interact?

Page 4: Hawthorne Effect

Effect of Font Size and Effect of Font Size and Screen Resolution on Screen Resolution on

ReadabilityReadability• Main Effect-SizeMain Effect-Size• Main Effect-Main Effect-

ResolutionResolution• No interactionNo interaction

0

1

2

3

4

5

6

7

8

9

Size 12 Size 16

Low Resolution High Resolution

Size 12 Size 16 Row MeansLow Resolution 3 5 4High Resolution 5 7 6Column Means 4 6 5

Page 5: Hawthorne Effect

Effect of Font Size and Effect of Font Size and Screen Resolution on Screen Resolution on

ReadabilityReadability• Main Effect-SizeMain Effect-Size• No Main Effect-No Main Effect-

ResolutionResolution• No interactionNo interaction

0

1

2

3

4

5

6

7

Size 12 Size 16

Low Resolution High Resolution

Size 12 Size 16 Row MeansLow Resolution 3 6 4.5High Resolution 3.2 6.2 4.7Column Means 3.1 6.1 4.6

Page 6: Hawthorne Effect

Effect of Font Size and Effect of Font Size and Screen Resolution on Screen Resolution on

ReadabilityReadability• Main Effect-SizeMain Effect-Size• Main Effect-ResolutionMain Effect-Resolution• InteractionInteraction

High sizes at High resolution High sizes at High resolution have great readabilityhave great readability

0

2

4

6

8

10

12

Size 12 Size 16

Low Resolution High Resolution

Size 12 Size 16 Row MeansLow Resolution 3 5 4.0High Resolution 4 10 7.0Column Means 3.5 7.5 5.5

Page 7: Hawthorne Effect

Effect of Font Size and Effect of Font Size and Screen Resolution on Screen Resolution on

ReadabilityReadability• Main Effect-SizeMain Effect-Size• Main Effect-Main Effect-

ResolutionResolution• InteractionInteraction

0

2

4

6

8

10

12

Size 12 Size 16

Low Resolution High Resolution

Size 12 Size 16 Row MeansLow Resolution 1 9 4.8High Resolution 5 10 7.5Column Means 2.8 9.5 6.1

Page 8: Hawthorne Effect

• Main Effects: When we look at a main effect Main Effects: When we look at a main effect (effect of one variable averaged over the (effect of one variable averaged over the other), we are ignoring the other variableother), we are ignoring the other variable

• Interaction: concerned with the joint effects Interaction: concerned with the joint effects of both the variablesof both the variables When lines are parallel, interaction not present. In When lines are parallel, interaction not present. In

case of interaction, lines will cross theoretically at case of interaction, lines will cross theoretically at some pointsome point

Independent Variables can be depicted on either Independent Variables can be depicted on either axisaxis

Page 9: Hawthorne Effect

Establishing a Cause-Effect Establishing a Cause-Effect RelationshipRelationship

Temporal Precedence Temporal Precedence • Cause happened before your effect. Cause happened before your effect.

Real life relationships between Real life relationships between variables are never simple. variables are never simple.

Cyclical situations, involving ongoing Cyclical situations, involving ongoing processes that interact are hard to processes that interact are hard to interpret.interpret.

Page 10: Hawthorne Effect

Covariation of the Cause and Covariation of the Cause and Effect Effect

if X then Yif X then Yif not X then not Yif not X then not Y

• If you observe that whenever X is present, Y is If you observe that whenever X is present, Y is also present, and whenever X is absent, Y is also present, and whenever X is absent, Y is too, then there is covariation between the two.too, then there is covariation between the two.

• For Example:For Example:Better website, more visitorsBetter website, more visitorsBad website, less visitorsBad website, less visitors

Page 11: Hawthorne Effect

No Plausible Alternative No Plausible Alternative ExplanationsExplanations

• Covariation does not imply causation. Covariation does not imply causation. • Rule out Rule out alternative explanationsalternative explanations. (a third . (a third

variable that might be causing the outcome)variable that might be causing the outcome)• Referred to as the "Referred to as the "third variablethird variable" or " or

""missing variablemissing variable" problem. Also at the heart " problem. Also at the heart of establishing Internal validity. of establishing Internal validity.

• For Example: Better better site (better For Example: Better better site (better company, more marketing) more visitorscompany, more marketing) more visitors

Page 12: Hawthorne Effect

Hypothetical Case Study:Hypothetical Case Study:Barnes and Noble site Barnes and Noble site

redesignredesign•Hired one of the famous “ient” web design companies to redesign site

•Purpose: make online shopping easy and site more attractive

•Paid a lot of money

•Does site redesign work: Lets look at sales figures

Page 13: Hawthorne Effect

Hypothetical DataHypothetical Data

• Sales increased!Sales increased!

Effect of Site Redesign on Online Sales

0

1

2

3

4

5

6

7

8

9

Old Site New Site

Sal

es

Page 14: Hawthorne Effect

Problems with Deducing Problems with Deducing that site redesign workedthat site redesign worked

• Temporal relationshipTemporal relationship• CovariationCovariation• Alternative Explanations:Alternative Explanations:

Page 15: Hawthorne Effect

ReliabilityReliability• ReplicabilityReplicability• Insure that random confounding Insure that random confounding

factors are not playing a rolefactors are not playing a role

Page 16: Hawthorne Effect

External ValidityExternal Validity• Related to generalizing. Degree to which Related to generalizing. Degree to which

the conclusions in your study would hold for the conclusions in your study would hold for other persons in other places and at other other persons in other places and at other times.times.

• Sampling ModelSampling Model: Identify the population : Identify the population you would like to generalize to. Then, you you would like to generalize to. Then, you draw random sample from that population. draw random sample from that population. You can generalize back to it.You can generalize back to it. Problems: Time and place constraintsProblems: Time and place constraints

Page 17: Hawthorne Effect

Threats to External Threats to External ValidityValidity

• PeoplesPeoples: Results of your study could unusual type of : Results of your study could unusual type of people who were in the study. people who were in the study.

• PlacesPlaces: Limited to experimental context.: Limited to experimental context. For example: if you conducted study in an office For example: if you conducted study in an office

atmosphere. atmosphere. • TimeTime: Limited to time period when you did your : Limited to time period when you did your

experiment.experiment. For example: study on web interfaces in 1997For example: study on web interfaces in 1997

• ObjectsObjects: In HCI your results might be extendable to : In HCI your results might be extendable to only similar objects / interfaces.only similar objects / interfaces.

Page 18: Hawthorne Effect

What is validityWhat is validity• Validity refers to the operationalization Validity refers to the operationalization

or measurement of concepts. or measurement of concepts. • Any time you translate a concept or Any time you translate a concept or

construct into a functioning and construct into a functioning and operating reality (operating reality (the the operationalizationoperationalization), you need to be ), you need to be concerned about how well you did the concerned about how well you did the translation. translation.

Page 19: Hawthorne Effect

Internal ValidityInternal ValidityConcerns inferences regarding cause-effect or causal relationships.

•Only relevant in studies that try to establish a causal relationship. •Not relevant in most observational or descriptive studies.

Important for studies that assess the effects of certain changes to websites, or to products.

Page 20: Hawthorne Effect

Are there alternative Are there alternative explanations?explanations?

• Example: Amazon.com increased the number of tabs in its home page.

• Assume that study showed increase in the no of tabs = increase in ease of

navigation.

Alternative explanations: • At same time Amazon.com launched a marketing

campaign.• The key question in internal validity is whether observed

changes can be attributed to your intervention (i.e., the cause) and not to other possible causes (sometimes described as "alternative explanations" for the outcome).

Page 21: Hawthorne Effect

Construct ValidityConstruct Validity• Degree to which you can generalize back to the Degree to which you can generalize back to the

theoretical construct you started from.theoretical construct you started from.• Construct validity can be thought of as a Construct validity can be thought of as a

"labeling" issue. "labeling" issue. • Real Objective: to make site easier to navigateReal Objective: to make site easier to navigate

Operationalization: give users more options on each Operationalization: give users more options on each page by increasing number of links.page by increasing number of links.

Is increasing number of links really giving users more Is increasing number of links really giving users more options.options.

Page 22: Hawthorne Effect

Kinds of construct validityKinds of construct validity• Face ValidityFace Validity• Content ValidityContent Validity

Page 23: Hawthorne Effect

Face ValidityFace Validity• Does operationalization of the concept seem Does operationalization of the concept seem

like a good translation “on its face" or like a good translation “on its face" or superficially speaking.superficially speaking.

• The weakest way to try to demonstrate The weakest way to try to demonstrate construct validity. construct validity.

• For example: you can check for a measure of For example: you can check for a measure of math ability, read through the questions, and math ability, read through the questions, and decide that, it seems like this is a good measure decide that, it seems like this is a good measure of math ability (i.e., the label "math ability" of math ability (i.e., the label "math ability" seems appropriate for this measure). seems appropriate for this measure).

Page 24: Hawthorne Effect

Content ValidityContent Validity• Check the operationalization against the Check the operationalization against the

relevant content domain for the construct.relevant content domain for the construct.• For example: you are trying to measure For example: you are trying to measure

usability. What are the sub domains of usabilityusability. What are the sub domains of usability EfficiencyEfficiency Attractiveness Attractiveness ControlControl

• Check your measure of usability against these Check your measure of usability against these domainsdomains

Page 25: Hawthorne Effect

Research DesignsResearch Designs

Single Group Experimental DesignsSingle Group Experimental Designs Repeated measurements are take across time for Repeated measurements are take across time for

one group.one group. Does not lend itself to clear statistical analysis and Does not lend itself to clear statistical analysis and

hypothesis testinghypothesis testing Cannot control for order effects, difficult to Cannot control for order effects, difficult to

generalizegeneralize Can provide us with important information which we Can provide us with important information which we

might not have access to by experimentsmight not have access to by experiments

Page 26: Hawthorne Effect

Randomized Group Randomized Group Experimental DesignsExperimental Designs

• This is what you want to aim forThis is what you want to aim for• You have an experimental and You have an experimental and

control group. Randomly assign control group. Randomly assign subjects to either groupsubjects to either group

• All sorts of causal inferences possible All sorts of causal inferences possible

Page 27: Hawthorne Effect

Quasi Experimental Quasi Experimental DesignDesign

• When you cannot control who gets assigned to When you cannot control who gets assigned to which groupwhich group

• For example: in an ex post facto study, IV has For example: in an ex post facto study, IV has already occurred, you want to draw inferences.already occurred, you want to draw inferences.

• For example: You want to compare users of For example: You want to compare users of Palm Pilot and Handspring. You have no Palm Pilot and Handspring. You have no control over who goes to which groupcontrol over who goes to which group

Page 28: Hawthorne Effect

Comparing Quasi-Comparing Quasi-Experimental and Experimental and

Experimental designsExperimental designs• The experimental design is as sound in The experimental design is as sound in

both casesboth cases• It is harder to make causal inferences in It is harder to make causal inferences in

case of quasi experimental designs, since case of quasi experimental designs, since groups were not equal to start withgroups were not equal to start with

• You can do pretest on groups, and do You can do pretest on groups, and do analysis of covarianceanalysis of covariance