the experimental approach september 15, 2009introduction to cognitive science lecture 3: the...

The Experimental ApproachThe Experimental Approach

September 15, 2009 Introduction to Cognitive Science Lecture 3: The Experimental Approach

Breakdown of an Experiment

• The Question

• Previous Work

• Hypothesis

• Experimental Design

• Data Analysis

• What Can One Conclude?


EXAMPLE


The Question

What issue are we trying to address?

How should information be displayed so that it can be best remembered?


Previous Work

1. Has this topic been examined before?

2. How did they try to answer the questions?– experimental design – analysis methods

3. Were there any problems/limitations?

Re-evaluate our original question


Previous Work

Let us say that our literature review on pubmed.gov reveals that the effects of contrast, color scheme, and flashing font on memorization have already been studied.

We notice that nobody has studied the effect of font size on memorization yet.

We decide to study the effect of font size on memorization.


Hypothesis

Intuitively, we think that information may be remembered better when presented in a large font than in a small font.

Information shown at a larger visual angle is “seen” by more neurons in the visual system and may thus have a better chance of being remembered.

(Notice that this is not a real study or a real argument.)

Experimental Design

• Experimental Task

• Dependent Variables

• Independent Variables

• Experimental Conditions

• Trial Sequence

• Subjects


The Experimental Task

0:000:00 0:050:05 0:150:15

4597145971 Distracting Distracting TaskTask _ _ _ _ __ _ _ _ _

0:200:20

TRIALTRIAL


Types of Variables

Dependent Variables:variables measured in the experiment:

% CORRECT

Other Examples:– Resting heart rate – Subjective rating

Independent Variables: variables manipulated by the experimenter:

FONT SIZE

Other Examples:– Hours of running/week – Types of emotion evoking

images shown


Experimental Conditions

Each level (value) of the independent variable (or combinations of levels for multiple variables) that we want to test is an experimental condition.

In our example, we only have one independent variable: font size.

Let us keep things as simple as possible and test only two different levels, i.e., two conditions: small font and large font.

We have to include trials for each condition in our experiment.


0:000:00 0:050:05 0:150:15

4597145971 Distracting Distracting TaskTask _ _ _ _ __ _ _ _ _

0:200:20

Small-font Small-font trial:trial:


Experimental Trials

0:000:00 0:050:05 0:150:15

4597459711

Distracting Distracting TaskTask _ _ _ _ __ _ _ _ _

0:200:20

Large-font Large-font trial:trial:

Trial Sequence

All measurements of dependent variables in human subjects are noisy, i.e., their values vary.

Therefore, it is important to make a large number of measurements in each experimental condition.

Depending on the task, subjects can perform comfortably for 40 to 60 minutes per session.

If necessary, multiple sessions per subject can be run.


Trial Sequence

In our example, one trial takes 20 seconds.

We decide to present 60 trials in each of the two conditions (small font vs. large font).

This gives us a total of 120 trials, i.e. 12020 = 2400 s = 40 minutes.

Should we first present the 60 small-font trials, followed by the 60 large-font trials?

Or should we use a different sequence?


Trial Sequence

No, first presenting all small-font trials, followed by all large-font trails, is not a good idea.

Subjects may improve through practice over the course of the experiment, or they may get tired.

If we found better memorization (% correct) for large font than for small font, we cannot be sure that it was actually the font that caused this difference.

It is possible that subjects were already better trained once they reached the large-font trials.


Randomization

The previous example shows a simple case of a confound – changes in the independent variable being accompanied by changes in other parameters.

Confounds prevent us from making sound conclusions about the effect of independent variables on dependent ones and thus must be avoided.

In our example, we can avoid the confound by presenting the 120 trials in random order.


Blocked Design

In some cases, we do not want to use a completely randomized trial sequence.

For example, if subjects have to perform very different tasks across conditions, switching between tasks too often may be disruptive.

In that case, we can use blocks of several trials of the same condition, and randomize the order of blocks.

If we are using large blocks, we should make sure that we counterbalance the order of blocks across subjects.


Blocked Design

In our example (even though it does not require blocking), we could form blocks of 20 trials, resulting in 3 small-font blocks and 3 large-font blocks.

We present these blocks in random order and make sure that we distribute conditions in each “time slot” evenly across subjects.


Blocked Design

Subject 1


Large-FontLarge-FontBlockBlock

Small-FontSmall-FontBlockBlock





Subject 2Large-FontLarge-Font

BlockBlockSmall-FontSmall-Font



BlockBlockLarge-FontLarge-Font


BlockBlock







BlockBlock







BlockBlock

Subjects

In order to draw conclusions about human performance in general, we should test a reasonable number of subjects.

We need more subjects if the expected difference in the dependent variable is small or if its expected variance is large.

In typical behavioral experiments like our example, about 10 to 20 subjects are appropriate.


Data Analysis

For many experimental designs, our hypothesis is confirmed if we find a significant effect, i.e., a difference in the mean value of the dependent variable across conditions in the predicted direction.

In our example, let us assume that subjects gave an average of 60% correct responses in the small-font condition and an average of 70% in the large-font condition.

Can we confirm our hypothesis, i.e., conclude that larger font leads to better memorization?


Data Analysis

Not yet!

The problem is the variance in our measure.

Due to this variance, we will get a different result every time we run an experiment, even for the same subjects.

Therefore, the difference in mean % correct between the conditions could have occurred coincidentally, without font size having any effect.


Analytical Statistics

To ensure that our results are reliable, we have to perform analytical statistics on our data (e.g., t-test, analysis of variance).

These tests will tell us the probability p that the difference in a dependent variable across two (or more) conditions is just coincidental.

In general, cognitive scientists require that p < 0.05 (probability less than 5%) in order to speak of a significant or reliable effect.


Standard Error (Bars)

For visualization of such reliability, researchers typically indicate in charts the standard error of the mean values.

The standard error is a measure of how reliable the estimate of the variable’s mean value is.

It increases with greater variance in the variable, and it decreases with more measurements being included.

Error bars in charts typically indicate standard error.


Standard Error (Bars)

Probably no significant effect


Probably significant effect

Conclusion

If the p-value is below a certain threshold, then we can conclude that our hypothesis was confirmed.

We also have to report this p-value to indicate the reliability of our result, and we have to reveal the details of our statistical analysis.

If the p-value is above the threshold, then our hypothesis was not confirmed.

Unfortunately, most negative results – unless very surprising – are never published.


Types of Experimental Designs

• Within Subject: Examine Effects within a given subject

• Between Subjects: Examine effects between groups of subjects

• Mixed Design: Examine effects within and across subjects


the experimental approach september 15, 2009introduction to cognitive science lecture 3: the...

Documents

experimental approach

cognitive science lecture

small font

large font

flashing font

smallfont trial

effect of font size

multiple variables