descriptive and experimental research by gurpreet

8/4/2019 Descriptive and Experimental Research by Gurpreet

1/21

In quantitative research your aim is to determine the relationshipbetween one thing (an independent variable) and another (a dependent oroutcome variable) in a population. Quantitative research designs are eitherdescriptive (subjects usually measured once) or experimental (subjectsmeasured before and after a treatment). A descriptive study establishesonly associations between variables. An experiment establishes causality.

For an accurate estimate of the relationship between variables, adescriptive study usually needs a sample of hundreds or even thousandsof subjects; an experiment, especially a crossover, may need only tens ofsubjects. The estimate of the relationship is less likely to be biased if youhave a high participation rate in a sample selected randomly from apopulation. In experiments, bias is also less likely if subjects are randomlyassigned to treatments, and if subjects and researchers are blind to theidentity of the treatments.

In all studies, subject characteristics can affect the relationship you areinvestigating. Limit their effect either by using a less heterogeneoussample of subjects or preferably by measuring the characteristics andincluding them in the analysis. In an experiment, try to measure variablesthat might explain the mechanism of the treatment. In an unblindedexperiment, such variables can help define the magnitude of any placeboeffect.

Quantitative research is all about quantifying relationships between variables.

Variables are things like weight, performance, time, and treatment. You

measure variables on a sample of subjects, which can be tissues, cells,animals, or humans. You express the relationship between variable using

effect statistics, such as correlations, relative frequencies, or differences

between means. I deal with these statistics and other aspects of

analysiselsewhere at this site. In this article I focus on the design of

quantitative research. First I describe the types of study you can use. Next Idiscuss how the nature of the sample affects your ability to make statements

about the relationship in the population. I then deal with various ways towork out the size of the sample. Finally I give advice about the kinds of

variable you need to measure.

TYPES OF STUDY

Studies aimed at quantifying relationships are of twotypes: descriptive andexperimental (Table 1). In a descriptive study, no

attempt is made to change behavior or conditions--you measure things as

they are. In an experimental study you take measurements, try some sort ofintervention, then take measurements again to see what happened.

Table 1: Types of research design

Descriptive or observational

case case series
http://www.sportsci.org/resource/stats/index.htmlhttp://www.sportsci.org/resource/stats/index.htmlhttp://www.sportsci.org/resource/stats/index.htmlhttp://www.sportsci.org/resource/stats/index.html


2/21

cross-sectional cohort or prospective or longitudinal case-control or retrospective

Experimental or longitudinal or repeated-measures

without a control group

time seriescrossover with a control group

Descriptive Studies

Descriptive studies are also called observational, because you observe the

subjects without otherwise intervening. The simplest descriptive study is

a case, which reports data on only one subject; examples are a study of an

outstanding athlete or of a dysfunctional institution. Descriptive studies of afew cases are calledcase series. In cross-sectional studies variables of

interest in a sample of subjects are assayed once and the relationshipsbetween them are determined. Inprospective or cohort studies, somevariables are assayed at the start of a study (e.g., dietary habits), then after a

period of time the outcomes are determined (e.g., incidence of heart disease).

Another label for this kind of study is longitudinal, although this term alsoapplies to experiments. Case-control studies comparecases (subjects with a

particular attribute, such as an injury or ability) withcontrols (subjects

without the attribute); comparison is made of the exposure to something

suspected of causing the cases, for example volume of high intensity training,or number of alcoholic drinks consumed per day. Case-control studies are

also called retrospective, because they focus on conditions in the past that

might have caused subjects to become cases rather than controls.

A common case-control design in the exercise science literature is acomparison of the behavioral, psychological or anthropometric characteristicsof elite and sub-elite athletes: you are interested in what the elite athletes

have been exposed to that makes them better than the sub-elites. Another

type of study compares athletes with sedentary people on some outcome such

as an injury, disease, or disease risk factor. Here you know the difference inexposure (training vs no training), so these studies are really cohort or

prospective, even though the exposure data are gathered retrospectively at

only one time point. The technical name for these studies

is historicalcohort.

Experimental Studies

Experimental studies are also known as longitudinal or repeated-measuresstudies, for obvious reasons. They are also referred toas interventions, because you do more than just observe the subjects.


3/21

In the simplest experiment, a time series, one or more measurements are

taken on all subjects before and after a treatment. A special case of the timeseries is the so-called single-subject design, in which measurements are

taken repeatedly (e.g., 10 times) before and after an intervention on one or a

few subjects.

Time series suffer from a major problem: any change you see could be due to

something other than the treatment. For example, subjects might do better onthe second test because of their experience of the first test, or they might

change their diet between tests because of a change in weather, and diet could

affect their performance of the test. The crossover design is one solution tothis problem. Normally the subjects are given two treatments, one being the

real treatment, the other a control or reference treatment. Half the subjects

receive the real treatment first, the other half the control first. After a period

of time sufficient to allow any treatment effect to wash out, the treatments arecrossed over. Any effect of retesting or of anything that happened between

the tests can then be subtracted out by an appropriate analysis. Multiplecrossover designs involving several treatments are also possible.

If the treatment effect is unlikely to wash out between measurements,

a control group has to be used. In these designs, all subjects are measured,but only some of them--the experimental group--then receive the treatment.

All subjects are then measured again, and the change in the experimental

group is compared with the change in the control group.

If the subjects are assigned randomly to experimental and control groups or

treatments, the design is known as a randomized controlled trial. Random

assignment minimizes the chance that either group is not typical of thepopulation. If the subjects are blind (or masked) to the identity of the

treatment, the design is asingle-blind controlled trial. The control or

reference treatment in such a study is called a placebo: the name physiciansuse for inactive pills or treatments that are given to patients in the guise of

effective treatments. Blinding of subjects eliminates the placebo effect,

whereby people react differently to a treatment if they think it is in some wayspecial. In a double-blind study, the experimenter also does not know which

treatment the subjects receive until all measurements are taken. Blinding of

the experimenter is important to stop him or her treating subjects in one

group differently from those in another. In the best studies even the data areanalyzed blind, to prevent conscious or unconscious fudging or prejudiced

interpretation.

Ethical considerations or lack of cooperation (compliance) by the subjects

sometimes prevent experiments from being performed. For example, a

randomized controlled trial of the effects of physical activity on heart diseasemay not have been performed yet, because it is unethical and unrealistic to

randomize people to 10 years of exercise or sloth. But there have been many


4/21

short-term studies of the effects of physical activity on disease risk factors

(e.g., blood pressure).

Quality of Designs

The various designs differ in the quality of evidence they provide for a cause-and-effect relationship between variables. Cases and case series are theweakest. A well-designed cross-sectional or case-control study can provide

good evidence for the absence of a relationship. But if such a study does

reveal a relationship, it generally represents only suggestive evidence of a

causal connection. A cross-sectional or case-control study is therefore a goodstarting point to decide whether it is worth proceeding to better designs.

Prospective studies are more difficult and time-consuming to perform, but

they produce more convincing conclusions about cause and effect.

Experimental studies provide the best evidence about how something affectssomething else, and double-blind randomized controlled trials are the best

experiments.

Confounding is a potential problem in descriptive studies that try to establish

cause and effect. Confounding occurs when part or all of a significant

association between two variables arises through both being causallyassociated with a third variable. For example, in a population study you could

easily show a negative association between habitual activity and most forms

of degenerative disease. But older people are less active, and older people aremore diseased, so you're bound to find an association between activity and

disease without one necessarily causing the other. To get over this problem

you have to control for potential confounding factors. For example, you

make sure all your subjects are the same age, or you include age in theanalysis to try to remove its effect on the relationship between the other two

variables.

SAMPLES

You almost always have to work with a sample of subjects rather than the

fullpopulation. But people are interested in the population, not your sample.

Togeneralize from the sample to the population, the sample has toberepresentative of the population. The safest way to ensure that it is

representative is to use a random selection procedure. You can also use

a stratified random sampling procedure, to make sure that you have

proportional representation of population subgroups (e.g., sexes, races,regions).

When the sample is not representative of the population, selection bias is apossibility. A statistic is biased if the value of the statistic tends to be wrong

(or more precisely, if the expected value--the average value from many

samples drawn using the same sampling method--is not the same as the


5/21

population value.) A typical source of bias in population studies is age or

socioeconomic status: people with extreme values for these variables tend notto take part in the studies. Thus a high compliance (the proportion of people

contacted who end up as subjects) is important in avoiding bias. Journal

editors are usually happy with compliance rates of at least 70%.

Failure to randomize subjects to control and treatment groups in experiments

can also produce bias. If you let people select themselves into the groups, orif you select the groups in any way that makes one group different from

another, then any result you get might reflect the group difference rather than

an effect of the treatment. For this reason, it's important to randomly assignsubjects in a way that ensures the groups are balanced in terms of important

variables that could modify the effect of the treatment (e.g., age, gender,

physical performance). Human subjects may not be happy about being

randomized, so you need to state clearly that it is a condition of taking part.

Often the most important variable to balance is the pre-test value of thedependent variable itself. You can get close to perfectly balancedrandomization for this or another numeric variable as follows: rank-order the

subjects on the value of the variable; split the list up into pairs (or triplets for

three treatments, etc.); assign the lowest ranked subject to a treatment byflipping a coin; assign the next two subjects (the other member of the pair,

and the first member of the next pair) to the other treatment; assign the next

two subjects to the first treatment, and so on. If you have male and female

subjects, or any other grouping that you think might affect the treatment,perform this randomization process for each group ranked separately. Data

from such pair-matched studies can be analyzed in ways that may increase

the precision of the estimate of the treatment effect. Watch this space for anupdate shortly.

When selecting subjects and designing protocols for experiments, researchersoften strive to eliminate all variation in subject characteristics and behaviors.

Their aim is to get greater precision in the estimate of the effect of the

treatment. The problem with this approach is that the effect generalizes onlyto subjects with the same narrow range of characteristics and behaviors as in

the sample. Depending on the nature of the study, you may therefore have to

strike a balance between precision and applicability. If you lean towards

applicability, your subjects will vary substantially on some characteristic orbehavior that you should measure and include in your analysis. Seebelow.

SAMPLE SIZE

How many subjects should you study? You can approach this crucial issue

via statistical significance, confidence intervals, or "on the fly".

Via Statistical Significance
http://www.sportsci.org/jour/0001/wghdesign.html#indiv_diffshttp://www.sportsci.org/jour/0001/wghdesign.html#indiv_diffshttp://www.sportsci.org/jour/0001/wghdesign.html#indiv_diffshttp://www.sportsci.org/jour/0001/wghdesign.html#indiv_diffs


6/21

Statistical significanceis the standard but somewhat complicated approach.

Your sample size has to be big enough for you to be sure you will detect thesmallest worthwhile effect or relationship between your variables. To be

sure means detecting the effect 80% of the time.Detectmeans getting a

statistically significant effect, which means that more than 95% of the time

you'd expect to see a value for the effect numerically smaller than what youobserved, if there was no effect at all in the population (in other words, the p

value for the effect has to be less than 0.05). Smallest worthwhile

effectmeans the smallest effect that would make a difference to the lives ofyour subjects or to your interpretation of whatever you are studying. If you

have too few subjects in your study and you get a statistically significant

effect, most people regard your finding as publishable. But if the effect is notsignificant with a small sample size, most people regard it (erroneously) as

unpublishable.

Via Confidence Intervals

Usingconfidence intervalsor confidence limits is a more accessible approachto sample-size estimation and interpretation of outcomes. You simply want

enough subjects to give acceptable precision for the effect you are

studying. Precisionrefers usually to a 95% confidence interval for the truevalue of the effect: the range within which the true (population) value for the

effect is 95% likely to fall.Acceptable means it won't matter to your subjects

(or to your interpretation of whatever you are studying) if the true value of

the effect is as large as the upper limit or as small as the lower limit. A bonusof using confidence intervals to justify your choice of sample size is that the

sample size is about half what you need if you use statistical significance.

"On the Fly"

An acceptable width for the confidence interval depends on the magnitude ofthe observed effect. If the observed effect is close to zero, the confidence

interval has to be narrow, to exclude the possibility that the true (population)

value could be substantially positive or substantially negative. If the observedeffect is large, the confidence interval can be wider, because the true value of

the effect is still large at either end of the confidence interval. I therefore

recommend getting yoursample size on the fly: start a study with a small

sample size, then increase the number of subjects until you get a confidenceinterval that is appropriate for the magnitude of the effect that you end up

with. I have run simulations to show the resulting magnitudes of effects are

not substantially biased.

Effect of Research Design

The type of design you choose for your study has a major impact on the

sample size. Descriptive studies need hundreds of subjects to give acceptable
http://www.sportsci.org/resource/stats/pvalues.htmlhttp://www.sportsci.org/resource/stats/pvalues.htmlhttp://www.sportsci.org/resource/stats/generalize.html#viaclhttp://www.sportsci.org/resource/stats/generalize.html#viaclhttp://www.sportsci.org/resource/stats/generalize.html#viaclhttp://www.sportsci.org/resource/stats/ssonthefly.htmlhttp://www.sportsci.org/resource/stats/ssonthefly.htmlhttp://www.sportsci.org/resource/stats/ssonthefly.htmlhttp://www.sportsci.org/resource/stats/ssonthefly.htmlhttp://www.sportsci.org/resource/stats/generalize.html#viaclhttp://www.sportsci.org/resource/stats/pvalues.html


7/21

confidence intervals (or to ensure statistical significance) for small effects.

Experiments generally need a lot less--often one-tenth as many--because it'seasier to see changes within subjects than differences between groups of

subjects. Crossovers need even less--one-quarter of the number for an

equivalent trial with a control group--because every subject gets the

experimental treatment. I givedetailson the stats pages at this site.

Effect of Validity and Reliability

The precision with which you measure things also has a major impact on

sample size: the worse your measurements, the more subjects you need to liftthe signal (the effect) out of the noise (the errors in measurement). Precision

is expressed asvalidity and reliability. Validity represents how well a

variable measures what it is supposed to. Validity is important in descriptive

studies: if the validity of the main variables is poor, you may need thousandsrather than hundreds of subjects. Reliability tells you how reproducible your

measures are on a retest, so it impacts experimental studies: the more reliablea measure, the less subjects you need to see a small change in the measure.For example, a controlled trial with 20 subjects in each group or a crossover

with 10 subjects may be sufficient to characterize even a small effect, if the

measure is highly reliable. See thedetailson the stats pages.

Pilot Studies

As a student researcher, you might not have enough time or resources to get a

sample of optimum size. Your study can nevertheless be a pilot for a larger

study. Perform a pilot study to develop, adapt, or check the feasibility of

techniques, to determine the reliability of measures, and/or to calculate howbig the final sample needs to be. In the latter case, the pilot should have the

same sampling procedure and techniques as in the larger study.

For experimental designs, a pilot study can consist of the first 10 or so

observations of a larger study. If you get respectable confidence limits, theremay be no point in continuing to a larger sample. Publish and move on to the

next project or lab!

If you can't test enough subjects to get an acceptably narrow confidence

interval, you should still be able to publish your finding, because your study

will set useful bounds on how big and how small the effect can be. A

statistician can also combine your finding with the findings of similar studiesin something called a meta-analysis, which derives a confidence interval for

the effect from several studies. If your study is not published, it can't

contribute to the meta-analysis! Many reviewers and editors do not appreciatethis important point, because they are locked into thinking that only

statistically significant results are publishable.
http://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.htmlhttp://www.sportsci.org/resource/stats/ssdetermine.html


8/21

WHAT TO MEASURE

In any study, you measure the characteristics of the subjects, and

theindependent and dependent variables defining the research question.

For experiments, you can also measure mechanism variables, which help

you explain how the treatment works.

Characteristics of Subjects

You must report sufficient information about your subjects to identify thepopulation group from which they were drawn. For human subjects, variables

such as sex, age, height, weight, socioeconomic status, and ethnic origin are

common, depending on the focus of the study.

Show the ability of athletic subjects as current or personal-best performance,

preferably expressed as a percent of world-record. For endurance athletes a

direct or indirect estimate of maximum oxygen consumption helpscharacterize ability in a manner that is largely independent of the sport.

Dependent and Independent Variables

Usually you have a good idea of the question you want to answer. Thatquestion defines the main variables to measure. For example, if you are

interested in enhancing sprint performance, your dependentvariable (or outcome variable) is automatically some measure of sprint

performance. Cast around for the way to measure this dependent variablewith as much precision as possible.

Next, identify all the things that could affect the dependent variable. Thesethings are the independent variables: training, sex, the treatment in an

experimental study, and so on.

For a descriptive study with a wide focus (a "fishing expedition"), your main

interest is estimating the effect of everything that is likely to affect thedependent variable, so you include as many independent variables as

resources allow. For the large sample sizes that you should use in a

descriptive study, including these variables does not lead to substantial loss

of precision in the effect statistics, but beware: the more effects you look for,

the more likely the true value of at least one of them lies outside itsconfidence interval (a problem I callcumulative Type 0 error). For a

descriptive study with a narrower focus (e.g., the relationship between

training and performance), you still measure variables likely to be associatedwith the outcome variable (e.g., age-group, sex, competitive status), because

either you restrict the sample to a particular subgroup defined by these

variables (e.g., veteran male elite athletes) or you include the variables in theanalysis.
http://www.sportsci.org/resource/stats/errors.htmlhttp://www.sportsci.org/resource/stats/errors.htmlhttp://www.sportsci.org/resource/stats/errors.htmlhttp://www.sportsci.org/resource/stats/errors.html


9/21

For an experimental study, the main independent variable is the one

indicating when the dependent variable is measured (e.g., before, during, andafter the treatment). If there is a control group (as in controlled trials) or

control treatment (as in crossovers), the identity of the group or treatment is

another essential independent variable (e.g., Drug A, Drug B, placebo in a

controlled trial; drug-first and placebo-first in a crossover). These variablesobviously have an affect on the dependent variable, so you automatically

include them in any analysis.

Variables such as sex, age, diet, training status, and variables from blood or

exercise tests can also affect the outcome in an experiment. For example, theresponse of males to the treatment might be different from that of females.

Such variables account for individual differences in the response to the

treatment, so it's important to take them into account. As for descriptive

studies, either you restrict the study to one sex, one age, and so on, or yousample both sexes, various ages, and so on, then analyze the data with these

variables included ascovariates. I favor the latter approach, because it widensthe applicability of your findings, but once again there is the problem ofcumulative Type 0 error for the effect of these covariates. An additionalproblem with small sample sizes is loss of precision of the estimate of the

effect, if you include more than two or three of these variables in the analysis.

Mechanism Variables

With experiments, the main challenge is to determine the magnitude and

confidence intervals of the treatment effect. But sometimes you want to know

the mechanismof the treatment--how the treatment works or doesn't work.

To address this issue, try to find one or more variables that might connect thetreatment to the outcome variable, and measure these at the same times as the

dependent variable. For example, you might want to determine whether a

particular training method enhanced strength by increasing muscle mass, soyou might measure limb girths at the same time as the strength tests. When

you analyze the data, look for associations between change in limb girth and

change in strength. Keep in mind that errors of measurement will tend toobscure the true association.

This kind of approach to mechanisms is effectively a descriptive study on the

difference scores of the variables, so it can provide only suggestive evidencefor or against a particular mechanism. To understand this point, think about

the example of the limb girths and strength: an increase in muscle size does

not necessarily cause an increase in strength--other changes that you haven'tmeasured might have done that. To really nail a mechanism, you have to

devise another experiment aimed at changing the putative mechanism

variable while you control everything else. But that's another researchproject. Meanwhile, it is sensible to use your current experiment to find

suggestive evidence of a mechanism, provided it doesn't entail too much


10/21

extra work or expense. And if it's research for a PhD, you are expected to

measure one or more mechanism variables and discuss intelligently what thedata mean.

Finally, a useful application for mechanism variables: they can define the

magnitude ofplacebo effects in unblinded experiments. In such experiments,there is always a doubt that any treatment effect can be partly or wholly a

placebo effect. But if you find a correlation between the change in thedependent variable and change in an objective mechanism variable--one that

cannot be affected by the psychological state of the subject--then you can say

for sure that the treatment effect is not all placebo. And the stronger thecorrelation, the smaller the placebo effect. The method works only if there

are individual differences in the response to the treatment, because you can't

get a correlation if every subject has the same change in the dependent

variable. (Keep in mind that some apparent variability in the responsebetween subjects is likely to be random error in the dependent variable, rather

than true individual differences in the response to the treatment.)

Surprisingly, the objective variable can be almost anything, provided the

subject is unaware of any change in it. In our example of strength training,

limb girth is not a good variable to exclude a placebo effect: subjects mayhave noticed their muscles get bigger, so they may have expected to do better

in a strength test. In fact, any noticeable changes could inspire a placebo

effect, so any objective variables that correlate with the noticeable change

won't be useful to exclude a placebo effect. Think about it. But if the subjectsnoticed nothing other than a change in strength, and you found an association

between change in blood lipids, say, and change in strength, then the change

in strength cannot all be a placebo effect. Unless, of course, changes in bloodlipids are related to susceptibility to suggestion...unlikely, don't you think?

editor=AT=sportsci.orgHomepage2000Webmastered by Will HopkinsUploaded 4 May 00Updated extensively from anearlier version(1998)

MODULE R12DESCRIPTIVE RESEARCH

Descriptive research is used to obtain information concerning the current status of the phenomena todescribe "what exists" with respect to variables or conditions in a situation. The methods involvedrange from the survey which describes the status quo, the correlation study which investigates therelationship between variables, to developmental studies which seek to determine changes over time.

Statement of the problem
mailto:editor=AT=sportsci.orgmailto:editor=AT=sportsci.orghttp://www.sportsci.org/index.htmlhttp://www.sportsci.org/index.htmlhttp://www.sportsci.org/index.htmlhttp://www.sportsci.org/copyright.htmlhttp://www.sportsci.org/copyright.htmlhttp://www.sportsci.org/copyright.htmlhttp://www.sportsci.org/resource/design/design.htmlhttp://www.sportsci.org/resource/design/design.htmlhttp://www.sportsci.org/resource/design/design.htmlhttp://www.sportsci.org/resource/design/design.htmlhttp://www.sportsci.org/copyright.htmlhttp://www.sportsci.org/index.htmlmailto:editor=AT=sportsci.org


11/21

Identification of information needed to solve the problem

Selection or development of instruments for gathering the information

Identification of target population and determination of sampling procedure

Design of procedure for information collection

Collection of information

Analysis of informationGeneralizations and/or predictions

Survey StudiesSurvey studies assess the characteristics of whole populations of people or situations.

School Surveys - Used to gather data concerned with internal or externalcharacteristics of a school systemJob Analysis - Used to gather information to be used in structuring a trainingprogram for a particular jobDocumentary Analysis - Closely akin to historical research; deals withdocumenting present situationsPublic Opinion Surveys - Used to enhance the decision making process bygovernment officialsCommunity Surveys - Used to gather data concerned with internal or externalcharacteristics of a community

Interrelationship StudiesInterrelationship Studies trace relationships among the facts obtained to gain a deeper insight intothe situation

Case Studies - Probes in depth into an individual situation or personality with theintent of diagnosing a particular condition and recommending corrective measures Causal Comparative Studies - Compares the likeness and difference amongphenomena to determine if certain factors or circumstances tend to accompanycertain events, conditions, or processesCorrelation Studies - Determine the extent of the relationship between two ormore variables

Developmental Studies


12/21

Developmental studies are concerned with the existing status and interrelationships of phenomenaand changes that take place as a function of time

Growth Studies - May be either longitudinal or cross-sectional. The longitudinaltechnique is the most satisfactory for studying human development. The cross-sectional technique is more commonly used because it is less expensive. Trend Studies - Used to make predictions from social trends, economic conditions,technological advances, etc. to future statusModel or System Development - Creative development of a model or system(paradigm) based on a thorough determination of the present situation or systemand the goals sought

EvaluationCritical Examination of Source Materials - Descriptive studies cannot produceuseful findings if the investigation is based on erroneous information Technical Terminology - An absence of clearly assigned meanings for terms resultsin ambiguous communications which cannot provide a solid foundation forscientific understandingFormulation of Hypotheses - Ambiguously formulated overgeneralized or logicallyunsound hypotheses lead to erroneous conclusions

Observation and Experimentation - Experimentation involves the manipulation ofindependent variables under controlled laboratory conditions and directobservation of the results. Observing the phenomena as they exist may be the onlyway to examine and analyze the factors that are associated with their occurrences

in a natural situationGeneralization and Prediction - Universal generalizations that permit highlyaccurate predictions may be ideal, but even the physical scientists are less certaintoday than they once were of their ability to predict in certain areas except interms of scientific probability.MODULE R13

EXPERIMENTAL RESEARCH AND DESIGNExperimental Research - An attempt by the researcher to maintain control over all factors that may

affect the result of an experiment. In doing this, the researcher attempts to determine or predictwhat may occur.

Experimental Design - A blueprint of the procedure that enables the researcher to test hishypothesis by reaching valid conclusions about relationships between independent and dependentvariables. It refers to the conceptual framework within which the experiment is conducted.


13/21

Steps involved in conducting an experimental studyIdentify and define the problem.

Formulate hypotheses and deduce their consequences.

Construct an experimental design that represents all the elements, conditions, and

relations of the consequences.

1. Select sample of subjects.

2. Group or pair subjects.

3. Identify and control non experimental factors.

4. Select or construct, and validate instruments to measure outcomes.

5. Conduct pilot study.

6. Determine place, time, and duration of the experiment.

Conduct the experiment.

Compile raw data and reduce to usable form.

Apply an appropriate test of significance.

Essentials of Experimental ResearchManipulation of an independent variable.An attempt is made to hold all other variables except the dependent variableconstant - control.Effect is observed of the manipulation of the independent variable on thedependent variable - observation.

Experimental control attempts to predict events that will occur in the experimentalsetting by neutralizing the effects of other factors.

Methods of Experimental Control

Physical Control

Gives all subjects equal exposure to the independent variable.Controls non experimental variables that affect the dependent variable. Selective Control - Manipulate indirectly by selecting in or out variables thatcannot be controlled.Statistical Control - Variables not conducive to physical or selective manipulationmay be controlled by statistical techniques (example: covariance).


14/21

Validity of Experimental DesignInternal Validityasks did the experimental treatment make the difference in this

specific instance rather than other extraneous variables? External Validityasks to what populations, settings, treatment variables, andmeasurement variables can this observed effect be generalized?

Factors Jeopardizing Internal Validity

History - The events occurring between the first and second measurements inaddition to the experimental variable which might affect the measurement.

Example: Researcher collects gross sales data before and after a 5 day 50%off sale. During the sale a hurricane occurs and results of the study may beaffected because of the hurricane, not the sale.

Maturation - The process of maturing which takes place in the individual duringthe duration of the experiment which is not a result of specific events but ofsimply growing older, growing more tired, or similar changes.

Example: Subjects become tired after completing a training session, andtheir responses on the Posttest are affected.

Pre-testing - The effect created on the second measurement by having a

measurement before the experiment.

Example: Subjects take a Pretest and think about some of the items. On thePosttest they change to answers they feel are more acceptable.Experimental group learns from the pretest.

Measuring Instruments - Changes in instruments, calibration of instruments,observers, or scorers may cause changes in the measurements.

Example: Interviewers are very careful with their first two or threeinterviews but on the 4th, 5th, 6th become fatigued and are less careful andmake errors.

Statistical Regression - Groups are chosen because of extreme scores ofmeasurements; those scores or measurements tend to move toward the mean withrepeated measurements even without an experimental variable.


15/21

Example: Managers who are performing poorly are selected for training.Their average Posttest scores will be higher than their Pretest scoresbecause of statistical regression, even if no training were given.

Differential Selection - Different individuals or groups would have different

previous knowledge or ability which would affect the final measurement if nottaken into account.

Example: A group of subjects who have viewed a TV program is comparedwith a group which has not. There is no way of knowing that the groupswould have been equivalent since they were not randomly assigned to viewthe TV program.

Experimental Mortality - The loss of subjects from comparison groups couldgreatly affect the comparisons because of unique characteristics of those subjects.

Groups to be compared need to be the same after as before the experiment.

Example: Over a 6 month experiment aimed to change accounting practices,12 accountants drop out of the experimental group and none drop out of thecontrol group. Not only is there differential loss in the two groups, but the12 dropouts may be very different from those who remained in theexperimental group.

Interaction of Factors, such as Selection Maturation, etc. - Combinations of thesefactors may interact especially in multiple group comparisons to produce erroneousmeasurements.

Factors Jeopardizing External Validity or Generalizability

Pre-Testing -Individuals who were pretested might be less or more sensitiveto the experimental variable or might have "learned" from the pre-testmaking them unrepresentative of the population who had not been pre-tested.

Example: Prior to viewing a film on Environmental Effects of Chemical, agroup of subjects is given a 60 item antichemical test. Taking the Pretest

may increase the effect of the film. The film may not be effective for anonpretested group.

Differential Selection - The selection of the subjects determines how thefindings can be generalized. Subjects selected from a small group or onewith particular characteristics would limit generalizability. Randomly chosen


16/21

subjects from the entire population could be generalized to the entirepopulation.

Example: Researcher, requesting permission to conduct experiment, isturned down by 11 corporations, but the 12th corporation grant permission.

The 12th corporation is obviously different then the others because theyaccepted. Thus subjects in the 12th corporation may be more accepting orsensitive to the treatment.

Experimental Procedures - The experimental procedures and arrangementshave a certain amount of effect on the subjects in the experimentalsettings. Generalization to persons not in the experimental setting may beprecluded.

Example: Department heads realize they are being studied, try to guess

what the experimenter wants and respond accordingly rather than respondto the treatment.

Multiple Treatment Interference - If the subjects are exposed to more thanone treatment then the findings could only be generalized to individualsexposed to the same treatments in the same order of presentation.

Example: A group of CPAs is given training in working with managersfollowed by training in working with comptrollers. Since training effectscannot be deleted, the first training will affect the second.

Tools of Experimental Design Used to Control Factors JeopardizingValidity

Pre-Test - The pre-test, or measurement before the experiment begins, can aidcontrol for differential selection by determining the presence or knowledge of theexperimental variable before the experiment begins. It can aid control ofexperimental mortality because the subjects can be removed from the entirecomparison by removing their pre-tests.

However, pre-tests cause problems by their effect on the secondmeasurement and by causing generalizability problems to a population notpre-tested and those with no experimental arrangements.

Control Group -The use of a matched or similar group which is not exposed to theexperimental variable can help reduce the effect of History, Maturation,Instrumentation, and Interaction of Factors. The control group is exposed to allconditions of the experiment except the experimental variable.


17/21

Randomization - Use of random selection procedures for subjects can aid incontrol of Statistical Regression, Differential Selection, and the Interaction ofFactors. It greatly increases generalizability by helping make the groupsrepresentative of the populations.Additional Groups - The effects of Pre-tests and Experimental Procedures can be

partially controlled through the use of groups which were not pre-tested orexposed to experimental arrangements. They would have to be used in conjunctionwith other pre-tested groups or other factors jeopardizing validity would bepresent.

The method by which treatments are applied to subjects using these tools tocontrol factors jeopardizing validity is the essence of experimental design.

Tools of Control

Internal SourcesPre-Test/

Post Test

Control Group Randomization Additional

Groups

History X

Maturation X

Pre-Testing X

Measuring Instrument X

Statistical Regression X X

Differential Selection X X

Experimental Mortality X

Interaction of Factors X X

External SourcesPre-Testing X

Differential Selection X X

Procedures X

Multiple Treatment


18/21

Experimental DesignsPre-Experimental Design - loose in structure, could be biased

Aim of the Research Name of the Design Notation Paradigm Comments

To attempt to explain aconsequent by anantecedent

One-shotexperimental casestudy

X O An approach that prematurelylinks antecedents andconsequences. The leastreliable of all experimentalapproaches.

To evaluate the influenceof a variable One group pretest-posttest O X O An approach that provides ameasure of change but canprovide no conclusive results.

To determine the influenceof a variable on one groupand not on another

Static groupcomparison

Group 1: X O

Group 2: - O

Weakness lies in noexamination of pre-experimental equivalence ofgroups. Conclusion is reachedby comparing the performanceof each group to determine theeffect of a variable on one ofthem.

True Experimental Design - greater control and refinement, greater control of validityAim of the Research Name of the Design Notation Paradigm Comments

To study the effect of aninfluence on a carefullycontrolled sample

Pretest-posttestcontrol group

R - - [ O X O

[ O - O

This design has been called "the oldworkhorse of traditionalexperimentation." If effectivelycarried out, this design controls foreight threats of internal validity.Data are analyzed by analysis ofcovariance on posttest scores withthe pretest the covariate.

To minimize the effect of

pretesting

Solomon four-group

designR - - [ O X O

[ O - O

[- X

This is an extension of the pretest-

posttest control group design andprobably the most powerfulexperimental approach. Data areanalyzed by analysis of variance onposttest scores.


19/21

O

[ - - O

To evaluate a situationthat cannot be pretested Posttest only controlgroup R - - [ X O

[ - O

An adaptation of the last two groupsin the Solomon four-group design.Randomness is critical. Probably, thesimplest and best test forsignificance in this design is the t-test.

Quasi-Experimental Design - not randomly selectedAim of the Research Name of the Design Notation Paradigm Comments

To investigate a situationin which random selectionand assignment are notpossible

Nonrandomizedcontrol group pretest-posttest

O X O

O - O

One of the strongest and most widelyused quasi-experimental designs.Differs from experimental designsbecause test and control groups arenot equivalent. Comparing pretestresults will indicate degree ofequivalency between experimentaland control groups.

To determine the influenceof a variable introducedonly after a series of initialobservations and onlywhere one group isavailable

Time seriesexperiment

O O X O O

If substantial change followsintroduction of the variable, then thevariable can be suspect as to thecause of the change. To increaseexternal validity, repeat theexperiment in different places underdifferent conditions.

To bolster the validity ofthe above design with theaddition of a control group

Control group timeseries

O O X O

O

O O - O O

A variant of the above design byaccompanying it with a parallel setof observations without theintroduction of the experimentalvariable.

To control history in timedesigns with a variant ofthe above design

Equivalent time-samples

[X1 O1] [X0 O2] [x1 O3]

An on-again, off-again design inwhich the experimental variable issometimes present, sometimesabsent.

Correlational and Ex Post Facto DesignAim of the Research Name of the Design Notation Paradigm Comments

To seek for cause-effectrelationships between twosets of data

Causal-comparativecorrelational studies

-

Oa Ob

-

A very deceptive procedure thatrequires much insight for its use.Causality cannot be inferred merelybecause a positive and closecorrelation ratio exists.


20/21

To search backward fromconsequent data forantecedent causes

Ex post facto studies This approach is experimentation inreverse. Seldom is proof throughdata substantiation possible. Logicand inference are the principal toolsof this design

Leedy, P.D. (1997). Practical research: Planning and design (6th ed.). Upper Saddle River, NJ:Prentice-Hall, Inc., p. 232-233.

DESCRIPTIVE RESEARCH QUESTIONS AND DESIGNS

Descriptive research is used to answer descriptive research questions: What is happening? How issomething happening? Why is something happening?

Examples:

Whatis the average number of staff development hours per year for teachers in the UnitedStates?

Whatis the association between student-teacher ratios and student achievement in thestates elementary schools?

How does instruction differ among teachers in the district who receive different amounts ofstaff development?

Why do teacher qualifications influence instruction?

Descriptive research designs include the following:

Simple descriptive Comparative descriptive Correlational

Asimple descriptive research design is used when data are collected to describe persons,organizations, settings, or phenomena. For example, a researcher administers a survey to arandomsampleof teachers in the state in order to describe the characteristics of the statespopulationofteachers.

With a comparative descriptive design, the researcher describes two or more groups ofparticipants. For example, a researcher administers a questionnaire to three groups of teachers abouttheir classroom practices. The researcher chooses the three schools because the schools vary in termsof the amount of professional development that they provide to teachers.

Acorrelational research design is used to describe the statistical association between two or morevariables. For example, a researcher measures the student-teacher ratio in each classroom in aschool district and measures the average student achievement on the state assessment in each ofthese same classrooms. Next the researcher uses statistical techniques to measure whether thestudent-teacher ratio and student achievement in the school district are connected numerically; forexample, when the student-teacher ratio changes in value, so does student achievement. Theresearcher can then use the student-teacher ratio to predict student achievement, a techniquecalledregression analysis. When there is more than one predictor variable, the technique ofmultipleregression analysisproduces a multiple correlation that is used for prediction.

EXPERIMENTAL RESEARCH QUESTIONS AND DESIGNS
http://newwindow%28%27..//primer/glossary_frame.asp?term=simple_descriptive%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=simple_descriptive%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparative_descriptive%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparative_descriptive%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=correlational_research_design%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=correlational_research_design%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_sample%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_sample%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_sample%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_sample%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=population%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=population%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=population%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=multiple_regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=multiple_regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=multiple_regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=multiple_regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=multiple_regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=multiple_regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=regression_analysis%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=population%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_sample%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_sample%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=correlational_research_design%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparative_descriptive%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=simple_descriptive%27,400,250);


21/21

Experimental research is used to answer causal research questions:Does something cause aneffect?For example, does a low student-teacher ratio cause higher student achievement?

Experimental research designs include the following:

True experimental(randomized trials) Quasi-experimental

In experimental research, the researcher manipulates or varies anindependent variableandmeasures its effects on one or moredependent variables. In a true experimental design, theresearcherrandomly assignsthe participants who are being studied (also called thesubjects) to twoor morecomparison groups. Sometimes the comparison groups are referred toastreatmentandcontrol groups. Participants in the treatment group receive some type of treatment,such as a special reading program. Participants in the control group do not receive the treatment.

For example, at the beginning of a school year, a researcher randomly assigns all classes in a schooldistrict to have either a low student-teacher ratio (small class, the treatment group) or a normalstudent-teacher ratio (large class, the control group). At the end of the school year, the researcher

measures each students achievement using the state assessment and compares the averageachievement of students in the two sizes of classes. In this example, class size is the independentvariable because class size is being varied or manipulated. Student achievement is the dependentvariable because student achievement is being measured. (Note: Researchers conducted a similarexperiment in the state of Tennessee starting in 1985. The study is known asProject STAR.)

In a quasi-experimental design, the researcher does not randomly assign participants to comparisongroups, usually because random assignment is not feasible. To improve a quasi-experimental design,the researcher canmatchthe comparison groups on characteristics that relate to the dependent

variable. For example, a researcher selects from a school district 10 classes to have low student-teacher ratios and 10 classes to maintain their current high student-teacher ratios. The researcherselects the high-ratio classes based on their similarity to the low-ratio classes in terms of studentsocioeconomic status, a variable that is related to student achieve
http://newwindow%28%27..//primer/glossary_frame.asp?term=experimental_research_design%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=experimental_research_design%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=quasi-experimental_research%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=quasi-experimental_research%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=independent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=independent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=dependent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=dependent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=dependent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_assignment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_assignment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_assignment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=subjects%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=subjects%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=subjects%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparison_groups%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparison_groups%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparison_groups%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=treatment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=treatment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=treatment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=control_group%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=control_group%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=control_group%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=matching%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=matching%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=matching%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=control_group%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=treatment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=comparison_groups%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=subjects%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=random_assignment%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=dependent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=independent_variable%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=quasi-experimental_research%27,400,250);http://newwindow%28%27..//primer/glossary_frame.asp?term=experimental_research_design%27,400,250);

descriptive and experimental research by gurpreet

Documents