topics in special education research

Topics in Special Education

ResearchSession 4: Causal-comparative & Correlational

Research& Survey Methods

Inter-observer agreement (IOA)- involves 2 observers measuring the same behaviors at the same time◦ It is most often used to determine the reliability of

observations of the DEPENDENT VARIABLE Treatment Integrity/Fidelity

(of treatment/intervention/ INDEPENDENT VARIABLE)◦ This is how the researcher measured how well the

treatment/intervention was implemented◦ Commonly done using checklists and other observers

recording the completion of these checklists

Difference between Inter-observer agreement & Treatment integrity/fidelity

The main difference between experimental/quasi-experimental research designs AND other designs is…

They MANIPULATE the independent variable

Basically…..they introduce an intervention, while other methods (except for single-subject) do not systematically introduce an intervention

….seeks to make CAUSAL CONCLUSIONS

Experimental and Quasi-Experimental vs other designs

A detailed explanation of the assignment is posted on the wiki

What should you be doing in your groups?◦ At this point you should have a topic and start

coming up with your framework for your research project (based on literature).

◦ Start to draft your conceptual framework, research questions & identify your dependent and independent variables

◦ You should walk away from your group time with a list of tasks to complete.

Proposal Assignment & Group Work

1. Socially Important Issue:

2. Conceptual Model/Hypothesis:

3. Research Question(s):

4. Dependent Variable:

5. Dependent Variable Measure:

6. Independent Variable:

7. Independent Variable Measure:

8. Research Design:

PSU Human Subjects Research Review Committee (HSRC)

http://www.rsp.pdx.edu/compliance_human.php

Portland State University (PSU) is responsible for the rights and welfare of human subjects involved in research sponsored or conducted by the university. In order to meet this responsibility, the University established the Human Subjects Research Review Committee.

Members are charged with reviewing all research conducted under the auspices of PSU that involves human subjects to ensure adequate protections are in place.

http://www.rsp.pdx.edu/compliance_human.php

Review for Quiz

Steps in the Research/Scientific Process

1. Identify socially important issue

2. Review current literature

3. Define conceptual model

4. Define specific hypothesis(es) and research question(s)

5. Define dependent variable(s)/measure

6. Identify independent variable(s)/measures

7. Select appropriate research design

8. Obtain consents 9. Collect data 10. Analyze data 11. Communicate

results Written presentation Oral presentation

Seeks to make causal conclusions.

Direct manipulation of an independent variable (intervention)

Difference between experimental design and quasi-experimental design is the use of random selection of participants and conditions.

Experimental Design

Refers to whether a study is able to scientifically answer the questions it is intended to answer.

Extent to which your test (or study) measures what it intends to measure.

Validity

Changes observed in the dependent variable (outcome) are due to the effect of the independent variable (intervention)…..

& not to some other unintended variables (extraneous, alternative explanations)

12 threats to internal validity (noted by Mertens, 2010)

E.g.., History, maturation, testing, instrumentation, mortality, etc.

Internal Validity

External Validity= extent to which findings in one study can be applied to another situation.

AKA: ecological validity, generalizability

10 threats posed as questions (noted by Mertens, 2010)

E.g., detail/description of procedures, experimenter effects, sensitization, etc.

External Validity (think generalizability)

Correct Quiz

Steps in the Research/Scientific Process

1. Identify socially important issue

2. Review current literature

3. Define conceptual model

4. Define specific hypothesis(es) and research question(s)

5. Define dependent variable(s)/measure

6. Identify independent variable(s)/measures

7. Select appropriate research design

8. Obtain consents 9. Collect data 10. Analyze data 11. Communicate

results Written presentation Oral presentation

Seeks to make causal conclusions.

Direct manipulation of an independent variable (intervention)

Difference between experimental design and quasi-experimental design is the use of random selection of participants and conditions.

Experimental Design

Refers to whether a study is able to scientifically answer the questions it is intended to answer.

Extent to which your test (or study) measures what it intends to measure.

Validity

Changes observed in the dependent variable (outcome) are due to the effect of the independent variable (intervention)…..

& not to some other unintended variables (extraneous, alternative explanations)

12 threats to internal validity (noted by Mertens, 2010)

E.g.., History, maturation, testing, instrumentation, mortality, etc.

Internal Validity

External Validity= extent to which findings in one study can be applied to another situation.

AKA: ecological validity, generalizability

10 threats posed as questions (noted by Mertens, 2010)

E.g., detail/description of procedures, experimenter effects, sensitization, etc.

External Validity (think generalizability)

Discussion

Lecture

Prepared by M. Hara ([email protected])

Statistics, statistics

Descriptive StatisticsWho is in your data?

sample population

Inferential StatisticsWhat your sample says about the population

sample

population

Mean, Median, Mode, standard deviation, variance Tests of significance

(t-, F-Tests)

Central Tendency◦ Mean- average◦ Median- midpoint in distribution of scores◦ Mode- most frequent score

Variability◦ Range- total extension of the data (e.g., 1-10)◦ Standard Deviation- sum of deviations from the

mean squared. How well the mean summarizes the data.

◦ Variance- standard deviation squared. Used in sophisticated analyses

Descriptive Statistics


Statistics, statistics

Inferential StatisticsWhat your sample says about the population

sample

population

T tests- used when have two groups to compare. ◦ Independent samples t- if groups are independent

Different people in each group◦ Dependent samples t-: if two sets of scores are available for the same

people Matched groups

ANOVA (analysis of variance)- when you have more than 2 groups to compare OR more than one independent variable (reports an F-statistic, which is basically a t-value squared)

ANCOVA (analysis of covariance)- ANOVA that allows for control of the influence of an IV (e.g., characteristics of people) that may vary between your groups before treatment is introduced. ◦ Post-hoc method for matching groups on variables such as age, prior

education, SES, or a measure of performance

Inferential Statistics

Tests of Significance Statistical analyses to determine whether a

difference is statistically significant (probability for result to occur by chance).

Yes or No answer

Alpha level (p=)◦ An established probability level which serves as the

criterion to determine whether to accept or reject the null hypothesis

◦ Common levels in education .01 .05 .10

Objectives 4.1 & 6.1


Data

Addressing “WHAT” questions?

Depth of Information

Qualitative Data

Quantitative DataQuantitative Data

• Survey • Large Scale Assessments

Representative, Generalizability

Details, Depth, and Variability


Numbers with Different Meanings

28

Male

Female

(0)

(1)

Variable Type Example

• Gender

No

Yes

(0)

(1)

• Yes/No

Nominal


Coding Data

29

Check All that Apply

MS Word

MS Excel

Q. Which of the following applications have you used with your students? (Please check ALL that apply)

MS PowerPoint

SPSS

iMovie

iDVD

iTunes

iWeb



30

Male

Female

(0)

(1)


• Gender

No

Yes

(0)

(1)

• Yes/No

• Likert-scale

StronglyDisagree

(1)Disagree

(2)Agree

(3)

StronglyAgree

(4)

Nominal

Ordinal


Coding Data

31

Likert-scale

Q1. Overall, I have a good Parent-teacher Relationship.

DisagreeSomewhat Disagree

Somewhat Agree

AgreeStrongly Agree

Strongly Disagree


Coding Data

32

Likert-scale

Q1. Overall, I have a good Parent-teacher Relationship. 1 2 3 4 5 6

NOTE: Distinguishable

Strongly Disagree

DisagreeSomewhat Disagree

Somewhat Agree

AgreeStrongly Agree


Coding Data

33

Multiple Categories

African American

Asian

Ethnicity

High School

Some College

Education Completed

BA/BS

Master’s

Doctoral

Caucasian

Hispanic

Other ____________

Decline to state


Coding Data

34

Multiple Categories

African American

Asian

Ethnicity

High School

Some College

Education Completed

BA/BS

Master’s

Doctoral

Caucasian

Hispanic

Other ____________

Decline to state

1

2

1

2

3

4

5

3

4

5

888

NOTE: Mutually Exclusive, Exhaustive, and Distinguishable


35

Male

Female

(0)

(1)


• Gender

No

Yes

(0)

(1)

• Yes/No

• Likert-scale

StronglyDisagree

(1)Disagree

(2)Agree

(3)

StronglyAgree

(4)

• Age, Annual Income, Test-score

Nominal

(Interval/Ratio)Scale

Ordinal


Numbers with Different Meaning

36

Nominal


Male

Female

(0)

(1)


• Gender

No

Yes

(0)

(1)

• Yes/No

• Likert-scale

StronglyDisagree

(1)Disagree

(2)Agree

(3)

StronglyAgree

(4)


Ca

teg

ori

ca

lN

um

eric

al/

Co

nti

nu

ou

s

Ordinal



37

Nominal


Male

Female

(0)

(1)


• Gender

No

Yes

(0)

(1)

• Yes/No

• Likert-scale

StronglyDisagree

(1)Disagree

(2)Agree

(3)

StronglyAgree

(4)


Ca

teg

ori

ca

lN

um

eric

al/

Co

nti

nu

ou

s

Who C

ARES??

Ordinal



38

Nominal


Male

Female

(0)

(1)


• Gender

No

Yes

(0)

(1)

• Yes/No

• Likert-scale

StronglyDisagree

(1)Disagree

(2)Agree

(3)

StronglyAgree

(4)


Ca

teg

ori

ca

lN

um

eric

al/

Co

nti

nu

ou

s

Why C

ARE??Ordinal


Variable Types and Analysis

39

DependentVariable

IndependentVariable

Is therean association?

(a.k.a., Outcome) (a.k.a., Predictor)



40

DependentVariable

IndependentVariable


(a.k.a., Outcome) (a.k.a., Predictor)

Where differences culminate



41

DependentVariable

IndependentVariable


(a.k.a., Outcome) (a.k.a., Predictor, Intervention)

Where differences culminate

ContributingFactors


Categorical

Numerical/Continuous

Categorical



42

DependentVariable

IndependentVariable



Categorical


Categorical


Contingency Tables(a.k.a. Cross-tabs)


43

DependentVariable

IndependentVariable

Chi-square testOr χ²-test

http://en.wikipedia.org/wiki/Chi_(letter)



44

DependentVariable

IndependentVariable

Categorical CategoricalContingency Tables(a.k.a. Cross-tabs)

GenderAnnual Salary



45

DependentVariable

IndependentVariable

Categorical CategoricalContingency Tables(a.k.a. Cross-tabs)

Gender

Male Female

An

nu

al S

ala

ry

25K or below

26K - 35K

46K - 55K

36K - 45K

56K - 65K

66K and up

12% 4%

18% 6%

24% 11%

36% 39%

8% 28%

2% 12%


Categorical


Categorical


Analysis of Variance

(a.k.a. ANOVA)


46

DependentVariable

IndependentVariable



47

DependentVariable

IndependentVariable

CategoricalNumerical/Continuous

Analysis of Variance(a.k.a. ANOVA)

SAT9 Math Score

Males Females

t-test or F-test


Categorical


Categorical


Regression


48

DependentVariable

IndependentVariable



49

DependentVariable

IndependentVariable



Regression

SAT9 Math Score

Household Income


Categorical


Categorical


Logistic Regression


50

DependentVariable

IndependentVariable



51

DependentVariable

IndependentVariable

Numerical/ContinuousCategorical

Logistic Regression

High school Exit Exam

SAT 9 Math

Fail

Pass

Probability of passing h.s. exam based on SAT-9 score



52

Nominal


Male

Female

(0)

(1)


• Gender

No

Yes

(0)

(1)

• Yes/No

• Likert-scale

StronglyDisagree

(1)Disagree

(2)Agree

(3)

StronglyAgree

(4)


Ca

teg

ori

ca

lN

um

eric

al/

Co

nti

nu

ou

s

Why C

ARE??Ordinal


Categorical


Categorical


Contingency Tables(a.k.a. Cross-tabs)


53

DependentVariable

IndependentVariable

Chi-square testOr χ²-test

http://en.wikipedia.org/wiki/Chi_(letter)


Categorical


Categorical


Analysis of Variance

(a.k.a. ANOVA)


54

DependentVariable

IndependentVariable


Categorical


Categorical


Regression


55

DependentVariable

IndependentVariable


Categorical


Categorical


Logistic Regression


56

DependentVariable

IndependentVariable

Statistics that function to describe the strength and direction of a relationship between two or more variables◦ Simple correlation coefficient (r=)

◦ Coefficient Determination- (r-squared). Amount of variance that is accounted for by the explanatory (independent or predictor) variable in the response variable (criterion variable).

◦ Multiple regression- to indicate the mount of variance that all of the predictor variables explain.

Correlational Statistics

Way of quantifying the difference between two groups.

Not just was there an effect, but the magnitude of the effect.

Many ways to calculate

ES= [Mean of experimental group] – [Mean of control group]/Standard Deviation

R-squared, Cohens-D

Standard deviation is how well the mean summarizes the data

Effect Size

Investigators attempt to determine the cause of differences that already exist between or among groups of individuals.

Describes conditions that already exist (a.k.a. ex post facto).

The group difference variable is either a variable that cannot be manipulated or one that might have been manipulated but for one reason or another, has not been.

Studies in medicine and sociology are causal-comparative in nature, as are studies of differences between men and women.

What is Causal Comparative Research?

Similarities and Differences Between Causal-Comparative and

Correlational Research

• Similarities– Ex Post Facto research– Attempt to explain

phenomena of interest– Seek to identify variables

that are worthy of later exploration through experimental research

– Neither permits the manipulation of variables

– Attempt to explore causation

• Differences– Causal studies compare

two or more groups of subjects

– Causal studies involve at least one categorical variable

– Causal studies often compare averages or use tables instead of scatterplots and correlation coefficients

Similarities and Differences Between Causal-Comparative and Experimental Research

• Similarities– Require at least one

categorical variable– Both compare group

performances to determine relationships

– Both compare separate groups of subjects

• Differences– In experimental research, the

independent variable is manipulated

– Causal studies are likely to provide much weaker evidence for causation

– In experimental studies, researchers can assign subjects to treatment groups

– The researcher has greater flexibility in formulating the structure of the design in experimental research

Steps Involved in Causal-Comparative Research Problem Formulation

The first step is to identify and define the particular phenomena of interest and consider possible causes

Sample Selection of the sample of individuals to be studied by carefully

identifying the characteristics of select groups Instrumentation

There are no limits on the types of instruments that are used in Causal-comparative studies

Design The basic design involves selecting two or more groups that

differ on a particular variable of interest and comparing them on another variable(s) without manipulation

The Basic Causal-Comparative Designs

Independent DependentGroup variable variable

(a) I C O(Group possesses (Measurement)

characteristic)

II –C O(Group does (Measurement)not possess

characteristic)

(b) I C1 O(Group possesses (Measurement)characteristic 1)

II C2 O(Group possesses (Measurement)characteristic 2)

Examples of the Basic Causal-Comparative Design

Threats to Internal Validity in Causal-Comparative Research Subject Characteristics

The possibility exists that the groups are not equivalent on one or more important variables

One way to control for an extraneous variable is to match subjects from the comparison groups on that variable

Creating or finding homogeneous subgroups would be another way to control for an extraneous variable

The third way to control for an extraneous variable is to use the technique of statistical matching

Other Threats

• Loss of subjects• Location• Instrumentation• History• Maturation

• Data collector bias• Attitude

Data Analysis In a Causal-Comparative Study, the first step is to

construct frequency charts & graphs.

Means and SD are usually calculated if the variables involved are quantitative.

The most commonly used inference test is a t-test for differences between means.

ANCOVAs are useful for these types of studies.

Results should always be interpreted with caution since they do not prove cause and effect.

PSU uses a free web-based survey software called Qualtrics (available for student use)

http://oit.pdx.edu/node/908

Survey Methods

http://oit.pdx.edu/node/908

Is there another way of collecting the information?

Is this method the most efficient and cost effective?

Will it provide you with the information you want in a valid manner?

Determine the purpose of your survey

Have specific goals for the survey Consider alternatives to using a survey to

collect information. Select samples that well represent the

population studied. Use designs that balance costs with errors. Take great care in matching wording to

concepts being measured and the population being studied

Pretest questionnaires and procedures to identify problems prior to the survey

Best Practices

Construct quality checks for each stage of the survey

Maximize cooperation or response rates within the limits of ethical treatment.

Carefully develop and fulfill pledges of confidentiality given to respondents.

Disclose all methods of the survey to permit evaluation and replication

Best practices continued

Step 1- Determine PurposeStep 2- Identify a Sampling Plan & ModeStep 3- Design survey instrument Step 4- Test survey instrumentStep 5- Send out a letter of transmittalStep 6- Deliver the surveyStep 7- Analyze data from survey

Steps to designing, delivering, and analyzing surveys

State specific objectives Consider the types of information needed Use your purpose to guide each of your other steps

and check back to this purpose often!◦ To make sure you are answering research questions

Simple descriptive- one-shot to describe moment in time

Cross-sectional- several groups at one point in time (e.g., 1st, 3rd, 5th grade students)

Longitudinal- cohort at different points (e.g., year 1, 2…after leaving school

Strengths and limitations of each??

Step 1- Determine Purpose of Survey

Identify respondents (based on population you want to sample)◦ Think external validity…way you will generalize your results.

probability or nonprobability methods◦ Probability- each person has a probability of being surveyed.

random sampling, systematic sampling, and stratified sampling◦ Nonprobability methods

convenience sampling, judgment sampling, quota sampling, and snowball sampling

Based on what know about population decide how collect data:◦ Email, Web-based surveys, mail, telephone, personal

interviews, etc.

Step 2- Identify a Sampling Plan & Mode of data collection

Random sampling- Each member of the population has an equal and known chance of being selected

Systematic sampling- called an Nth name selection technique

Stratified sampling- (1) Identify a subset of the population that share at least one common characteristic (males & females; managers & non-managers) and (2) their actual representation in the population, then (3) random sampling is used to select a sufficient number of subjects from each stratum.

Probability sampling

Convenience sampling- selected because they are convenient. often used during preliminary research efforts to get a gross estimate of the

results, without incurring the cost or time required to select a random sample.

Judgment sampling- sample based on judgment. For example, draw the entire sample from one "representative" city, even though the population includes all cities. When using this method, the researcher must be confident that the chosen sample is truly representative of the entire population.

Quota sampling- like stratified sampling: (1) identify like characteristics and percentage of population, (2) convenience or judgment to select participants to represent characteristics and meet quota to represent population

Snowball sampling- Snowball sampling relies on referrals from initial subjects to generate additional subjects.

Non-probability sampling

Review the literature There may be a survey that matches what you need (then use

and cite OR modify and cite) Determine question format

◦ Open-ended, close-ended, likert scale (0-5; never, sometimes, always) Avoid sensitive questions Be very clear Short items are better Avoid negative wording OR use positive wording Avoid asking about more than one idea…stay away from

AND…make another question to ask for additional information Avoid jargon/big words Emphasize …. (underline) critical words

Step 3- Design survey instrument

Test instrument on yourself and others (not part of your sample!)

Pilot test to similar population. Use feedback to improve administration of

survey

Step 4- Test survey instrument

Cover letter for mail surveys or as introductory letter/email for a phone, web, email, or personal interview survey.

Increases participation and limits incomplete surveys

Step 5- Send out a letter of transmittal

Deliver the survey according to timeline Web-based or email surveys may have a

start date and date for completion by. Send follow up emails/letter/calls Monitor data as they come in to check for

errors----sooner than later. Send thank you emails/letters/calls

Step 6- Deliver the survey

Follow up with non-respondents so you have some ideas why there was no response◦ You can then compare respondents vs non-

respondents Decide how to handle missing data Complete descriptive statistics: frequency,

percentages, mean, median, etc. Look for interesting patterns in the data Do sub-group analyses if possible Display data in tables & graphs

Step 7- Analyze the data

Get together with your research group Based on your overall topic and possible

research questions...outline survey methods that you might give to participants or other stakeholders for the purpose of your research.

In-Class Application Activity

topics in special education research

Documents

research project

research questions5

independent variable

behavioral research

dependent variable outcome

appropriate research

independent variablesmeasures7

quasiexperimental design