lect-5, enhancing rigor in quantitative research

ENHANCING RIGOR IN QUANTITATIVE

RESEARCH

Introduction - this topic involve the following:

1. describe strategies that could strengthen quantitative research design

2. ways to minimize biases

3. controlling extraneous variables

Validity

- approximate strength of Inference

- a property of Inference, not by the

research design

Threats to validity

- reasons that an inference could be wrong

- could be anticipated by researchers

and Introduce design to eliminate

or minimize

Types of Validity

1. Statistical Conclusion Validity

- concerns the validity of Inferences that there truly is an empirical relationship or correlation

between

the presumed cause and effect

- the researcher should prove that

the relationship is real

2. Construct validity

- concerns the degree to which an

Intervention is a good representation of the underlying constructs that was

theorized as having the potential to

cause beneficial outcome

3. External Validity

- concerns the generalizability of causal

Inferences

- concern with the observed relationship

will hold over variations in person, setting, time or measure of the

outcome

4. Internal Validity

- concern with the validity that the effect to the Dependent variable is caused by the Independent variable

rather than other factors

Controlling Intrinsic Source of Extraneous Variability

1. Randomization

- most effective method, controls all possible extraneous variable

- secure comparable groups to equalize study groups with

respect to the extraneous variable

2. Crossover

- powerful method of ensuring

equivalence between groups being compared

- subjects served as own control

- not suitable to all studies due to the problem of carry-over effect

3. Homogeneity

- used as an alternative if both randomization and cross-over are

not available

- disadvantage: the result could only be used to same type of subjects

- e.g. If gender is a confounding variable, use all males or all

females as subjects

4. Blocking/Stratification

- Incorporating the extraneous variable in the design

- enhance the likelihood of detecting differences in the experimental and control group

5. Statistical Control

- use of statistical analysis

- e.g. ANCOVA, controls by statistically removing the effect of E.V. to D.V.

6. Matching/Pair Matching

- use knowledge of subjects characteristics to create comparable group

- e.g. If age and gender are E.V., we have to pair the subjects with respect to the age and gender

- disadvantage:

1. to match effectively, researcher should know in advance possible C.V.

2. two or more variables often becomes impossible to pair

Statistical Conclusion Validity

- three important threats

1. Low statistical power

- statistical power is the ability to

detect true relationship

- adequate S.P. could be achieved

a. Using large population

b. constuction or definition of

Independent variable

c. maximizing precision

- using reliable measuring tool and

powerful statistical methods

2. Restriction range

- e.g. Homogeneity of population

- limits the generalizability of study

findings

- threaten the statistical conclusion

validity

3. Variable Implementation of Treatment

- known as treatment fidelity

- concerns the extent to which the implementation of an Intervention is faithful to the plan

- problem: non-full participation of the

subjects (Intervention should be made enjoyable and motivational

- Researcher should maintain constancy

1. Use of standard protocols (manual)

2. Ensure that control group don’t

gained access to the Intervention

3. Manipulation check could be used

- assess whether the treatment was

in place

- was understood and perceived

in an intended manner

Construct Validity

Enhancing

- careful explication of the treatment

IV, outcome setting and person construct of Interest

- carefully select instances that match those constructs as closely as possible

- ensure the use of appropriate tool in

measuring outcome

Threats to Construct Validity

1. Reactivity to the study situation

- subjects may behave in a particular

manner because they are aware of

their participation in the study

- could be reduced by masking or

blinding

2. Researchers expectancies

- researchers Influence on participants ` responses through subtle communication of desired outcome

- could be corrected by masking

3. Novelty effects

- happens when treatment is new subjects and researcher may alter their behavior in various ways, they may be enthusiastic or skeptical

4. Treatment diffusion/Contamination

- known as Blurring, occurs when control group receives services similar to those given to treatment group

- also occurs if participants in treatment group drops or do not fully participates

5. Compensatory Effect

- If healthcare staff or family member try to compensate for the control group failure to receive beneficial treatment

External Validity

Enhancement

- concerns about the correct representatives of population to which the generalization is Intended

- proper setting of the study

- Replicability (multi-sites is powerful)

- enhance generalizability of the result

-create study situation as similar as possible to the real world circumstances

Threats

1. Interaction between relationships and people, an effect might be observed

to a certain group of population but could not in other group

2. Interaction between causal effects and

treatment variations

In external validity, the issue is constancy of relationship rather than if the magnitude of the effect is constant

Internal Validity

- true experimental design posses high degree of Internal validity due to manipulation of certain variable and

randomization

Threats

1. Temporal Ambiguity

- ensure that Independent variable

precedes Dependent variable

- this problem is common in correlation

study

2. Selection

- encompasses biases resulting from

differences between groups

- requires proper random assignment

to achieve equivalence of groups

- common encountered in non- experimental design

3. History

- occurrence of external events that takes place concurrently with Independent Variable that can affect the Dependent Variable

4. Maturation

- processes occurring within the subjects as a result of passage of time rather than a result of Independent

Variable or treatment

- e.g. Physical growth, emotional maturity

5. Mortality/ Attrition

- attrition in groups being compared

- dropping out due to illness, death etc.

6. Testing and Instrumentation

a. Testing

- refers to effects of taking a pre-test on subject performance on a post-test

- the first administration of the test might sensitized the subject

b. Instrumentation

- reflects changes in measuring Instruments or methods between two points of data collection

- e.g. Using baseline and revised Instrument

Internal Validity and Research Design

- each threat represent an alternative explanation (rival hypothesis) that competes the Independent variable

- the aim of strong Research design is to rule out competing explanation

- Experimental design normally rules out most rival hypothesis, however, majority of research design are vulnerable to the threats

Design Threats

1. Case-control and other Retrospective study

Temporal ambiguity

2. Case-control, Natural experiment, Non-equivalence group

Selection

3. Time series, Prospective co-hort, One group post-test, Crossover

History

4. One group pre-test post-test Maturation

5. Prospective co-hort, Longitudinal study, Quasi-experimental

Mortality/Attrition

6. Pre-test - Post-test design Testing/Instrumentation

Internal Validity and Data Analysis

- best strategy to enhance Internal validity; use strong Research Design (with control mechanism and careful design features

- conduct analysis to determine the nature and extent of biases arose

- when biases are detected, the information can be used to Interpret substantive result

but should be statistically controlled

END