INTRO TO RESEARCH METHODS: Empirical Knowledge: based on observations. Answer questions why, whom, how, and when. Experimental research: treatments are given for the purpose of research. Experimental group vs. control group. True Experiments: when participants are divided at random. Non Experimental Research: no treatment. Use survey or poll. Variables of Non Experimental Studies:

Variables: trait or characteristic with 2 or more categories. (i.e. gender and male or female) Mutually exclusive categories: each respondent will belong to only one category. (i.e. choosing a single candidate will you vote) Exhaustive: variable must have a category for each respondents opinion. (i.e. adding an option for being undecided on the voting poll)

Independent vs. Dependent: IV aka predictor (stimulus or input) causes change in the DV aka criterion(respondent or output). (i.e. causal-comparative study to see if smoking (IV) causes lung cancer (DV)) Causal Comparative Study: aka ex post facto study. No treatment and no experiment. Essential characteristics – 1. Researchers observe and describe some condition. 2. Researchers look to the past to try to identify the possible causes of the condition. Longitudinal Research: repeatedly measure traits of participants over time to trace developmental trends. Correlational Research: degree of relationship among two or more quantitative variables. (i.e comparing gpas and scores on sat’s) Quantitative Research: gather data that is easy to quantify allowing for statistical analysis. Deductive. Use random sampling. Use when funding is low, participants aren’t available for long period of time, when you need hard numbers. (numbers/statistics) Qualitative Research: gather data that must be analyzed through informed judgment to identify themes expressed by participants. Inductive. Use purposive sampling. Use when little is known on a topic. To learn about culture.

Hypotheses: prediction of the outcome of a study. Directional vs. Nondirectional (or could use a research purpose/question): predict which group will be higher or have more of some attribute vs. there will be a difference but does not predict the direction of the difference. Null hypothesis: statistical hypothesis.

Conceptual Definition: dictionary definition. Operational Definition: redefining a variable in terms of its physical steps. Program Evaluation: evaluation research/applied research. Used for assessment.

Formative: collect information during course of program to assist in the process of modifying program while it is being implemented. Collect information on the progress toward the ultimate goals. Summative: collect information about participants’ attainment of ultimate goals at the end of the program.

Theory: unified explanation for discrete observation that would otherwise be viewed as unrelated.

REVIEWING LITERATURE: *See class notes for info on lit reviews SAMPLING: Census: total population (only government ever gets this) Unbiased Sample: every member of population gets an equal chance to be included. Simple Random Sample: drawing random names of individuals in population. Sample of Convenience: aka accidental samples, using sample of participants most convenient to researcher. (i.e. a teacher researcher his own students) Systematic Sampling: every nth individual is chosen. The number n can be any number. (i.e. the number could be 2 and the researcher would choose every second individual) Volunteerism: any participant who offers to be part of a study Sampling Error: aka precision, random samples may contain a disproportionate representation of census. (i.e. too many males were randomly chosen) Stratified Random Sampling: divide population into strata (i.e. gender) and then pull randomly from each stratum. Cluster Sampling: drawing groups rather than individuals randomly. Purposive Sampling: purposively select individuals who will be good sources of information. Snowball Sampling: try to find participants that are hard to find. Researcher finds one and they provide information on how to locate others. (i.e. studying heroin addicts) Demographics: background characteristics of participants in research. (i.e. gender, age, income) Morality: if participants drop out of experiment mid course. Type of sampling bias. Sample Size: secondary condition. Increases precision (results vary only a small amount from sample to sample) Pilot Study: obtain preliminary information on how new treatments and instruments work.

Variability: same should accurately represent population.

INSTRUMENTATION: Instrument: any type of measurement device (i.e. test, questionnaire, interview, personality scale) Instrumentation: describing measurement devices. Validity: instrument should measure what it is designed to measure and perform correctly. Validity is a matter of degree – you must discuss whether it is valid. Perfect validity is difficult to have. Content validity: researchers make judgments about instruments appropriateness of its contents. Face validity: judgment made on if instrument appears to be valid on superficial inspection.

*it is possible to have high face validity and low content validity.

Empirical validity: make planned comparisons to see if an instrument yields scores that relate to a criterion. Predictive validity: poses question – to what extent does the test predict the outcome it is supposed to predict? After examinees have had a chance to exhibit the predicted behavior Concurrent coefficient: an independent measure of the same trait that the test is designed to measure. At about the same time that the test is administered. Validity coefficient: aka correlation coefficient use to express validity. Ranges from 0.00 to 1.00. 1.00 equals a perfect validity. Construct validity: relies on subjective judgments and empirical data. (construct – collection of related behaviors associated in a meaningful way. Approaches to validity:

1. Judgmental: Content – make expert judgment of the appropriateness of the content. Face – make judgment based on superficial appearance.

2. Empirical: Predictive – correlate test scores with criterion scores obtained after examinees have had a chance to achieve the outcome that is predicted by the test. Concurrent – correlate test scores with criterion scores obtained at same time as test is administered.

3. Judgmental-Empirical: Construct – hypothesize a relationship between the test scores and scores on another variable. Then, test the hypothesis.

Reliability vs. Validity: test is reliable if it yields consistent results. Can have reliability without validity. Interobserver reliability: use two observers when collecting data. Each collecting a different piece of data (i.e. height and non verbal cues) Test-Retest reliability: measure two different points in time. Administer the same test at two different times. Parallel-Forms Reliability: administer one form of a test then after some time administer second form of test to same participants. Split-Half Reliability: administer test but score items in test as if they were 2 separate tests. (i.e. odd/even split) researcher correlates the scores between the two, yielding what is known as split half reliability coefficient. Range from 0.00 to 1.00. only checks consistency within the test. Belongs to reliability estimates called internal consistency. Non-Referenced Tests: test designed to facilitate comparison of an individual’s performance with that of a norm group. Can look at individuals percentile ranking compared to total scores. Criterion-Referenced Test: test designed to measure the extent to which individual examinees have met performance standards. Interpretation of how examinee performs is independent from other test scores. Achievement test: measures knowledge and skills individuals have acquired. Conduct research on the effectiveness of direct planned instruction. Aptitude Test: predict some specific type of achievement. (i.e. sat’s) Intelligence Test: predict achievement in general, not any specific type. Most popular are cultural, knowledge and skills that can be acquired with instruction. Likert-Type Scales: scale that ranges from strongly agree to strongly disagree. Reverse scoring when strongly disagree correlates with the highest number (1-5)

EXPERIMENTAL DESIGN: Pretest-posttest randomized control group design: by assigning participants at random to groups, there will be no bias. r- random assignment to group o- observation or measurement where pre or post test x- experimental treatment *control condition is represented by a blank space Pretest Sensitization: aka reactive effect of testing. changes observed may be a combination of the pretest and the treatment. R O X O R O O Posttest-only randomized control group design: the random assignment makes the two groups comparable not the pretest. R X O R O Solomon randomized four-group design: combination of design 1 and 2. Can compare first 2 groups to determine how much gain was achieved and can compare the last 2 groups to determine whether the treatment is more effective that the control condition in the absence of the pretest. R O X O R O O R X 0 R 0

*all 3 designs are true experimental designs: all characterized by random assignment to treatments

Threat to internal validity:

History: other environmental influences on the participants between the pretest and posttest. Maturation: participants become older and wiser Instrumentation: changes in instrument from time is was used as a pretest to posttest. Testing: effects of the pretest on the performance exhibited on the posttest Statistical Regression: participants are selected on the basis of extreme scores. Intact Groups: researcher uses 2 comparison groups that are not formed at random. Using pre-existing groups.

Selection: because participants are not assigned at random there is possibility the 2 groups are not the same in all respects. Selection-History Interaction: because selection wasn’t random, there may be systematically subjected to different life experiences. Selection-Maturation Interaction: 2 groups were at different developmental stages at time of pretest, which will lead to different rates of maturation. Mortality: different loss of participants from the groups to be compared. (i.e. people drop out)

Threats to external validity:

Generalize: is it accurate to assume that the treatment administered to the experimental to the experimental group will work as well in the population as it did in the sample? Selection Bias: if a sample is biased a researcher will not be able to generalize results of treatment to entire population. Reactive Effects of experimental Arrangements: experimental setting is different from natural setting in which the population operates the effects observed will not generalize. Reactive Effect of Testing: aka pretest sensitization. Pretest might influence how participants respond to experimental treatment. Multiple-Treatment Interference: group of participants is given more than one treatment

Internal vs. external validity: External = to whom and under what circumstances can the results be generalized. Internal = is the treatment in this particular case responsible for the observed changes. Preexperimental design: limited value of investigating cause-and-effect relationships because of poor internal validity One group pretest-posttest design: O X O One-shot case study: X O Static-Group Comparison Design: 2 groups and dashed line in between indicates intact group aka preexisting group that was not formed at random: X O O Confounding in Experiments: source of confusion in explanation given.

Hawthorne Effect: aka attention effect. 2 different explanations from same experiment. Use 3 groups to control – 1. Experimental group 2. Control group that receives attention 3. Control group receives no attention.

John Henry Effect: control group might become aware of its inferior status and respond by trying to outperform the experimental group. Placebo Effect: individuals to improve simply because they know they are being treated. Researchers control by giving control group a placebo Demand Characteristics: cue that lets participants know expected outcome. Participants try to please researcher and respond they way they feel they are supposed to.

UNDERSTANDING STATISTICS: Scales or Levels of Measurement: Nominal: naming level. Names not numbers used Ordinal: participants are put in rank order (i.e ranking according to height) Interval/ratio: measure how much participants differ from one another.

Ratio has absolute zero. Together called scale No Oil In Rivers Frequency: number of participants or cases. N, number of participants. Percentage: number per one hundred. 44% are democrats, out of 100 44 people are democrats. To determine total number of participants: Multiply total number 2200 by .44 = 968 democrats To calculate percentage - divide the smaller number by the total then multiply by 100. 22 of 84 children are afraid of the dark. 22/84=.2619 x 100 = 26.19% Proportion: part of one. Multiple .26 by 100. Harder to interpret. Frequency Distribution: table that shows how many participants have each score. X f 22 1 21 3 20 4 19 8 18 5 17 2 16 0 15 1 N = 24 Normal Curve: aka bell curve. Most important *insert bell curve Positive Skew: tail to the right *insert positive skew curve

Negative Skew: tail to the left *insert negative skew curve Bimodal Distribution: 2 high points *insert bimodal curve Mean: X. average. Add scores and divide by total number of participants. Balance point in distribution of scores. Median: middle point of distribution. Mode: most frequently occurring score. *in a normal distribution the mean, median, and mode have the same value. *in a positive skew the mean is higher than the median *in a negative skew the median is higher than the mean Range: difference between highest and lowest score. Based on 2 most extreme scores. Outliers: scores that lie vastly outside the range of the majority of other scores. Interquartile Range: is the range of the middle 50% of the participants. It ignores outliers. Find middle 50% then find midpoint at each end and subtract *see class notes for examples Standard Deviation: measure of variability. *insert SD curve example *Insert formula

CORRELATIONAL STATISTICS: Correlation: refers to extent to which two variables are related across a group of participants. Direct/Positive Relationship: those who score high on one variable tent to score high on the other, and those who score low on one variable tend to score low on the other. Inverse/Negative Relationship: those who score high on one variable tend to score low on the other. Pearson r: pearson product-moment correlation coefficient. Pearson r ranges from -1.00 to 1.00. both extreme scores indicate a perfect relationships. 1.00 = perfect direct relationship. -1.00 = perfect inverse relationship. A value of 0.00 indicates an absence of a relationship. Coefficient of Determination: used to interpret pearson r. symbol is r2. To obtain square r. when converted to a percentage, indicates how much variance on one variable is accounted for the variance of another. A percentage can be obtained by multiplying r-squared x 100. (i.e. r = .90: .90 x .90 =.81 x 100 = 81% thus 81% of the variance is accounted for Scattegram: illustrates correlation between 2 variables. When relationship is direct and perfect it will follow a single line from the lower left to upper right. The greater the amount of scatter around the line the weaker the relationship. *insert example of strong direct and strong inverse. Multiple Correlation: ability to predict improves with using 2 variables. *see copied example from pg 82/83

INFERENTIAL STATISTICS: Standard Error of the Mean: is a margin of error. Add the SEm to the mean and then subtract the SEm from the mean = the 2 limits of the 68% confidence interval for the mean are obtained. 75-2 = 73 and 75+2=77. 73 and 77 Point Estimate: single point value. 3 in between the two limits calculated from above. 75. Interval Estimate: reporting the 2 numbers. 73 and 77 Confidence Interval: assist in interpreting means that are subject to random errors. A 95% confidence interval indicates the range of values in which we can have 95% confidence that the true lies. Null Hypothesis: there will be no significant difference in the research. Type 1 Error: error of rejecting the null hypothesis when it is correct. Type 2 Error: failing to reject the null hypothesis when it is false. .06 level = not significant, do not reject the null .05 level = significant, reject the null .01 level = more significant, reject the null .001 level = highly significant, reject the null T Test: the larger the sample the more likely the null will be rejected. The larger the observed difference between 2 means the more likely the null will be rejected. The smaller the variance the more likely the null will be rejected. Independent Data: uncorrelated data. There is no pairing or matching of individuals across the 2 samples. Dependent data: correlated data. Matching of participants assures us that the 2 groups are more similar that 2 independent samples. *see class notes for more information on t-test and formula. One Way Anova: used to test differences among 2 or more means. *see class notes for more information and formulas. Pg 128 shows table Chi-Square: test for differences among frequencies. Effect Size: standardizes the size of the differences between 2 means.

Cohens d: subtract control group mean from the experimental group mean and divide by the standard deviation of the control group. In terms of value of d, an experimental group rarely exceeds a control group by more than 3.00. Experimental group: M = 40.00, SD = 11.00 Control Group: M = 300.00, SD = 10.00 Cohen’s d: (40.00 – 30.00)/10 = 1.00