exercise ch 08

Chapter 8: Validity and Reliability

Activity 8.1: Instrument Validity

Activity 8.2: Instrument Reliability (1)


Activity 8.4: What Kind of Evidence: Content-Related, Criterion-Related, or Construct-Related?

Activity 8.5: What Constitutes Construct-Related Evidence of Validity

Activity 8.1: Instrument Validity

A valid instrument is one that measures what it says it measures. If a researcher is interested in measuring how much a student knows about the U.S. Civil War, for example, he or she needs an instrument that will measure exactly that -- the student’s knowledge -- not his or her feelings, attitudes, beliefs, or skills. For each of the two objectives listed below, write one example of the kind of question or observation you might engage in to measure, at least to some extent, attainment of the objective.

1. Objective: To measure the degree to which a person enjoys modern art

Instrument question or observation strategy:

2. Objective: To measure the level of anxiety that exists among university students during final exam period


3. Objective: To measure the attitudes of local residents toward the building of a new ballpark in downtown San Francisco



A reliable instrument is one that is consistent in what it measures. If an individual scores highly on the first administration of a test, for example, he or she should, if the test is reliable, score highly on a second administration. In this activity, you are going to evaluate the reliability of an instrument.

Imagine that you are conducting a study for which you must develop a mastery test in mathematics for ninth-grade students. You develop a 30-point test and distribute it to a class of 13 ninth-graders in a certain school district on the west coast of the United States in May of 2004. You then give the test again one month later to the day in June, 2004. The scores of the students on the two administrations of the test are shown below. Plot each pair of scores on the scatterplot started below. We have entered “A’s” scores to get you started. What do they suggest to you about the reliability of this? Explain.

30-POINT MATHEMATICS MASTERY 30-POINT MATHEMATICS MASTERYTEST (FIRST ADMINISTRATION) TEST (SECOND ADMINISTRATION)

A 17 15_______________ B 22 18_______________ C 25 21_______________ D 12 15_______________ E 7 14_______________ F 28 27_______________ G 27 24_______________ H 8 5_______________ I 21 25_______________ J 24 21_______________ K 27 27_______________ L 21 19_______________ M 10 15_______________


For each of the situations listed below, match the type of reliability with what the researchers involved are evaluating.

Column A: Situation Column B: Instrument Reliability

1. ______ A researcher develops two versions of a test meant to measure interests in students prior to their taking an examination. He gives one version of the test to a group of college sophomores on a Monday, and the other version of the test to them the next day.

2. ______ A teacher develops a new test for high school biology. She gives the test twice, once to the students in her morning class and once to the students in her afternoon class. She then compares the scores for the two classes of students.

3. ______ A college professor is interested in evaluating her end-of-semester course evaluations that are completed by her students. The instrument consists of 20 five-point rating scale items. She obtains an average score for each student on the first 10 items, and also an average score for each student on the second 10 items. She then compares the scores.

4. ______ A researcher prepares a 15-item multiple-choice test designed to measure student knowledge of the causes of the Spanish-American War. She asks two of her colleagues, specialists in American history, to identify any items that they think do not measure what she is after.

5. ______ A teacher prepares an algebra test and gives it to her students at the end of the semester and again two months later.

a. internal consistencyb. test-retest reliabilityc. equivalent forms reliabilityd. none of the above

6. Which of the following would be a way of assessing unreliability due to content and time?a. Administering a reading test (Form X) on Monday and again one month laterb. Administering a reading test (Form X) on Monday and calculating a split-half

correlationc. Administering a reading test (Form X) on Monday and Form Y one month laterd. Administering a reading test (Form X) on Monday and deleting any questions that

more than 50 percent of those taking the test missed

Activity 8.4: What Kind of Evidence: Content-Related, Criterion-Related or Construct-Related?

As we mention in the text, validity depends on the amount and type of evidence there is to support one’s interpretations concerning data that has been collected. In Chapter Eight, we describe three kinds of evidence that a researcher might collect: content-related, criterion-related, and construct-related evidence of validity.

Listed below are a number of questions that each represent one of these three types. In the space provided, write content if the question refers to content-related evidence, criterion if the question related to criterion-related evidence, and construct if the question refers to construct-related evidence of validity.

1. How strong is the relationship between student scores obtained using this instrument and their teacher’s rating of their ability? ______________

2. How adequately do the questions in the instrument represent that which is being measured? ______________

3. Do the items that the instrument contains logically get at that which is being measured? ______________

4. Are there a variety of different types of evidence (e.g., test scores, teacher ratings, correlations, etc.) that all measure this variable? ______________

5. How well do the scores obtained using this instrument predict future performance? ______________

6. Is the format of the instrument appropriate? ______________

Activity 8.5: What Constitutes Construct-Related Evidence of Validity?

On page 156 of the text, we provide an example of one piece of evidence that could be used to establish construct validity for a pencil and paper test on honesty.

1. In the space provided below, after discussing this with a partner, suggest some additional information that a researcher might collect as evidence of honesty in an effort to establish construct validity for the test.__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

2. What about interest in the subject of chemistry? Suppose another researcher wishes to develop a test to measure an individual’s interest in chemistry. What sort of information might he or she collect in an attempt to establish construct validity for the test?__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem Sheet 8: Validity and Reliability

1. If you plan to use an existing instrument, describe what you have learned about the validity and reliability of scores obtained with this instrument. _________________________________________________________________________________________________________________________________________________________________________________________________________

2. If you plan to develop an instrument, explain how you will try to ensure the validity and reliability of results obtained with this instrument by using one or more of the tips described on page 114 (specify which).:_________________________________________________________________________________________________________________________________________________________________________________________________________

3. If you have not already indicated so above for each instrument that you plan to use, tell specifically how you will check for:

a. internal consistency:_____________________________________________________________________________________________________________________________________________________________________________________

b. stability (reliability over time): _____________________________________________________________________________________________________________________________________________________________________________

c. validity: _______________________________________________________________________________________________________________________________________________________________________________________________

exercise ch 08

Documents