Prepared by : The instructor :

Amanj. A. Saeed Dr. Cisa Cavusoglu

Near East University

ELT

2011 - 2012

Validity : refers to the degree to which a study accurately reflects or assesses the specific concept that the researcher is attempting to measure. Validity is concerned with the study's success at measuring what the researchers set out to measures.

Validity : is the appropriateness, meaningfulness, and usefulness of the specific inferences researcher make based on the data they collect.

Validation of an instrument : is the process of collecting evidence to support such inferences.

-An appropriate inference: would be one that is relevant, that is, related, to the purpose of the study.

-A meaningful inference : is one that says something about the meaning of the information ( such as test scores ) obtained through the use of an instrument .

-A usefulness inferences : is one that helps researchers make a decision related to what they were trying to find out.

An example of validity. Many recreational activities of high school

students involve driving cars. A researcher, wanting to measure whether recreational activities have a negative effect on grade point average in high school students, might conduct a survey asking how many students drive to school and then attempt to find a correlation between these two factors. Because many students might use their cars for purposes other than or in addition to recreation (e.g., driving to work after school, driving to school rather than walking or taking a bus), this research study might prove invalid. Even if a strong correlation was found between driving and grade point average, driving to school in and of itself would seem to be an invalid measure of recreational activity.

How we know the measure is valid ?

VALID

TESTSAYSYES

TESTSAYSNO

REALITY YES

RIGHT

FALSENEGATIVE

REALITY NO

FALSEPOSITIVE

RIGHT

How to ensure test validity :

An effective test development process

1 -Job analysis

–Identify Requirements

2 -Test

Development

3 -First test

-Obtain current norm

4 -Test

Finalization

5-Measure

Success

What kinds of evidence might a researcher collect ?

•1-Content-related evidence of Validity.

•2-Criterion-related evidence of Validity.

•3-Construct-related evidence of Validity

•4-Face Validity

•5-Internal Validity

•6-External Validity

•7-Experimental Validity

1-Content-related evidence of Validity. Content validity is based on the extent to which a

measurement reflects the specific intended domain of content (Carmines & Zeller, 1991).

It Refers to the nature of the content included within the instrument and the specifications the researcher used to formulate the content.

How appropriate is the content? How comprehensive? Does it logically get at the intend variable? How adequately does the sample of items or

questions represent the content to be assessed?

.

Content validity is illustrated using the following examples: Researchers aim to study mathematical learning and create a survey to test for mathematical skill. If these researchers only tested for multiplication and then drew conclusions from that survey, their study would not show content validity because it excludes other mathematical functions.

Although the establishment of content validity for placement-type exams seems relatively straight-forward, the process becomes more complex as it moves into the more abstract domain of socio-cultural studies. For example, a researcher needing to measure an attitude like self-esteem must decide what constitutes a relevant domain of content for that attitude. For socio-cultural studies, content validity forces the researchers to define the very domains they are attempting to study.

One key element in content- related validity evidence, then, revolves around the adequacy of the sampling.

To illustrate how a researcher might go about trying to establish content-related validity let us consider an example.

Example : Suppose a researcher desires to measure students ability to ( Use information that they have previously acquired ). When asked what she mean by this phrase, she offers the following definition.

As evidence that students can use previously acquired information, they should be able to :

1- Draw a correct conclusion (verbally or in writing )that is based on information they are given.

2- Identify one or more logical implications that follow from a given point if view.

3- state ( orally or in writing ) whether two ideas are initial, similar, unrelated, or contradictory.

2-Criterion- related evidence of validity

Refers to the relationship between scores obtained using the instrument and scores obtained using one or other instruments or measures ( often called a criterion ).

-How strong is this relationship? -How well do much scores estimate present or

predict future performance on certain tasks?

There are two forms of criterion- related validity

Productive and Concurrent- Productive validity : To obtain evidence of

productive validity researcher allow a time interval to elapse between administration of the instrument and obtaining the criterion scores. For example, a researcher might administer a science aptitude test to a group of high school students to predict and later compare their scores on the test with their end-of-the semester grades in science courses.

- Concurrent validity : When instrument data

and criterion data are gathered at nearly the same time, and the results compared, this is an attempt by researcher to obtain evidence of concurrent validity. An example is when a researcher administers of self-esteem inventory to a group of eight grades and compares their scores on it with their teachers ratings of student self-esteem obtain at about the same time.

A key index in both forms of criterion- related validity is the correlation coefficient.

The difference between productive validity and concurrent validity

Productive validity

We need a time interval to elapse .

We compare the conclusion and the scores of the test at the end.

3- Construct-related evidence of validity. Refers to the nature of the psychological

construct or characteristic being measured by the instrument.

-How well does this construct explain difference in the behaviours of individuals of their performance in certain tasks?

Usually, there are three steps involved in obtaining construct-related evidence of validity.

1-The variable being measured is clearly defined 2- hypotheses , based on a theory underlying the

variable .3- The hypotheses are tested both logically and

empirically .

4- Face ValidityFace validity is concerned with how a measure or

procedure appears.

-Does it seem like a reasonable way to gain the information the researchers are attempting to obtain?

-Does it seem well designed?

-Does it seem as though it will work reliably?

Unlike content validity face validity does not depend on established theories for support (Fink, 1995).

5- Internal Validity Internal validity occurs when a researcher

controls all extraneous variables and the only variable influencing the results of a study is the one being manipulated by the researcher. This means that the variable the researcher intended to study is indeed the one affecting the results and not some other, unwanted variables.

Threats to internal validity including: history, maturation of participants, testing, instrument decay, statistical regression, selecting, and mortality.

School 1 Class 1 of 8th grade Same

physical Teacher A boys randomly assigned performance Program 1 test give to both

classes School 1 Class 2 of 8th grade by Teacher

A Teacher A boys randomly assigned Program 2

16 weeks of program implementation

Overall Scheme of a Hypothetical Research Study That Has High Internal Validity .

School 1 4 Classes Test constructed Science Achievement

Teacher A Heterogeneous By Teacher A Tested

Materials 1 Ability School 11 3 Classes Test constructed Teacher B Heterogeneous By Teacher B Materials 2 Ability School 111 4 Classes Test constructed Teacher C High Ability By Teacher C Materials 3

9 weeks Intrusion

General Scheme of a Hypothetical Research Study That Lacks Internal Validity

6-External validity Involves the extent to which the results of

a study can be generalized (applied) beyond the sample. In other words, can you apply what you found in your study to other people (population validity) or settings (ecological validity).  A study of fifth graders in a rural school that found one method of teaching spelling was superior to another may not be applicable with third graders (population) in an urban school (ecological).

POPULATION VALIDITYPopulation validity is a type of external validity

which describes how well the sample used can be extrapolated to a population as a whole. by Martyn Shuttleworth (2009)

It evaluates whether the sample population represents the entire population, and also whether the sampling method is acceptable.

Experienced scientists ensure that their sample groups are as representative as possible, striving to use random selection rather than convenience sampling

Threats to External ValidityPopulation Validity :the extent to which the

results of a study can be generalized from the specific sample that was studied to a larger group of subjects

Ecological Validity : the extent to which the results of an experiment can be generalized from the set of environmental conditions created by the researcher to other environmental conditions (settings and conditions).

7- Experimental validity

Is used here as defined by Campbell and Stanley. ( 1963 ) and by Cook and Campbell ( 1979 ) and is consisted to be of four types : internal , external , construct, and statistical conclusion.

Statistical conclusion validity : refers to the validity of the decision that there is a statistically significant difference between the experimental group and the control group. This is the first step in deciding whether the experimental treatment has had an effect. A mistake at this point will lead to inaccurate conclusion about the impact of the independent variable to dependent variable.

The Relation between Validity and Reliability.

Figure : 1

The bulls eye in each target represent the information That is desired. Each dot represents a separate score obtained with the instrument. A dot in the bulls eye indicates that the information obtained ( the score ) is the information the researcher desires.

Figure : 2

Figure : 3

GOOD LUCK FOR ALL Prepared By : Amanj. A. Saeed

Master Student at English Language Teaching

Near East University

2011 – 2012

References

Fraenkel, J., R., & Wallen, N., E., (1990). How to design and evaluate research in education. New York.

http://www.experiment-resources.com/population-

validity.Wiersma, W. , & Jurs , S. G., ( 2005 ).

Research Methods in Education. Boston: Pearson Education