Measurement & Evaluation

Measurement• Measurement is the estimation of the magnitude of some attribute of an object, such

as its length or weight, relative to a unit of measurement. Measurement usually involves using a measuring instrument, such as a ruler or scale, which is calibrated to compare the object to some standard, such as a meter or a kilogram. In science, however, where accurate measurement is crucial, a measurement is understood to have three parts: first, the measurement itself, second, the margin of error, and third, the confidence level -- that is, the probability that the actual property of the physical object is within the margin of error. For example, we might measure the length of an object as 2.34 meters plus or minus 0.01 meter, with a 95% level of confidence.

• Metrology is the scientific study of measurement. In measurement theory a measurement is an observation that reduces an uncertainty expressed as a quantity. As a verb, measurement is making such observations[1]. It includes the estimation of a physical quantity such as distance, energy, temperature, or time. It could also include such things as assessment of attitudes, values and perception in surveys or the testing of aptitudes of individuals.

• In the physical sciences, measurement is most commonly thought of as the ratio of some physical quantity to a standard quantity of the same type, thus a measurement of length is the ratio of a physical length to some standard length, such as a standard meter. Measurements are usually given in terms of a real number times a unit of measurement, for example 2.53 meters, but sometimes measurements use complex numbers, as in measurements of electrical impedance.

Evaluation• "Evaluation" is the systematic determination of merit, worth, and

significance of something or someone. Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises, including the Arts, criminal justice, foundations and non-profit organizations, government, health care, and other human services.

• Evaluation is a methodological area that is closely related to, but distinguishable from more traditional social research. Evaluation utilizes many of the same methodologies used in traditional social research, but because evaluation takes place within a political and organizational context, it requires group skills, management ability, political dexterity, sensitivity to multiple stakeholders and other skills that social research in general does not rely on as much. Here we introduce the idea of evaluation and some of the major terms and issues in the field.

• “House” considers all major evaluation approaches to be based on a common ideology, liberal democracy. Important principles of this ideology include freedom of choice, the uniqueness of the individual, and empirical inquiry grounded in objectivity. He also contends they all are based on subjectivist ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian, in which “the good” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist / pluralist, in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified.

Types of Evaluation

• There are many different types of evaluations depending on the object being evaluated and the purpose of the evaluation. Perhaps the most important basic distinction in evaluation types is that between formative and summative evaluation. Formative evaluations strengthen or improve the object being evaluated -- they help form it by examining the delivery of the program or technology, the quality of its implementation, and the assessment of the organizational context, personnel, procedures, inputs, and so on. Summative evaluations, in contrast, examine the effects or outcomes of some object -- they summarize it by describing what happens subsequent to delivery of the program or technology; assessing whether the object can be said to have caused the outcome; determining the overall impact of the causal factor beyond only the immediate target outcomes; and, estimating the relative costs associated with the object.

Formative evaluation includes several evaluation types:

• needs assessment determines who needs the program, how great the need is, and what might work to meet the need

• evaluability assessment determines whether an evaluation is feasible and how stakeholders can help shape its usefulness

• structured conceptualization helps stakeholders define the program or technology, the target population, and the possible outcomes

• implementation evaluation monitors the fidelity of the program or technology delivery

• process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures

Measurement

• Meaning– Measurement is the process by which the developed

abilities of the people are expressed on quantitative form.

– My measuring the content, skill and the results of abilities are expressed in numbers, scores, percentage and average are used to change the method according to their achievement.

– According to “Campbell” Measurement is the assignment of numerals to objects or events according to rules.

Measurement

• Definitions– Measurement means the description of the

data in terms of numbers and this, in turn, means taking advantage of the many benefits that operate with the numbers and mathematical provide thinking

» J.P.Guilford

– Measurement is assignment of numerals to objects or events according to certain rules is called measurement

» Campbell

Types of measurement proposed by Stevens

• The definition of measurement was purportedly broadened by Stanely S. Stevens.[2] He defined types of measurements to include nominal, ordinal, interval, and ratio. In practice, this scheme is used mainly in the social sciences but even there its use is controversial because it includes definitions that do not meet the more strict requirements of the classical theory and additive conjoint measurement. However, the classifications of interval and ratio level measurement are not controversial.

• Nominal: Discrete data which represent group membership to a category which does not have an underlying numerical value. Examples include ethnicity, color, pattern, soil type, media type, license plate numbers, football jersey numbers, etc. May also be dichotomous such as present/absent, male/female, live/dead

• Ordinal: Includes variables that can be ordered but for which there is no zero point and no exact numerical value. Examples: preference ranks (Thurstone rating scale), Mohs hardness scale, movie ratings, shirt sizes (S,M,L,XL), and college rankings. Also includes the Likert scale used in surveys – strongly agree, agree, undecided, disagree, strongly disagree. Distances between each ordered category are not necessarily the same (a four star movie isn't necessarily just "twice" as good as a two star movie).

• Interval: Describes the distance between two values but a ratio is not relevant. A numerical scale with an arbitrary zero point. Most common examples Celsius and Fahrenheit. Some consider indexes such as IQ to be interval measurements whereas others consider them only counts. Interval-level measurements can be obtained through application of the Rasch model.

• Ratio: This is what is most commonly associated with measurements in the physical sciences. The zero value is not arbitrary and units are uniform. This is the only measurement type where ratio comparisons are meaningful. Examples include weight, speed, volume, etc.

Measurement & Evaluation

The term “Measurement” and “Evaluation” are often used interchangeably. However, in psychological, sociological and educational researches, these two terms connote two different meanings. “Measurement” refers to the process of assigning numerals to events, objects, etc

Definitions of Evaluation

• Probably the most frequently given definition is:

• Evaluation is the systematic assessment of the worth or merit of some object

The Goals of Evaluation• The generic goal of most evaluations is to provide "useful feedback"

to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies.

• Most often, feedback is perceived as "useful" if it aids in decision-making. But the relationship between an evaluation and its impact is not a simple one -- studies that seem critical sometimes fail to influence short-term decisions, and studies that initially seem to have no influence can have a delayed impact when more congenial conditions arise.

• Despite this, there is broad consensus that the major goal of evaluation should be to influence decision-making or policy formulation through the provision of empirically-driven feedback.

Types of Evaluation

– There are many different types of evaluations depending on the object being evaluated and the purpose of the evaluation.

• Formative Evaluation

• Summative Evaluation

• Norm Based Evaluation

• Criterion based evaluation

Formative Evaluation

• Formative evaluation is a type of evaluation which has the purpose of improving programmes. It goes under other names such as developmental evaluation and implementation evaluation. It can be contrasted with other types of evaluation which have other purposes, in particular process evaluation and outcome evaluation. An example of this is its use in instructional design to assess ongoing projects during their construction to implement improvements. Formative evaluation can use any of the techniques which are used in other types of evaluation: surveys, interviews, data collection and experiments (where these are used to examine the outcomes of pilot projects).

Formative Evaluation

• Formative evaluation is done with a small group of people to "test run" various aspects of instructional materials. For example, you might ask a friend to look over your web pages to see if they are graphically pleasing, if there are errors you've missed, if it has navigational problems. It's like having someone look over your shoulder during the development phase to help you catch things that you miss, but a fresh set of eye might not. At times, you might need to have this help from a target audience. For example, if you're designing learning materials for third graders, you should have a third grader as part of your Formative Evaluation.

Formative Evaluation

• Definition Formative evaluation is a method of judging the worth of a program while the program activities are forming or happening. Formative evaluation focuses on the process (Bhola 1990).

• Examples Here are some examples of formative evaluation: – Testing the arrangement of lessons in a primer before

its publication – Collecting continuous feedback from participants in a

program in order to revise the program as needed

Definitions of formative Evaluation

• Scriven, (1991)• Formative evaluation is typically conducted during the development or

improvement of a program or product (or person, and so on) and it is conducted,often more than once, for in-house staff of the program with the intent to improve. The reports normally remain in-house; but serious formative evaluation may be done by an internal or an external evaluator or preferably, a combination; of course, many program staff are, in an informal sense, constantly doing formative evaluation.

• Weston, Mc Alpine, and Bordonaro, (1995)• The purpose of formative evaluation is to validate or ensure that the goals of the

instruction are being achieved and to improve the instruction, if necessary, by means of identification and subsequent remediation of problematic aspects.

• Worthen, Sanders, and Fitzpatrick, (1997)• Formative evaluation is conducted to provide program staff evaluative information

useful in improving the program. • Robert Stakes• "When the cook tastes the soup, that’s formative; when the guests taste the soup,

that’s summative."

Summative Evaluation

• Definition Summative evaluation is a method of judging the worth of a program at the end of the program activities. The focus is on the outcome (Bhola 1990).Examples Here are some examples of summative evaluation: 

• Determining attitudes and achievement related to using a primer after it has been used in a training course

• Collecting data on the impact of a program operating in a community for a period of time

Summative evaluation can also be subdivided:

• outcome evaluations investigate whether the program or technology caused demonstrable effects on specifically defined target outcomes

• impact evaluation is broader and assesses the overall or net effects -- intended or unintended -- of the program or technology as a whole

• cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs and values

• secondary analysis reexamines existing data to address new questions or use methods not previously employed

• meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question