EVALUATIONIN

ORAL HEALTHGuided by :Dr Mahmood Moothedath

Dr Aseela Ahmed Submitted by :

Ramshiya P M Final year Part-

1

CONTENTS INTRODUCTION DEFINITION CRITERIA USED IN THE EVALUATION OF DENTAL SERVICES -EFFECTIVENESS -EFFICIENCY -APPROPRIATENESS -ADEQUACY -RELEVANCE -ACCESSIBILITY -ACCEPTABILITY PUROSE OF EVALUATION TYPES OF EVALUATION -FORMATIVE EVALUATION -SUMMATIVE EVALUATION -RELEVANCE EVALUATION -PROCESS EVALUATION -EFFECTIVENESS EVALUATION -IMPACT EVALUATION -EFFICIENCY EVALUATION GUIDELINES TO BE FOLLOWED BY THE EVALUATOR GENERAL STEPS OF EVALUATION CONCLUSION REFERENCES

INTRODUCTION

Health services have become complex. There has been a growing concern about their functioning in both developed and developing countries. Questions are raised about the quality of medical care, utilization and coverage of health services, and improvement in the health status of the recipients. An evaluation study addresses itself to these issues.

DEFINITION“Evaluation measures the degree to which objectives and targets are fulfilled

and the quality of results obtained. It measures the productivity of available resources in achieving clearly defined objectives. It measures how much output or cost effectiveness is achieved. It makes possible the reallocation of priorities and of resources on the basis of changing health needs.”

-WHO 1967

Evaluation is defined as the collection and analysis of information to determine program performance. It is mostly concerned with final outcome and the factors associated with it.

CRITERIA USED IN THE EVALUATION OF DENTAL SERVICES (WHO 1972)

Effectiveness

Efficiency

Appropriateness

Adequacy

Others

EFFECTIVENESS

• It has been defined as “the ratio between the achievement of the program activity and the desired level which, during the planning process, the planners had proposed would result from the program” –WHO 1974

• Three variables are useful in evaluating effectiveness: 1) Resources: are evaluated by dividing the actual

expenditure of resources by the planned expenditure. 2) Activities: are evaluated by dividing actual performance

by planned performance. 3) Objectives : are evaluated by dividing actual attainment

of objectives by planned attainment.

EFFICIENCY

• It has been defined as “the result that might be achieved through expenditure of a specific amount resources and the result that might be achieved through a minimum of expenditure”-WHO 1974

• It is a measure of resources spent(money,men,material and time) in the process of providing a health care program.

• It expresses the relationship between the effect obtained and the resources spent.

APPROPRIATENESS

• The appropriateness of the program will be judged by lay decision makers. They will weigh up whether the problem defined by the program personnel is a problem for the community.

• The appropriateness of the strategy of the program should be evaluated because even though a program is effective it may not be appropriate.

• Evaluation of appropriateness can be carried out at two levels: 1)whether the aims and objectives of the program are

appropriate. 2)whether the strategy of the program is appropriate.

ADEQUACY

• A measure of adequacy is the extent to which the population in need was covered by the services or the extent to which the services covered the various aspects of the underlying problem.

• It should be distinguished from effectiveness, which is concerned only with result of the program in those who were covered by it.

• It also implies that sufficient attention has been paid to certain previously determined courses of action.

• For example, the staff allocated to a certain program may be described as inadequate if sufficient attention was not paid to the quantum of workload and target to be achieved.

RELEVANCE

• Relevance or requisiteness relates to the appropriateness of the service, whether it is needed at all.If there is no need, the service can hardly be of any value.

• For example, vaccination against smallpox is now irrelevant because the disease no longer exists.

ACCESSIBILITY

• It is the proportion of the given population that can be expected to use a specified facility,service,etc.

• The barriers to accessibility -physical (eg:-distance,travel,time) -economic (eg:- travel cost,free charge) -social and cultural (eg:- caste or language barrier)

ACCEPTABILITY• The service provided may be accessible but not acceptable to all , eg- male

sterlisation, screening for rectal cancer.

PURPOSE OF EVALUATION

Evaluation allows us to: • Measure the progress of each activity• Measure the effectiveness of each activity• Identify problems in carrrying out the activities• Plan revision and modification• Justify the costs of administering the

programme and if necessary, justify seeking additional funds

TYPES OF EVALUATION

•FORMATIVE EVALUATION

•SUMMATIVE EVALUATION

•OTHER TYPES

FORMATIVE EVALUATION • It refers to the internal evaluation of a program. • It is an examination of the activities of a program, as

they are taking place.• It is usually carried out to aid in the development of

program in its early phase.• Used primarily by the program developers to check

whether they are workable or whether changes should be made to improve program activities.

• Eg: A fluoride rinse program is initiated at a neighborhood health center in which paraprofessionals are trained to administer three types of fluoride under a strict sequence of procedures.

After 3 days of operation,the work of paraprofessionals is observed to determine the extent to which the strict sequence of procedures is being adhered to. The observation and determination of correct or incorrect procedure sequence provide an example of examining the activities of a program as they are occuring.

If the sequence is incorrect, formative evaluation allows the program to make remedial changes at that point, without waiting until the end of the program.

SUMMATIVE EVALUATION

• It judges the merit or the worth of a program after it has been in operation.It is an attempt to determine whether a fully operational program is meeting the goals for which it was developed.

• Aimed at program decision makes, who will decide about a programs continuation or termination and also at decision makers of other programs who might be considering adoption of the program.

OTHER TYPES OF EVALUATION

• RELEVANCE EVALUATION• PROCESS EVALUATION• EFFECTIVENESS EVALUATION• IMPACT EVALUATION• EFFICIENCY EVALUATION

RELEVANCE EVALUATION• It refers to activities designed to determine whether the program is needed or

whether the program is targeting its efforts at the individuals in need.• The seven key questions around which relevance evaluations may be organized are:

1) what problem does the program address? 2) How adequate is the definition of the problem? 3)What is

the level of need for services associated with the problem? 4) How accurate is the information about the problem? 5)How adequate is the definition of the program?

6) Is the program appropriate to the defined problem? 7) Are those identified to be in need of services receiving the problem?

PROCESS EVALUATION • It refers to efforts made to assess the extent to which program

implementation complies with the program plan. • Similar to formative evaluation.• Helpful in making early adjustments of the program and in making

decisions concerning program continuation and expansion. • Questions around which a process evaluation may be organized include: 1)Are the required personal, equipment and financial resources in place at

the times and locations necessary to meet program need? 2)Do program activities clearly conform to the original plan? 3)Are there any unanticipated factors influencing program

implementation? 4)Are the various activities or components of the program being provided

to all program targets with uniform quality & quantity?

• This type of evaluation is used early in the implementation of the health program.

• Four types of recommendation might logically follow from a process evaluation :

1) Terminate the project 2) Reorganize the project 3) ”Fine tune” the project 4) Proceed with the project as it has been

implemented

EFECTIVENESS EVALUATION

• It refers to whether the program results meet predetermined objectives. Here, the emphasis is on immediate outcomes of program activities and whether these outcomes meet the activities specified by the program planners.

• Questions answered by an effectiveness evaluation include: 1.Did the program meet its stated objectives? 2. Were the program provides satisfied with the effects of

program activities? 3.Were the program beneficiaries satisfied with the effect of

program activities? 4.Is the problem reduced or eliminated as a result of the

program?

IMPACT EVALUATION

• It refers to the long term outcomes of the program. It considers whether the intervention had any long lasting effects on the ultimate problems that the program is intended to remedy.

• It is an expression of the overall effect of a program on health status and socioeconomic development.

EFFICIENCY EVALUATION

• It attempts to relate the results obtained from a specific program to the resources expended to maintain the program.

• The types of question central to efficiency evaluation include:

1) Do the program benefits exceeds the cost incurred?

2) Are program benefits more or less costly per unit of outcome when compared to other programs designed to achieve the same objectives?

GUIDELINES TO BE FOLLOWED BY THE EVALUATOR

1) Explore the motivations for evaluation.2) The original objectives of the program and its rationale and procedures must be uppermost in the evaluator’s mind.3) Evaluation should be continuous.4) An assessment must be made to check whether it is

possible to make changes in the program, if the evaluation sugests that these are desirable.

5) The results should be well presented and widely publicized.

GENERAL STEPS OF EVALUATION1)Determine what is to be

evaluated

2)Establish standards and criteria

3)Plan the methodology to be applied

4)Gather information

5)Analyze the results

6)Take action

7)Reevaluate

1)Determine what is to be evaluated

Generally speaking there are 3 types of evaluation: a) Evaluation of structure: This is evaluation of whether facilities, equipment,manpower

and organisation meet a standard accepted by experts as good. b) Evaluation of process The process of medical care includes the problems of

recognition,diagnostic procedures, treatment and clinical management, care and prevention.

The way in which the various activities of the program is carried out is evaluated by comparing with a predetermined standard.

• An objective and systematic way of evaluating the physician (or nurse ) performance is known as “ Medical (or nursing) Audit “ c)Evaluation of outcome This is concerned with the end results, that

is, whether persons using health services experience measurable benefits such as improved survival or reduced disability.

2) Establishment of standards and criteria

• Standards and criteria must be established,so that they can be compared against.

• It is necessary to determine how well the desired objectives have been attained.

• Naturally such standards are a prerequisite for evaluation.• Standards and criteria must be developed in accordance with the focus of

evaluation. Structural criteria – eg Physical facilities and equipment. Process criteria –eg Every school going child should receive dental check up

once in 6months. Outcome criteria –eg alteration in patient health status (cured,

death,disability) or behavior resulting from health care(satisfaction, dissatisfaction); or the educational process (eg- cessation of smoking, acceptance of a small family norm) etc.

3) Planning the methodology

A format must be prepared for gathering the desired information.

Standards and criteria must be included at the planning stage.

4) Gathering information • Evaluation requires collection of data or information.

The type of information required may include political, cultural, economic, environmental and administrative factors influencing the health situation.

• It may also concern health and socio-economic policies and programs well as the extent,scope and use of health system, services and institution.

• The type and amount of data required will depend on the purpose and

use of the evaluation.

5) Analysis of results The analysis and interpretation of data and

feedback to all individuals concerned should take place within the time feasible, once information has been gathered.

In addition, opportunities should be provided for discussing the evaluation results.

6) Taking action

• For evaluation to be truly productive, emphasis should be placed on actions designed to support,strenghthen or otherwise modify the services involved.

• This may also call for shifting priorities,revising objectives, or development of new programs or services to meet unidentified needs.

7) Re-evaluation

Evaluation is an on-going process aimed mainly at rendering health activities more relevant,more efficient and more effective.

CONCLUSION

Evaluation of any health program is very important to find out how well it has worked,how much and in what respect, people have been benefited by this.

Planning and evaluation must be viewed as a continuous process, leading to continual modification of both objectives and plans.

During evaluation of the program, all the problems should be identified and solved. By this type of planning and management, the future programs can be improved and made more effective and beneficial.

REFERENCES

• Essentials of preventive and community dentistry- Soben Peter

• PARK’s textbook of preventive and social medicine

• Textbook of community dentistry- Satish Chandra,Shaleen Chandra

• Jong’s community dental health- Gluck Morgansten

TH

AN KY

OU