velema & cornielje 2003 reflect before you act

REVIEW

Reflect before you act: providing structure to theevaluation of rehabilitation programmes

JOHAN P. VELEMA{* and HUIB CORNIELJE{{ Evaluation & Monitoring Service, The Leprosy Mission International, Apeldoorn,

The Netherlands{ Hogeschool Leiden, Department of Public Health, Leiden, The Netherlands

Accepted for publication: June 2003

Abstract

Purpose: This paper is concerned with understanding andevaluating potentially diverse rehabilitation programmes. Ithelps evaluators and programme managers to focus attentionon specific aspects of the rehabilitation process and selectevaluation questions relevant to each.Method: Distinction is made between the rehabilitationprogramme itself, the programme environment and therelationships between the two. For each of these areas,evaluation questions have been formulated. For servicesoffered to individual clients, questions address whether thestatus of clients has improved, what interventions are offeredand who benefit from them, the relationships between theservice providers and the clients, and who may be involved inthe rehabilitation process besides the client. To assess theprogramme environment, questions address the epidemiologyof disability, the resources available to persons with disabil-ities, the inclusiveness of education and employment and anumber of eco-social variables. Relationships between theprogramme and its environment concern the support of thecommunity for the programme, the way the programme seeksto influence the community, the referral of clients to otherservices available in the community and the extent to which theprogramme is a learning organization.Results: Lists of evaluation questions are presented fromwhich the evaluator can select those most relevant to theprogramme to be evaluated. This provides a framework for theevaluation and for the information to be gathered. Rather thanproviding a blue print, this framework permits flexibility toadapt to the specific situation of the programme to beevaluated.Conclusion: This paper presents a useful guideline thatstimulates the thinking of those preparing for the evaluationof rehabilitation programmes.

Introduction

Organizations active in the field of Community BasedRehabilitation (CBR) development rightfully raise ques-tions about the appropriateness, relevance, effectivenessand efficiency of such programmes. Unfortunately, firmevidence for claims made by those in favour of CBR is,to a large extent, missing. To provide programmes witha solid basis for planning and management, systematicinformation gathering is needed about: (1) the statusof individual clients, (2) the services offered, (3) theprogramme environment, and (4) the relationshipsbetween the clients, the programme and the programmeenvironment. Particularly information about theprogramme environment should be gathered evenduring the planning stages of CBR programmes (base-line information).Programme evaluation is a way of systematically

gathering information with the aim of comparing thesituation in a given programme to agreed standards, toprogramme objectives formulated earlier, to the situa-tion at an earlier point in time or to the situation inother programmes.1 – 4 Programme evaluation is usefulto planners as it helps them to see whether their expecta-tions are fulfilled and to adjust strategies accordingly.5, 6

It also helps to measure and promote quality of theservices provided. In other words, evaluation assesseswhether the right things are being done in the rightway. As such, evaluation is a requirement for plannersand managers in their search for providing quality care.Given the broad spectrum of rehabilitation

programmes, we define rehabilitation as both the equal-ization of opportunities and integration of people withdisabilities in society and the interventions that may takeplace at the individual level. The first aspect of rehabili-tation has a collective (and broader) notion focusing onsociety at large, the programme environment (see figure

* Author for correspondence; Johan P. Velema, Evaluation &Monitoring Service, The Leprosy Mission International, P.O.Box 902, 7301 BD, Apeldoorn, The Netherlands.e-mail: [email protected]

DISABILITY AND REHABILITATION, 2003; VOL. 25, NO. 22, 1252–1264

Disability and Rehabilitation ISSN 0963–8288 print/ISSN 1464–5165 online # 2003 Taylor & Francis Ltdhttp://www.tandf.co.uk/journals

DOI: 10.1080/09638280310001599970

1A), indicating that the other individually oriented inter-ventions do not take place in a vacuum, but form anintrinsic part of a broader, more overall goal that aimsfor the development towards a more just and unrest-ricted society. Important exchanges take place betweenthe programme and the environment of information,influence and resources. The second feature of rehabili-tation, the interventions that may take place at the indi-vidual level within a rehabilitation programme, isvisualized in figure 1B. It shows the four dimensionsof rehabilitation we distinguished earlier:7, 8 the servicesoffered to clients, the outcome expected, the power rela-

tions involved and the involvement of others besides theperson with disability him- or herself. Services mayrange from one or a limited number of interventions,e.g., surgery, counselling, training, to a broaderapproach in which the rehabilitation programme actsas a broker, assisting clients and their relatives to findwhat they need in other existing programmes, resourcesand facilities that are not necessarily explicitly focusedon the rehabilitation of disabled people; Outcomes interms of quality of life, may range from normalizationof physical or mental function to a fuller integration insociety including the capacity to generate income; Power

A

B

Figure 1 (A) Programme environment in relation to rehabilitation programme. (B) Individual rehabilitation in relation torehabilitation programme.

Evaluation of rehabilitation programmes

1253

is defined in terms of the degree of influence a client hasover the choice of rehabilitation interventions; Involve-ment of others may range from a focus on the individualclient to a commitment to principles of communityparticipation.

Thus within rehabilitation there are both the focus onthe whole community to try and make society moreinclusive on all levels (such as physical accessibility,inclusive education, better services, respite care, andchanged attitudes) and the individual client’s needs thatshould be addressed (such as restoring function at theperson level, training and income generating projects).

CBR programmes differ in terms of organization,structure and philosophy as well as scope. The spectrumranges from programmes that are basically (single)service oriented (which may fit a more individual ormedical paradigm) to programmes that are moresocio-politically oriented, working within a developmen-tal and social paradigm. We believe that, given the localeco-social context, justification for each type ofprogramme could be given. Therefore, we have beencareful to not make any judgement about appropriate-ness and relevance of programmes. For instance, giventhe political context it may be counterproductive toactively promote a more socio-political model of rehabi-litation whereby emphasis is placed upon issues of indi-vidual rights. Or it may be hard to justify solemnlyinclusive education in a context of absolute povertywhereby even the non-disabled youth cannot participatein quality education. Inclusive education becomes mean-ingless in such contexts and turns out to be good rheto-ric only; the special school, on the contrary, may formthe most appropriate approach for the time being thatdeserves to be nurtured.

The description of the rehabilitation process givenabove has guided our thinking in defining informationthat might serve to evaluate rehabilitation programmesand, as such, provide programme managers with insightto improve overseeing their programmes. Our funda-mental premise is that the questions appropriately askedin one type of rehabilitation programme do not necessa-rily apply to another. For example, questions asked in aprogramme that offers surgical correction of claw hands(a complication of leprosy) do not apply to aprogramme offering loans for income-generating activ-ities. Similarly, indicators of the involvement of thecommunity do not apply to a programme where no suchinvolvement is sought.

The development of uniform disability indicators andtheir incorporation in evaluation studies is an endeavoursome researchers consider to be of crucial importance tomove towards more evidence-based practices.9, 10 We

would argue, however, that the great variety ofprogrammes and programme goals do not allow foruniform indicator development. Indicators should bederived from each programme’s own goals and objec-tives. Of course, programme planners and evaluatorsmay select from the literature those indicators that arerelevant and appropriate to their particular programme.We do not aim to describe the steps involved in the

conduct of a programme evaluation; neither do wepresent detailed methodologies for gathering quantita-tive or qualitative information. Although we havesuggested sources where and ways in which informationmight be collected, we are well aware of their limitationsand believe there is a lot of scope for expanding therepertoire of information gathering techniques accord-ing to the skill and creativity of the evaluators.However, we do aim to help the evaluator focus his/her attention on specific aspects of the rehabilitationprocess and pose evaluation questions relevant to each.The material presented here was developed in the

context of the psychological and socio-economic rehabi-litation of persons affected by leprosy.11 We believe,however, that it applies to all rehabilitation, regardlessof the type of disability addressed or the level of socio-economic development of a national state. While CBRmainly evolved within the context of so-called lesser-developed countries, we share the opinion that the prin-ciples of CBR are also applicable to the praxis of reha-bilitation in middle income and high incomecountries.12 – 14 The framework of evaluation questionssuggested here was developed with a wide range ofprogramme types in mind.7, 8 It is, therefore, our viewthat it is equally relevant to CBR programmes in lowincome countries as to the current orthodox rehabilita-tion practice in middle income and high income coun-tries.

Questions on the programme environment

The first set of questions concerns the programmeenvironment. These inquire, firstly, into the size ofthe problem of disability in the population served bythe programme and its main causes (table 1A).(Disability is used here as the umbrella term encom-passing the three levels of impairment, activity limita-tion and participation restriction). In one populationpoliomyelitis may cause most impairment whereas inanother population blindness may be most prevalent,caused, for example, by onchocerchiasis. Knowledgeof the main determinants of disability in a given regionwill provide pointers to the kind of interventions thatwill be needed.

J. P. Velema and H. Cornielje

1254

Other questions, also presented in table 1, refer tothe environmental factors (both negative and positive)that contribute towards the experience and develop-ment of disability. The degree of participation restric-tion resulting from a given impairment may be quitedifferent in one context or another; impairment canremain just an impairment or become a significantparticipation restriction merely on the basis of theenvironment. Environmental factors include, forinstance, attitudes and practises towards people withdisabilities (PWDs), the inclusion of PWDs in schools

and employment, but also the existence of supportinginstitutions and facilities, and the enforcement of lawsand regulations in favour of people with disabilities.15

Boyce et al.9 list indicators for many of the issuesaddressed in table 1.

These environmental factors may have aggravatingeffects on the crude or specific disability incidence andprevalence rates.16 Knowledge of the occurrence ofdisability and its determinants together with knowledgeof the environmental factors that interact with them andlead to activity limitation and participation restriction

Table 1 Assessment of the programme environment

Focus Key questions Indicators Sources of information

A. Descriptiveepidemiology

Who form the target group(s)? What isthe magnitude of disablement?

(1) Number of persons with disablementof more than 6 months duration.

(Sub-) National Statistics Projectrecords

What are the domains of disablement?What are specific local determinantsof disablement?

(2) Prevalence of disablement (by sexand age) (3) Size of target grouprequiring rehabilitation (services)=number of people with restrictedparticipation in the differentdomains as defined by the ICF

(4) Incidence and prevalence of disablingdiseases

Information systemsKey informants

B. Resourcemobilization

Who are the stakeholders? Whatrehabilitation resources do exist?

(1) Stakeholders in rehabilitationprogramme

(National) information suchas records,publications, (annual) reports

What is the scope of the different (2) Number and type of rehabilitation Stakeholders analysisprogrammes? programmes Project reports, e.g., mission

(3) Existence of DPOs statements, organo-grams etc.(4) Type of supporting institutions Key informants(5) Rehabilitation budget per annum in

project area(6) (Type of) rehabilitation staff per 1000

target population 10

(7) (Recent) rehabilitation servicecoverage (incl. all rehabilitationprogrammes) by age/sex/disablementdomain

C. Socio-economicparticipation

To what extent do PWDs participatein the economic activities of thecommunity?

(1) Overall unemployment rate by age/sex(2) Unemployment rate among PWDs by

age/sex(3) Equality of opportunities(4) Overall literacy rates by age/sex(5) Literacy rates of PWDs by age/sex(6) Overall income per household(7) Income per household of PWD

(National) information such asrecords, statistics, publications

(National) statistics on labour/education/income

D. Eco-social How ‘PWD-friendly’ is the (1) (Implementation of) existing laws/ Review of laws and regulationsvariables environment? regulations/policies on disability (specifically the application of the

(2) Availability of information ondisability rights issues

UN Standard Rules19)Survey reports

(3) Local attitudes and practices towards (National) information suchas records,disablement publications

(4) Acknowledgement of PWDs in Literaturecommunity Key informants

(5) Local coping mechanisms withdisablement

(6) Discriminatory barriers and practices(7) Incidence of human rights abuses

(e.g. sexual, physical)


1255

will provide a holistic picture of the situation which willpermit setting priorities for rehabilitation programmes.17

The resources available in the population to meet theexisting rehabilitation needs (table 1B) have beendescribed separately as they ideally constitute a positivefactor towards overcoming activity limitations andrestrictions in participation. These may include the sizeof budgets allocated by government, national and inter-national NGOs to rehabilitation work; a list of the orga-nizations, both large and small, involved in rehabilitationwork; the type of work they do; and the numbers ofpeople benefiting from these services. The latter can becombined with estimates of the total numbers of disabledpersons to give a measure of coverage.18

Other information about resources relates tomanpower (i.e., numbers and types of staff) and theirlevel of training and specialities working in the field ofrehabilitation either directly or indirectly through otherinstitutions (e.g., community development projects)which specifically include disabled persons in their activ-ities.

Further details may include whether personsaffected by disability have organized themselves toform Disabled Peoples Organization (DPO) and whothe various stakeholders in rehabilitation programmesare. Common stakeholders are the beneficiaries, theirrelatives, the government, NGOs, professionals, andso on. Knowledge of these stakeholders and their poli-cies and/or attitudes will give an understanding of thesupporters and detractors of the programme beingevaluated. Insight into the powers and influences ofvarious stakeholders will provide insight into the feasi-bility, sustainability and acceptance of the programme.This is of strategic importance for future develop-ments.

Another area of attention is the function of peoplewith disabilities (PWDs) in schools, in the workplaceand in the marketplace (table 1C). Do they have accessto schools, to the labour market, to public facilities suchas transport and buildings? Do they have equal opportu-nities regarding education, training and employment?Are there special provisions to actively promote equalopportunities for PWDs? The level of inclusiveness ofPWDs will often be reflected in the existence of lawsand regulations to promote and protect them (table1D). Reference should be made to the United NationsStandard Rules19—endorsed by most national states—and the practical application of these rules in daily life.Other questions in this context concern common beliefs,attitudes and practices in response to disability and towhat extent up to date information about disabilitythemes is readily available.

The level of detail required regarding the programmeenvironment depends on the purpose of the evaluation.Often the focus will be on the progress, and efficiency ofthe programme, so that a general impression is enough.If, however, the focus is on developing new strategiesand activities, or evaluating how the programme hascontributed to the overall situation, information onthe needs of PWDs in the community cannot be omitted.The more the evaluation emphasizes the connection ofprogramme activities to its context, the more knowledgeof the programme environment is required.

The relationship between the programme and its

environment

Where the aim of a programme is advocacy i.e., not toimprove the situation of PWDs directly, but rather tochange the way the community as a whole perceivesand responds to disability, the outcome indicators areessentially those on socio-economic status of PWDsand the inclusiveness of society, already presented intable 1. The focus here should be on changes in theseindicators over the period evaluated.1 More specificquestions and indicators related to advocacy can befound in table 2A. Since advocacy work is not alwaysimmediately successful, it may be important to includeoutput indicators, which prove that the work has beendone, even if the effect is not (yet) apparent. Anotherindicator of successful advocacy is the strength of therelationships between the programme and (local) autho-rities.A programme that does not really offer services but

rather mediates between clients and services offered byothers, should be evaluated on the effectiveness withwhich it performs this mediation role (see table 2B). Thisrelates to the strength of the relationships theprogramme maintains with other organizations, whetherclients do indeed end up receiving the services for whichthey were referred and, as before, the criteria used inselecting clients for referral.Other signs of community involvement (table 2C) are

the support, both material and immaterial, given to andthe extent to which people think of the programme as‘their’ programme. At best, however, involvement maybe experienced in the sense of real inclusion: i.e., as partof the everyday life of the community, and not as ‘their’special effort for ‘these poor disabled people in ourvillage’. Indicators of such development include atti-tudes and behaviour towards PWDs, but also generalaccessibility of public buildings (such as clinics, tribaloffices and churches) and the existing practices towardsinclusion of disabled children in local schools and


1256

disabled adults in employment (see indicatorsmentioned in table 2A).

A number of questions are directed at the way theservices are nurtured so that they continue to be updatedin accordance with new insights, problems identified andneeds encountered (table 2D). Evaluators need to assessif programmes work in isolation or not and need toinvestigate linkages, modes of bench-marking, consulta-tion and ongoing and continuous education. Contactwith colleagues in other programmes, having access toup-to-date information and continuing education ofthe staff are all elements of such a strategy.

Services offered

Evaluation of services per se should include adescription of the type of services, the proposed and

the actual beneficiaries. First of all, there is the impor-tant question as to whether people make use of theservice being offered. Furthermore, a programme willusually select clients to enrol in its schemes. Thecriteria implicitly or explicitly employed in this selec-tion are important and can often be deduced fromthe characteristics of the clients who are making useof the service at the time of the visit. It is relevantto ask which criteria were actually used, comparedto the ones stated in programme documents andwhether important groups of clients may perhapsnot have access to the services because of either theformal or informal criteria being used.

Evaluation of programmes that offer specific servicesshould include a technical component, assessing whetherthe service meets technical standards. This may varyfrom performing surgery in accordance with the latest

Table 2 Relationships between programme and environment


A. Advocacy Is the project addressing disability rights Existence of (political) lobby, campaigns Documents producedissues? on disability issues Media reports/Newspaper clippings

Is the project influencing attitudes/ Success in getting attention for issues Progress reportsbehaviour in community towards relevant for PWDs Key informantsPWDs? Liaising of CBR project with (local)

What is the result of advocacy work? authoritiesChanges in socio-economic participation(table 1C)

Changes in eco-social variables(table 1D)

B. Referral Are clients referred to other service Number and nature of services available Project documentsproviders? to clients through referral Client records

What services can effectively be made Outside services to which at least one Survey dataavailable to clients through referral? client is referred Interviews with beneficiaries, staff,

Who benefit from referral? Percentage of those referred who obtain family, community, leaders etc.the desired service Follow-up of referred clients

Number and profile# of PWDs referredto other programmes

How does this compare to the profile ofthe target group?

C. Community To what extent does the community Material support given to programme Interviews with clients, staff, family,support identify with the project? Political support given to programme community leaders etc.

Involvement of community members in (Participatory) observationsDPOs

Role of community members inprogramme management

Community ownership of projectSatisfaction with programme

D. Influx of new Does the programme stay in touch with Linkages with other rehabilitation Interviews with staffideas and current developments in the programmes Photographs or other recordsadvice professional field? Is professional literature available to Observations

programme staff?Are materials used for IEC periodicallyrenewed?

Do external consultant come to theprogramme?

#Can think of age, sex, disability status, socio-economic situation, level of education etc.


1257

medical insights, to giving training in welding by meansof up-to-date machinery (i.e., current in the programmeenvironment). It also implies that new businesses shouldbe set up only if there is reasonable certainty about themarketability of the product. An evaluation team shouldinclude at least one member who has the expertise tomake these assessments, which are specific to the typeof service(s) being offered.

Another aspect of the services is related to the exis-tence of information systems and the application of suchsystems for managerial purposes (such as monitoring,evaluation and supervision and control). Lastly, it isimportant to assess staff competence and (related) levelsof training.

Questions on programme outcome

There are many actions that may be taken to helppeople increase their participation in society. Theymay address the individual, e.g., physiotherapy, skillstraining and so on, or may focus on the environmentso that function is improved even if the capacity of theindividual remains unchanged. An example of theformer is teaching leprosy-affected individuals to regu-larly soak their feet to reduce formation of ulcers, thusimproving their capacity to function with anaestheticfeet; an example of the latter is arranging for easy accessto water, placing the basin or the bucket in such a waythat it facilitates the person washing him or her self,or for family and friends to agree on a rota to supportthe person in this task. Practice of self-care (perfor-mance) will often drastically improve, once obstaclesin the environment are removed. Another example isthe wearing of special footwear to protect anaestheticfeet. Footwear provided is often not worn by personswith anaesthetic feet because people around them (theenvironment) see them as a sign that the person hasleprosy. Developing fashionable footwear that providesthe necessary protection or teaching clients to chooseregular shoes in the market that provide adequateprotection can greatly increase the actual use of protec-tive footwear. These adaptations of context-indepen-dent, individual-oriented interventions to the localenvironment are necessary to move from increasingcapacity to increasing actual performance and therebyto fuller participation in society.

Whatever action is undertaken, an expected outcomeof this action will be defined. If no desired outcome hasbeen formulated, it is usually not difficult for the evalua-tor to define this from the work being done and therecords being kept. It may be expected that aprogramme that works with individual clients maintains

individual records of problems identified and steps takento resolve these, as well as periodic assessments to docu-ment progress. At a minimum, assessments at intake andat separation should be made and documented. The typeof information recorded depends very much on the typeof disability and the kind of interventions being offered,whether in terms of restoring function (table 4A) or ofincreasing participation (table 4B).Ideally, rehabilitation workers should follow up

clients for some time after separation from theprogramme to see whether the improvements obtainedwhile the client was under their care are sustained.Follow-up is also important with regard to measuringlevels of participation as the ultimate outcome of rehabi-litation. As an example, there is a general consensusamong staff of vocational training centres in India thattheir success is best measured by the percentage oftrainees in active employment 1 year after graduation.Other measures of improvement include decreased

dependency of the disabled person on others (often rela-tives) or on adaptive devices, and an increased accep-tance of the client in social networks (see table 4B).The above is based on outcomes as defined by the

service provider. Evans et al.20 contrast the outcomesdefined by the service provider with those the clienthas in mind. Divergence between these two will resultin a lack of client satisfaction and may be indicative ofcommunication problems between the service providerand the client.

Locus of power

It is important to define the relationship between theclients and the rehabilitation workers. Are the workersprofessionals who, based on their expertise prescribe acourse of action? The more medical the interventionsoffered, the more likely this type of relationship willoccur. It should, however, be understood that even thesimple provision of a walking aid may be a steppingstone for further integration and participation intosociety. In such a relationship, compliance depends onwhether the client has understood the instructions, hastrust in the workers, is materially able to comply andis motivated to change his situation. It is also importantthat the instructions do not contradict his own beliefsystems. Evaluation should verify that communicationbetween clients and workers is such that compliance ispossible and, secondly, that compliance is achieved.In other programmes, the workers present a range of

options to the client, asking him or her to make aninformed choice between them. In this type of relation-ship, professionals negotiate with clients about their


1258

rehabilitation plan. The client’s capacity to make achoice depends on whether all the implications of eachoption are understood. There will often be the need tothink about it or discuss with others and come back withmore questions. Evaluators should consider the deci-sion-making process and verify that enough space isgiven for thinking through the options and consultingothers, for example relatives.

Once the choice for a particular rehabilitation scenar-io is made, the course of action is relatively fixed and it isof interest to see how much flexibility there is to accom-modate specific wishes of the client.

In programmes where the rehabilitation workersoperate from a more social model of disability, they willhave a stronger focus on action towards alleviating orcombating restriction in participation and make effortsto empower their clients. Thus, individual clients willbe given more opportunity to design their own rehabili-tation scenarios, or at least to play a major role in thatprocess. This implies agreement between the client andthe worker on the desired outcome of the rehabilitationprocess and that, where disagreement arises, there aremechanisms to resolve these. Often, relationships amongclients as well as their relatives and other participants inthe programme will be more important in programmesthat use this approach and often groups of participantswill develop rehabilitation initiatives. This means thatclients, as they are involved in the programme, willgradually begin to develop a shared vision and a senseof ownership. Emergence of leadership among partici-pants is in this context a sign that participants take theirdestiny into their own hands.

Of course, the above has been written with the client’sperspective in mind. From the point of view of the reha-bilitation worker, who may be a doctor, a physiothera-pist, an occupational therapist, a social worker, and soon, the important question is whether the interventionsoffered are those most suited to the client’s needs andmost likely to result in the achievement of the desiredoutcome.20 To evaluate whether rehabilitation workersadhere to professional standards requires an explanationof what those standards are and technical expertise onthe part of the evaluator.

Involving others

It is important to ask whether, besides the client,others are involved in the rehabilitation process. Thismay concern relatives of the client or other communityvolunteers. The extent of their involvement may beassessed by asking questions like these (see table 6):Are they mentioned in the programme documents? Do

programme staff address relatives or others who accom-pany the client? Do staff make a conscious effort toinvolve them? Do they help others to play their role insupport of the PWD?

From the relative’s point of view, one may askwhether they gained any greater knowledge or betterunderstanding of what is happening to the PWD and/or whether they learned skills that make them moreeffective in supporting the PWD.21

Community volunteers—often in the first place theneighbours—may also be involved in supporting PWDs,whether in their homes or in a central meeting place.Training and continued motivation of communityvolunteers for this role is an important condition forthe success of such an approach.

Preparing for an evaluation

In the eyes of many, the visit of the evaluators to theprogramme is the evaluation. We assume, however, thatbefore a site visit a process of communication betweenevaluators and stakeholders in the programme takesplace during which the purpose of the evaluation isdefined, the resources available for the evaluation(including time, people, travel, extent of data collection)are specified and documentation describing theprogramme is made available. Planning documentsusually include statements of objectives, strategiesemployed, resources available etc. Newspaper clippingsand internet sites may present stories about theprogramme and reflect the style of working, the waythe programme is perceived etc. It is during this processthat evaluators can use the framework presented here tobetter understand what kind of rehabilitationprogramme they are going to see and to prepare ques-tions relevant to this kind of programme and to thepurpose of the evaluation. The flow diagrams presentedin figure 2A and B illustrate the steps to be taken.

In the first diagram (figure 2A), addressing the rela-tionship between the programme and its environment,the following steps are described:

(1) For any programme evaluation, informationabout the programme environment will need tobe collected (presented in table 1). The level ofdetail needed depends on the purpose of theevaluation. Often the focus will be on theprogress and efficiency of the programme, sothat a general impression is enough. If, however,the focus is on developing new strategies andactivities or evaluating how the programme hascontributed to the overall situation (relevance),


1259

information on the needs of PWDs in thecommunity cannot be omitted. The more theevaluation emphasizes the connection of pro-gramme activities to its context, the moreknowledge of the programme environment isneeded.

(2) The question needs to be asked as to what extentthe programme takes on an advocacy role—i.e.,

aims specifically (not as a side-effect) to changeattitudes and perceptions in society. If resourcesare allocated to this part of the work, questionsneed to be asked about the activities carried outand the effect that they have. For examples seetable 2A.

(3) If the programme refers clients to servicesoperated by other organizations or programmes,

A B

Figure 2 Steps in preparing for the evaluation of a rehabilitation programme.


1260

Table 3 Services offered


A. Services What is/are the service(s) being offered? Description of intervention Project documents

Client records

B. Utilization Who can benefit from the service(s)? How does one qualify for enrolment#? Interviews with clients and workersof services Who do benefit? How does this compare to the profile of

the target group?How many beneficiaries enter/exit theprogramme each year?

C. Quality How does the project maintain standards Presence of a functioning informationof services of quality? system

Practice of internal evaluationFrequency and quality of training andsupervision

#Can think of age, sex, disability status, socio-economic situation, level of education etc.

Table 4 Outcomes pursued for clients


A. Restoration of What is the outcome of Percentage of PWDs in project to date in each type Client recordsphysical function the interventions? of disablement domain

Percentage of PWDs in project showing in past Interviews with patients1 year improvement in the various domains of Interviews with patients, family,disablement care-givers

Percentage of PWDs in project who have been Follow-up of patients 1 year afterdischarged during past year (=optimal dischargefunction achieved) from therapeutic –medicaltreatment. Grading from:

*status quo*some improvement*full recovery

Satisfaction of PWDs with results of interventionsPercentage of PWDs in project showing decreasingdependence on relatives/or others

Has decreased dependence been sustained 1 yearafter discharge?

B. Social and What is the outcome of Changes in psychological status: Interviews with (groups of)psychological the interventions? ability to articulate vision and hope for the future beneficiarieswell-being expectation regarding integration into society Interviews with (groups of) project

self-esteem, dignity and worth staffself-blame, negative behaviours, withdrawal Interviews with (groups of) family,negative emotions shown: anxiety, fear, community leaders etc.

depression etc. ObservationsChanges in social status (degree of Follow-up of patients 1 year aftermarginalization) in terms of: discharge

employmentwealth/incomeliteracy/educationquality and security of dwellingmarriage prospects7

intensity of social ties*Changes in participation in:community activities (e.g., religious, weddings,

funerals etc.)decision-making in household or community

*Intensity of social ties: Frequency of social contacts and having meaningful relationships as opposed to superficial contact.


1261

Table 5 Relationship between provider and client


A. Compliance Is the client able to comply with the Relevance/appropriateness/ Measures of complianceinstructions given? communicative quality of Observations of client/worker

instructions interactionLevel of comprehending instructions Interviews with clients and workersCompliance with instruction

B. Limited choice Is the client making informed choices Range of options given andconcerning his/her own information regarding each;rehabilitation? opportunity to ask questions

Negotiation with client related toplanning, execution and monitoringof intervention

C. Empowerment How much control do clients have over: Evidence of beneficiary participating inTheir own rehabilitation process? planning and decision-makingThe way the project is run? Awareness by beneficiaries of their own

monitoring indicatorsAre there formal (reporting)mechanisms available regardingmalpractice?

Is there local leadership among PWDs?Ability of beneficiaries to articulate thepartners’ vision and agree with it.

Sense of accountabilityOwnership of project activities

Table 6 Commitment to involve others


A. Relatives How are relatives/caregivers involved in Are relatives mentioned in project Project documentsthe rehabilitation process? documents?

How do they benefit from the project Attitudes of staff towards them Interviews with (groups of) staff,activities? Involvement/role in project beneficiaries, relatives

managementInvolvement of relatives in DPOsChanges in:Level of knowledge aboutproblems of disabled relativeSkills to help disabled relativeCoping practices

Material benefits receivedSatisfaction with programme

B. Neighbours How are community members involved Number of community volunteers who Interviews with beneficiaries, staff,in the rehabilitation process? provide support to PWD in his/her family, community leaders etc

To what extent does the community home (Participatory) observationsidentify with the project? Number of community volunteers

working in projectTraining received by communityvolunteers and means employed tomotivate them

Support given to CBR projectInvolvement of community members inDPOs

Community ownership of project


1262

questions about the extent and success of thisreferral system should be asked. Examples canbe found in table 2B.

(4) The way the programme is supported by thecommunity in which it operates is another areawhere information needs to be collected (cf. table2C).

(5) The extent to which the programme is a learningorganization needs to be assessed as it providesinsight into its potential for the future (cf. table2D).The second diagram (figure 2B) suggests steps inthe evaluation of services offered to individualclients:

(6) The services offered need to be documented,including the size of the operation, the targetgroups, the number of people making use of theservices, their profile, and the way quality ismaintained (table 3).

(7) Next, questions concerning the types of out-comes that the programme aims for through theservices it offers and the extent to which these areachieved. The criteria for progress and successwill differ whether the programme offers servicesto restore physical function or services toimprove socio-economic well-being and integra-tion in the community (cf. table 4A and B).

(8) The nature of the relationship between serviceproviders and clients will need to be definedand questions selected (table 5A, B or C) toevaluate how effective these relationships are ingiving the client the best possible benefits fromthe services.

(9) The involvement of relatives or neighbours ofthe client in the rehabilitation process need to beconsidered. Firstly to see if such involvement ispart of the strategies used by the programme andsecondly to evaluate the extent and effectivenessof this involvement (table 6).

By going through these steps, the evaluators will haveprepared a list of evaluation questions relevant to theprogramme they will visit and relevant to the purposeof the evaluation. In addition, they will have learned agreat deal about the programme they are going to seeand will have discovered about which areas they havewritten information and about which areas more infor-mation is needed.

Once the evaluators do get to see the programme withtheir own eyes, they will be busy collecting informationthat will enable them to answer the evaluation questions.How best to do this is a separate issue that we have not

really addressed in depth in the present paper. Theevaluators need to make use of all available data sourcesand use their repertoire of quantitative and qualitativedata collection techniques. Thinking about possiblesources of information is part of preparing the evalua-tion.

It is important that the evaluators are open to correct-ing their initial understanding of the programme whenactually visiting the programme. Discrepancies betweenwhat is written and what is practised are common asadjustments may have been made or the evaluatorsmay have misinterpreted the written information. Byimplication, the evaluators may find in the course ofthe evaluation that some of the questions they selectedare not appropriate while others should be added. Suchflexibility is necessary if one is to do justice to the workthat is being done on the ground.

Analysis of and reflection on the informationcollected will result in answers to the evaluation ques-tions. How firm those answers can be depends on thequantity and quality of the information available on aparticular issue.

Weighing up the evidence and comparing the situa-tion in one programme to the situation in otherprogrammes and to professional norms and values isthe essence of the evaluation work. By being transparentabout their reasoning and the logical steps throughwhich conclusions are reached, the evaluators canreduce tension and provide the basis for a meaningfuldiscussion among stakeholders about the problems theyidentify and the recommendations they present.

Acknowledgements

The authors wish to thank Dr. PG Nicholls for stimulating their

thoughts in early stages of this work and Drs. Pim Kuipers, Wim

Brandsma and Margie Schneider for helpful comments on the

manuscript.

References

1 Mitchell RA, Zhou D, Lu Y, Watts G. Community-basedrehabilitation: does it change community attitudes towards peoplewith disability? Disability and Rehabilitation 1993; 15: 179 – 183.

2 Lundgren-Lindquist B, Nordholm LA. The impact of community-based rehabilitation as perceived by disabled people in a village inBotswana. Disability and Rehabilitation 1996; 18: 329 – 334.

3 Nordholm LA, Lundgren-Lindquist B. Community-based rehabi-litation in Moshupa village, Botswana. Disability and Rehabilita-tion 1999; 21: 515 – 521.

4 Sharma M, Deepak S. A participatory evaluation of community-based rehabilitation programme in North Central Vietnam.Disability and Rehabilitation 2001; 23: 352 – 358.

5 Mitchell R. The research base of community-based rehabilitation.Disability and Rehabilitation 1999; 21: 459 – 468.


1263

6 Stuelz A. Community-based rehabilitation in Lao—comparison ofneeds and services. Disability and Rehabilitation 1999; 21: 508 – 514.

7 Cornielje H, Nicholls P, Velema J. Making sense of rehabilitationprojects: classification by objectives. Leprosy Review 2000; 71:472 – 485.

8 Cornielje H, Nicholls P, Velema J. Avoiding misperceptions:Classifying rehabilitation projects using letters rather than num-bers. Leprosy Review 2002; 73: 47 – 51.

9 Boyce W, Broers T, Paterson J. CBR and disability indicators. AsiaPacific Disability Rehabilitation Journal 2001; 12: 3 – 21.

10 McLaren P, Philpott S. Indicators in Disability Research. In:Proceedings of the Workshop on Research Informed RehabilitationPlanning in Southern Africa. Harare, Zimbabwe, 28 June – 3 July,1998; 35 – 41.

11 ILEP. Guidelines for the Social and Economic Rehabilitation ofPeople Affected by Leprosy. London; 1999.

12 Chermak GD. A global perspective on disability: a review of effortsto increase access and advance social integration for disabledpersons. International Disability Studies 1990; 12: 123 – 127.

13 Kendall E, Buys N, Lamer J. Community-based service delivery inrehabilitation: the promise and the paradox. Disability andRehabilitation 2000; 22: 435 – 445.

14 Sandstrom R, Hoppe KM, et al. Comprehensive medical rehabi-litation in the 1990s: the community integration rehabilitationmodel. Health Care Supervisor 1996; 15: 44 – 54.

15 Inthirat TR, Thonglith S. Community-based rehabilitation in theLao people’s democratic republic. Disability and Rehabilitation1999; 21: 469 – 473.

16 Cornielje H. The Prevalence of Disability and Handicap and thePattern of motor disability in the Gelukspan Health Ward. MScthesis, University of Witwatersrand, South Africa, 1991.

17 WHO. International classification of functioning disability andhealth (ICF). 2001. http://www.who.int/classification/icf

18 Tanahashi T. Health service coverage and its evaluation. Bulletin ofthe WHO 1978; 56: 295 – 303.

19 UN Resolution 48/96, Standard Rules on the Equalization ofOpportunities for Persons with Disabilities, 1994, http://www.un.org/esa/socdev/enable/dissre00.htm

20 Evans PJ, Zinkin P, Harpham T, Chaudury G. Evaluation ofmedical rehabilitation in community based rehabilitation. SocialScience and Medicine 2001; 53: 333 – 348.

21 Finkenflugel HJ, van Maanen V, Schut W, Vermeer A, Jelsma J,Moyo A. Appreciation of community-based rehabilitation bycaregivers of children with a disability. Disability and Rehabilitation1996; 18: 255 – 260.


1264