Proposal and Program Evaluation

Ivan Yau

Critical Reflections

Different Purposes / Targets lead to different Strategies and Demands on rigorousness of program evaluation – to public, visual documentation and stories can be used effectively (have a sense that these are representative), to academic and funder, generally, statistical analysis is used (but should have meaningful no.!!)

Critical Reflections

Poor in Program Evaluation in general so far!! – these can be reflected in program evaluation published with poor objectives, evaluation methods and its interpretation. So what are your roles as social workers?

Critical Reflections Evaluation on Program Effectiveness

are linked to ‘accountability’ and ‘monitoring’ both. If impact / effectiveness can be demonstrated (accountability), our service can only be monitored by output.

Program Evaluation is a progressive and goal-driven process, not a repetitive routine measure, e.g. after a rigorous evaluation on effectiveness, only certain kind of QC or a more in-depth evaluation is required.

Further development

Some examples

1. Hong Kong Family Welfare Society (1989) A research report on cognitive-behavioral Group Therapy. - further developed into more refined program on clients with more severe M.I. symptoms, parent group and other with more skill deficits

2. Chronic Disease Self Management Program (CDSMP) – more vigorous screening is required to ensure the commitment in the program and reduce drop-out rate

Critical Reflections Evidence – biased preference1.Numbers – even converting subjective

experience into Numbers!!

2.Counting and Rating3.Statistical Analysis (T-test, sign-test,

…logic of significant level and hypothesis testing)

Program Logic One Linear (common, but oversimplified)

Program Logic Two Holistic View: Analysis of Target and Problem, Mechanism of

Change (Intervention), Objectives, Evidence and Expected changes

Program Logic Three From SSLD perspective:

Program Evaluation Objectives:1.Change2. Important3.From SW concern / perspective!! Effective? - Evidence (changes

observed) is consistent with the objective.

Illustration – Program Logic

跨越鴻溝 – 促進親子溝通計劃（ 2001） 針對大埔區青少年不斷離家出走的現象 合作伙伴：警方（轉介） 潛伏危機：受誘惑而去吸毒、販毒、賣淫 服務目標所受的影響：減少再次離家出走的情況 社會控制的議程：加派警力巡查、對離家出走的青少年 /家長提出警告、向多次離家出走青少年的家庭提出保護令的申請、……

Illustration – Program Logic 現象分析：親子缺乏溝通是一般離家出走少年的起源？

理論：青少年成長理論、家庭發展歷程、…… 進一步探索：1. 青少年離家後在甚麼地方流連？有何活動？跟甚麼人連繫？（滿足甚麼需要？）

2. 這些青少年的家庭狀況？他們對家庭的觀感？家庭關係？（家庭無法滿足他們甚麼需要？）

Illustration – Program Logic 一些可能的需要： 進一步探索：1. 社群連繫（歸屬感、關顧、認同… )2. 刺激、歡樂、成功感、… 滿足需要的其他途徑…… ?

Program Plan

What are common problems in program / session plan?

Program Plan Common Problems

1.Linear without contingent thinking2.Not aware of mechanism of change

Program Plan

Mechanism of Change Some Illustrations

1. Skill training – Parenting skill training as an example structured exercise, brief lecture, demonstration, role-play and feedback, experimentation in real life

2. Cognitive restructuring – getting familiar with ABC (differentiating DT /DT, aware of undesirable impact of DT), identifying own DT, disputing these through empirical enquiry, self-restructuring process

Mechanism of Change Some Illustrations3. Self Understanding / acceptance – self-

exploration on critical incidents in trustful environment, feedback from others after intensive interaction, connecting with others deeply and comparing the difference

4. 生死教育 – reflection on the edge of life / death experience or stimulated experience

Program Plan

Program Evaluation

Some Clarifications Intervention is different from

Objective / Effectiveness. Process review is useful for

improving the practice. Client’s satisfaction is Not the

program effectiveness.

Objective Usually viewed as ‘Prevention of

Undesirable Consequence’ / ‘Alleviation of the (social) Problem’

Important and Desirable – from who’s perspective? Client’s?

Influence from stake-holder? Theory-driven? Social Control?

Effectiveness and Change Extent: Magnitude in relation to all

participants (sometimes Magnitude alone is sufficient!)

Confidence: Change is not the result of chance

Causal relationship: can hardly be established

Intervention Intervention must link to mechanism

of changes (theory-driven and common sense!!) and desirable outcome.

Group proposal – brief session outline first, and further detailed session plan as the group evolves gradually.

Effectiveness I

Participants’ change (pre-post, post only) Issues of measurement: validity (face

validity), reliability (items-correlation) and sensitivity

Changes reach statistically significant level Single-item comparison Scale construction: literature review, local

validation, feedback from participants,…

Effectiveness II

Outcome Indicator Criteria for success: how come? Baseline makes comparison feasible

(indicating the change) Meaningful (and significant) to reflect the

accumulated effort, or indirect evidence to achieve the ultimate goal (community level intervention)

Different Levels – G & P Group and Programs1. Specific target group2. Focused, limited intervention3. Operationalization of Objectives –

pre-post measures, post-measures4. Use of statistical analysis to assess

significant changes

Different Levels - Project Project1. Specific target group with rather

long-term intervention2. Focus on several key outcome

indicators3. Pre-post measures are also feasible

Different Levels - Service Service1.Diversified target groups with long-

term intervention and general / vague objectives (IFSC, DEC,…)

2.Aggregated data is always useful3.Focused on major groups, or use of

randomizations if necessary

Political Agenda Making use of client’s views in

advocating for changes, e.g. IFSC, CSSA, SW field work supervision, 自力更新服務

Connecting impact of the program with advocacy / policy change, e.g. 類風濕性關節炎自我管理課程、中學義工服務、小學學校社工服務、兒童之家服務、…

Rounding Up Aggregate data (client’s satisfaction, objective reached,

target complaints, etc) is useful for service evaluation, key outcome indicator (decrease in A & E of child injury due to child neglect, decrease in suicide) for project evaluation, while accumulated impact (make use of community resources by NA, support network, no. of high risk case receiving service) for both.

Rounding Up Evaluation should be better linked to

continuous improvement (don’t repeat the

evaluation every year), service development (promotion of a better service model) or social movement (advocacy) (e.g. changing concepts of chronically ill patient – ownership of health, instead of replying on medical treatment).

Rounding Up Evaluation can be and should be Political

– particularly help marginalized group to voice out their views / needs.

Social workers should:1. Be able to link up the objective,

intervention and outcome.2. Be quipped with themselves of different

ways / strategies for program evaluation.

Reference 陳錦棠（ 2006 ）社會服務成效評估：程序邏輯模式之應

用 . 香港：香港基督教女青年會 羅致光等編（ 1999）靑少年成長與活動案例 . 香港：香港基督敎服務處。

羅致光主編（ 1992 ）靑少年中心活動成效評估 : 方法與案例 . 香港：香港基督敎服務處

United Way of America (2006) Measuring Program Outcomes: A Practical Approach