formative evaluation of unicef’s #thisability flagship

Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative

2017–2019 in Malaysia

EVALUATION REPORT NOVEMBER 2019

Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia Evaluation Report

Year of evaluation: 2019Report submitted: November 2019Commissioning organization: UNICEF MalaysiaAuthors: Lorraine Wapling and Nick Corby (Equal International); Wan Nuradiah (Ipsos Malaysia), Jessica Bruce (Ipsos MORI)Cover photo: UNICEF Malaysia/2013

© UNICEF MalaysiaUnited Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 Putrajaya

November 2019

The purpose of publishing evaluation reports produced by UNICEF Malaysia is to fulfil a corporate commitment to transparency through the publication of all completed evaluations. The reports are designed to stimulate a free exchange of ideas among those interested in the topic and to assure those supporting the work of UNICEF that it rigorously examines its strategies, results, and overall effectiveness.

The contents of the report do not necessarily reflect the policies or views of UNICEF.

The text has not been edited to official publication standards and UNICEF accepts no responsibility for error.

The designations in this publication do not imply an opinion on the legal status of any country or territory, or of its authorities, or the delimitation of frontiers.

The copyright for this report is held by the United Nations Children’s Fund. Permission is required to reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires a written request to be submitted. For non-commercial uses, the permission will normally be granted free of charge. Please write to UNICEF Malaysia at the address below to initiate a permission request.

For further information, please contact:

UNICEF MalaysiaUnited Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 PutrajayaEmail: [email protected]

UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report

i

CONTENTSAcronyms ................................................................................................................................ iiiExecutive summary ............................................................................................................... iv

Background ......................................................................................................................... ivEvaluation purpose, objectives and scope ........................................................................... vMethodology ........................................................................................................................ viMain findings .......................................................................................................................viiMain recommendations ......................................................................................................viii

1. Introduction ..................................................................................................................... 11.1. Disability inclusion within UNICEF Malaysia .............................................................. 11.2. UNICEF Malaysia’s #ThisAbility Flagship Initiative within the national context ......... 2

1.2.1. Disability in Malaysia: Policy context ................................................................... 21.2.2. Perceptions of children with disabilities in Malaysia ............................................ 31.2.3. Inclusion in education for children with disabilities .............................................. 51.2.4. Official data on children with disabilities in Malaysia ........................................... 7

1.3. #ThisAbility Flagship Initiative strategic goals ............................................................ 92. Evaluation purpose, objectives and scope.................................................................11

2.1. Purpose .................................................................................................................... 112.2. Evaluation objectives ............................................................................................... 112.3. Scope ....................................................................................................................... 122.4. Key evaluation questions ......................................................................................... 13

3. Evaluation methodology ............................................................................................... 153.1. Design and methods ................................................................................................ 153.2. Sampling .................................................................................................................. 153.3. Data management and handling .............................................................................. 163.4. Approach to equity, gender and human rights ......................................................... 163.5. Ethical issues and considerations ............................................................................ 173.6. Methodological limitations ........................................................................................ 18

4. Findings ......................................................................................................................... 194.1. Effectiveness ............................................................................................................ 19

4.1.1. Satisfaction levels and public awareness .......................................................... 204.1.2. The effectiveness of Disability Equality Training ................................................ 254.1.3. The effectiveness of partnerships ...................................................................... 274.1.4. Impact of partnerships on empowerment and self-advocacy ............................ 284.1.5. The effectiveness of public advocacy campaigns .............................................. 294.1.6. Evidence of cross-sector work ........................................................................... 31


ii

4.1.7. The extent of gender inclusion ................................................................................ 324.2. Relevance ..................................................................................................................... 32

4.2.1. Alignment to Agenda 2030 ...................................................................................... 334.2.2. Alignment to Malaysia Vision 2020 ......................................................................... 334.2.3. Alignment to UNICEF’s Strategic Plan ................................................................... 344.2.4. Intersectionality and the voice of children with disabilities ...................................... 35

4.3. Efficiency ....................................................................................................................... 364.3.1. Human, technical and financial resourcing ............................................................. 364.3.2. The adequacy of partners’ capacity ........................................................................ 37

4.4. Sustainability ................................................................................................................. 384.4.1. Sustainability at national and sub-national level ..................................................... 384.4.2. Enabling and constraining factors for future work ................................................... 39

4.5. Utility of a Flagship Initiative.......................................................................................... 395. Conclusions ........................................................................................................................ 436. Lessons learned ................................................................................................................. 457. Recommendations ............................................................................................................. 47Annex A: Evaluation Terms of Reference .................................................................................... 50Annex B: Evaluation Matrix ......................................................................................................... 64Annex C: Research Tools ............................................................................................................ 67Annex D: Stakeholders interviewed............................................................................................. 78Annex E: Document analysis map............................................................................................... 79Annex F: Theory of Change ........................................................................................................ 92


iii

ACRONYMSBIM Borneo International Marathon

C4D Communication for Development

CPAP Country Programme Action Plan

CPD Country Programme Document

CPMP Country Programme Management Plan

CRC UN Convention on the Rights of the Child

CRPD Convention on the Rights of Persons with Disabilities

DAC Development Assistance Committee

DET Disability Equality Training

FGD Focus group discussion

IPG Integrated Play Group

KAP Knowledge, Attitudes and Practices

KII Key informant interview

MoU Memorandum of Understanding

NGO Non-governmental organization

OECD Organisation for Economic Co-operation and Development

OKU Orang Kurang Upaya (less- abled person)

SDGs Sustainable Development Goals

ToC Theory of Change

TOR Terms of Reference

UN United Nations

UNEG United Nations Evaluation Group

UNICEF United Nations Children’s Fund

WHO World Health Organization


iv

EXECUTIVE SUMMARY

BackgroundThe World Health Organization (WHO) World Report on Disability estimates that globally over 1 billion people are living with disabilities, representing just over 15 per cent of the world’s population.1 Around one household in every four includes a person with disabilities, which means that over 2 billion people live with the impact of disability on a daily basis. Moreover, the prevalence of disability is growing due to ageing populations and global increases in chronic health conditions and non-communicable diseases.

The lives of children and young people with disabilities in Malaysia, like those with disabilities in many other countries, tend to be disadvantaged compared with their peers without disabilities.2 While the Government of Malaysia has shown commitment to improving the lives of persons with disabilities, including by ratifying the United Nations Convention on the Rights of Persons with Disabilities (CRPD) in 2010 and through its Disability Act (2008), gaps remain in the implementation of CRPD compliant policies and plans.3 There is a lack of data on the prevalence of disability among children and young people in Malaysia; their voices and lived experiences remain generally unheard, and the underlying drivers for stigma and discrimination still exist.

In 2013, the United Nations Children’s Fund (UNICEF) published a groundbreaking State of the World’s Children report, which focused specifically on children with disabilities. It highlighted the impact that disability can have on children’s life chances, even when their lives may already be difficult. It made a powerful case for demonstrating that children and young people with disabilities are at greater risk of being poor than their peers without disabilities, and they experience their poverty more acutely because of the additional barriers they face living with their impairments.

UNICEF Malaysia began planning its #ThisAbility Flagship Initiative in 2016. The goal was to make positive changes for children with disabilities in Malaysia so that they are free from discrimination and able to access services to support their growth, development and protection. The plan was to run the Flagship Initiative from 2017 to 2020, however, in early 2019, UNICEF Malaysia decided to no longer refer to its work on disability as a ‘flagship’, and committed to mainstreaming disability-inclusive programming.

This evaluation focused on activities carried out under the #ThisAbility Flagship Initiative between 2017 and 2019. UNICEF Malaysia did not define what a ‘flagship’ is but, by establishing a ‘flagship initiative’, it signalled intent to scale up its focus on disability, with opportunities to establish new partnerships, carry out more focused activities, and learn from the experience of implementing disability initiatives.

1 World Health Organization, World Report on Disability, WHO, Geneva, 2011. <www.who.int/disabilities/world_report/2011/en>, accessed June 27 2018.

2 Nurhidayah Abdullah, Haniri Hanafi and Nur Izzati Mohd Hamdi, ‘The rights of persons with disabilities in Malaysia: the underlying reasons for ineffectiveness of Persons with Disabilities Act 2008’, International Journal for Studies on Children, Women, Elderly and Disabled, vol.1, January 2017. <www.ijcwed.com/wp-content/uploads/2016/12/IJCWED-165.pdf>

3 Based on the UN Convention on the Rights of Persons with Disabilities, children and young people with disabilities are defined as those under 18 years of age and: ‘...who have long-term physical, mental, intellectual or sensory impairments which in interaction with various barriers may hinder their full and effective participation in society on an equal basis with others’.


v

UNICEF Malaysia did not define a fixed budget, target number of beneficiaries or geographical coverage for the Initiative. At the time of the evaluation, UNICEF Malaysia did not appear to have plans to scale up or replicate the Initiative or its associated activities, but the activities it delivered will provide valuable learning for future disability-inclusion activities delivered by the Country Programme.

Evaluation purpose, objectives and scopeUNICEF Malaysia commissioned this evaluation as a formative, forward-looking process. The specific objectives of the evaluation, as defined in the Terms of Reference (ToR) (see Annex A) were to:

• analyse the strategies used to map out the trajectory of the #ThisAbility Flagship, and re-construct the Theory of Change (ToC) behind making positive changes for children with disabilities in Malaysia;

• assess the extent to which the #ThisAbility Flagship has been appropriately designed, efficiently and effectively managed, and integrated across the Country Programme, including the incorporation of equity, gender equality and human rights considerations;

• examine the evolution of UNICEF’s work on children with disabilities in Malaysia, and in particular the intermediate results achieved by UNICEF’s #ThisAbility Flagship, enabling and disabling factors, considering aspects of both reduction of discrimination as well as access to services – including advocacy and awareness-raising both at the national and sub-national level; and its likely sustainability;

• analyse the utility of creating a flagship on specific focus issues to promote cross-sectoral work, including strengths and weaknesses.

This evaluation consequently looked at both the design and management of the initiative as well as assessing its programmatic effectiveness, relevance, efficiency and sustainability. The relatively nascent nature of the Initiative meant that the evaluation focused on finding evidence of outputs and early-stage outcomes, not on measuring its impact.

It was anticipated that the evaluation would identify areas of promising practice and lessons learned, and provide recommendations as to how UNICEF Malaysia might mainstream disability across its programme. These recommendations include a ToC that aims to assist UNICEF Malaysia to better meet its commitments towards championing the rights of children with disabilities.

The scope of this evaluation included all aspects of the #ThisAbility Flagship Initiative implemented since 2017. At the request of UNICEF’s Evaluation Specialist, the external evaluators paid particular attention to 10 select activities, including the Disability Equality Training (DET) and the Borneo International Marathon (BIM). The utility of the Initiative is also considered, but this was not the focus of the evaluation questions (see Figure 1).


vi

Figure 1: Evaluation questions

Organisation for Economic Cooperation and Development (OECD)/ Development Assistance Committee (DAC) criteria

Effectiveness of UNICEF’s strategy to children with disabilities and its implementation considering:

To what degree has UNICEF’s work on disability, and in particular through its #ThisAbility Flagship Initiative, contributed to the creation of positive conditions to reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia? Have there been any unintended results?

Relevance of the programme strategy and approach, considering:

How relevant is UNICEF’s work on disability and its #ThisAbility Flagship Initiative to the developmental challenges in Malaysia, and particularly attitudes and practices with regard to disability?

Efficiency of the programme implementation and management, considering:

To what extent and how has UNICEF mobilized and used its resources (human, technical and financial) and improved its cross-sectoral coordination to achieve its planned results for children with disabilities?

Sustainability in the programme, considering:

To what extent are the benefits and achievements of the #ThisAbility Flagship Initiative likely to continue after 2020 through national ownership and changes in terms of attitudes, knowledge and practices?

The primary users of this evaluation are expected to include UNICEF Malaysia, the Government of Malaysia and implementing partners. It is expected that the primary users will employ the conclusions and lessons learned to shape the forthcoming Country Programme Document (CPD) and to inform the development of new partnerships, programmes and initiatives. Secondary users are expected to include but not be limited to: other agencies working on disability-inclusive programmes for children and young people; UNICEF’s East Asia and Pacific Regional Office; UNICEF headquarters (especially the Disability and Child Protection Divisions); and the parents of and children with disabilities in Malaysia. It is envisaged that the secondary users of this evaluation will utilize the conclusions and lessons learned to strengthen disability inclusion in their work, identify potential synergies and/or identify opportunities for collaboration.

MethodologyThe methodological framework for this evaluation was based on the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation (2016) and informed by UNICEF guidelines. The evaluation followed a mixed-methods approach, non-experimental design to addressing the key evaluation questions. This evaluation drew upon three key strands of data collection: three focus group discussions (FGDs) with children and young people with disabilities and their parents/carers; 42 key informant interviews (KIIs) with staff from within UNICEF, as well as those in partner


vii

organizations, and a small number of government representatives; and a comprehensive desk review of internal UNICEF documents alongside available external materials. UNICEF provided a sample frame of participants for both the FGDs and the KIIs; the external evaluators used purposive sampling to ensure diverse perspectives on the #ThisAbility Flagship Initiative.

It is worth noting that a lack of outcome data and the mid-term nature of the evaluation meant that impact criteria were excluded. Furthermore, in the absence of financial data, value for money was not measured.

To fully examine the #ThisAbility Flagship Initiative, data was analysed against key barriers to inclusion, including those that were attitudinal, environmental and institutional. This analysis undertook to identify barriers experienced by children and young people with disabilities in the context of the Initiative; this was used to help determine whether the Flagship addressed key issues of relevance to programme participants.

Main findings In terms of effectivenessMost of those directly involved in activities implemented under the #ThisAbility Flagship Initiative found them to be interesting, exciting and of benefit to the children with disabilities who took part. However, since these activities were all one-off events, there was little opportunity for effecting lasting attitudinal change, and no clear evidence of effectiveness in reducing discrimination and enabling access to services by children and adolescents with disabilities. Due to the lack of an advocacy strategy, behavioural change activities, follow-on activities, and an absence of monitoring structures (including the collection of disability-disaggregated data), UNICEF Malaysia was unable to determine whether its interventions were having an impact on changing attitudes. There is a disconnect between the Initiative’s goals and objectives and the intent for the Initiative to promote mainstreaming of disability programming within the UNICEF Malaysia office, which has limited the effectiveness of this Initiative in all respects.

In terms of relevanceUNICEF Malaysia has signalled a strong desire to work on disability inclusion within its CPD and country strategy documents. In turn, the Government of Malaysia has also made public its intentions to create more equal and inclusive services. Given that the Government has ratified the CRPD but has struggled with implementation, UNICEF’s intention to work on disability is highly relevant. UNICEF Malaysia has missed opportunities to better align its activities under the Initiative with broader influencing around the rights of persons with disabilities. This is because it has not made explicit use of international frameworks such as the CRPD in campaigning. The lack of an overall advocacy strategy has limited the success of individual events to have a broader impact.

In terms of efficiencyStakeholders generally observed the activities delivered under the Initiative as well resourced and well managed, with a high degree of technical support provided. Internally concerns were raised that the prioritization of resources, especially human resources, was insufficient for organization-wide change and that a more strategic approach to disability is required. The lack of systematic monitoring made it difficult for UNICEF Malaysia to assess whether its resources have been used in the most efficient way.


viii

In terms of sustainabilityBecause the #ThisAbility Flagship Initiative was intended to be primarily an internal management device to increase disability work across the organization, it lacked the national ownership required for sustainability. While the Initiative has held many events and campaigns to change attitudes, knowledge and practices (as noted in previous sections), the lack of an advocacy strategy and any systematic monitoring data meant this evaluation could not determine their impact or sustainability.

Main recommendationsRecommendation 1 To: Country Representative; Deputy Representative; Planning Monitoring and

Evaluation ChiefPriority: HighTimeline: 6-months

UNICEF Malaysia needs to make a strategic commitment to disability inclusion by fully embedding disability into key planning documents with the expectation that children with disabilities will be included and monitored across all areas of programming. We encourage UNICEF Malaysia to start doing this in the short-term, beginning with the forthcoming strategic planning.

Recommendation 2To: Country Representative; Deputy Representative; Communications Chief; Private

Funding and Partnerships Chief; C4D SpecialistPriority: MediumTimeline: 12-months UNICEF Malaysia, starting immediately, needs to be much more explicitly rights-focused in its messaging around disability, and ensure its campaigns fully align with key international frameworks, including the CRPD.

Recommendation 3 To: Country Representative; Deputy Representative; Communications Chief;

Programme StaffPriority: MediumTimeline: 12-months

UNICEF Malaysia, starting immediately, needs to provide more opportunities to meaning-fully engage children and young people (with and without disabilities) during the design of campaigns and messaging.


ix

Recommendation 4 To: Country Representative; Deputy Representative; Communications Chief;

Programme Chiefs; Programme StaffPriority: HighTimeline: 6-months

UNICEF Malaysia urgently needs to develop robust and rigorous safeguarding review processes for all activities involving children and young people with disabilities.

Recommendation 5 To: Country Representative; Deputy Representative; Gender Focal Point;

Communications Chief; Programme Chiefs; Programme StaffPriority: MediumTimeline: 12-months

UNICEF Malaysia, starting as soon as possible, to routinely assess new and existing programmes to ensure gender and social inclusion norms and unequal power relations are considered in full.


1

1. INTRODUCTION

1.1. Disability inclusion within UNICEF MalaysiaUNICEF Malaysia has placed itself at the forefront of UNICEF’s growing focus on disability. In 2013, the United Nations Children’s Fund (UNICEF) published a groundbreaking State of the World’s Children report, which focused specifically on children with disabilities. UNICEF Malaysia contributed to the report and supported its launch and dissemination. The report underlined the impact that disability can have on children’s life chances even when their lives may already be difficult. It emphasized that children and young people with disabilities are at greater risk of being poor than their peers without disabilities and experience their poverty more acutely because of the additional barriers they face living with their impairments.

In 2014, UNICEF Malaysia published a comprehensive study mapping the policies, programmes and interventions relevant to children with disabilities at that time. The study pointed to some significant systemic gaps around the provision of health, education, rehabilitation, protection and welfare services for children with disabilities and their families. UNICEF Malaysia led a Communication for Development campaign in 2014 (Disabled2Enable) and supported the establishment of the Malaysian Partnership on Children with Disabilities in 2015 and 2016, creating solid connections between UNICEF and the disability sector.

In 2016, UNICEF Malaysia commissioned a further study, this time focusing specifically on the experiences, challenges and aspirations of children and young people with disabilities. The resulting Knowledge, Attitudes and Practice (KAP) Survey on children with disabilities in Malaysia (2016)4 provided UNICEF Malaysia with a solid evidence base highlighting the lack of awareness of the rights of children with disabilities; the extent of the barriers to participation across services; and the impact that continued misconceptions around disability have on limiting the opportunities available for children and young people with disabilities to develop into confident, independent citizens.5

Building on these foundations, UNICEF Malaysia began to formulate its #ThisAbility Flagship Initiative in 2016, the goal being to effect positive change for children with disabilities across Malaysia. The intention was to run the Initiative from 2017 to 2020. However, in early 2019, UNICEF Malaysia decided to no longer refer to its work on disability as a’ flagship’ and committed to mainstreaming disability-inclusive programming. At the beginning of this evaluation, the Flagship Initiative was, therefore, at a midpoint in its intended implementation. By establishing a flagship initiative, UNICEF Malaysia committed to scaling up its focus on disability, with the opportunity to establish new partnerships, carry out more focused activities and learn from the experience of implementing disability programming. UNICEF Malaysia did not determine a budget, target number of beneficiaries or geographical coverage for the Initiative.

4 United Nations Children’s Fund (UNICEF), Childhood Disability in Malaysia: A Study of Knowledge Attitudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017 <www.unicef.org/malaysia/media/281/file/Childhood%20Disability%20in%20Malaysia.pdf>

5 Moore K., and Bedford, J., Childhood disability in Malaysia: a study of knowledge, attitudes and practices. UNICEF Malaysia, n.d.


2

1.2. UNICEF Malaysia’s #ThisAbility Flagship Initiative within the national context

Malaysia is a culturally diverse upper-middle-income country, home to around 30 million people of whom about 33 per cent are below 18 years of age.6 The Government aims for Malaysia to become a high-income country by 2020. Malaysia’s Vision 2020 plan promotes not just economic progress, but also commits to improving the quality of life for everyone through areas such as political stability and social justice. Malaysia’s National Transformation 2050 sets a target for Malaysia to become one of the top 20 nations in the world economically and socially. The Eleventh Malaysia Plan (2016–2020) also recognizes that as well as making economic progress, it is important to ensure that growth is inclusive and, as such, it needs to do better in being able to reach the most marginalized in society, including adults and children with disabilities.7

Within its Country Programme Document 2016–2020 (CPD), UNICEF commits to supporting Malaysia to make the transition to high-income status. UNICEF Malaysia’s Country Programme Action Plan (CPAP) and the Country Programme Management Plan (CPMP) recognize the Country Programme’s unique position by focusing on the most hard-to-reach children to ensure that no child is left behind. Disability is identified in both the CPD and the CPMP, in effect connecting national development targets with UNICEF’s priorities.

1.2.1. Disability in Malaysia: Policy contextSeveral national social welfare and service policies help to define the rights of people with disabilities in Malaysia; however, as the most recent Universal Periodic Review report notes, Malaysia’s Constitution (1957) does not include any reference to children or adults with disabilities.8 The National Welfare Policy (1990) made provision for those with disabilities and was an early attempt to set out the Government’s intention to create a society that promotes ‘equalization of opportunities’. The National Social Policy (2003) specified that people with disabilities should ‘enjoy equal rights and full participation in society’ and focused on fifteen different areas where inclusion should be prioritized including, for example, rehabilitation, education, employment, social assistance and accessibility. Women and children with disabilities were also specifically mentioned as a target group.

The most significant step forward in recognizing the rights of people with disabilities came in 2008 with both the signing of the United Nations Convention on the Rights of Persons with Disabilities (CRPD) and the enactment of the Persons with Disabilities Act. This was the first time the Government formally acknowledged the need to reduce barriers to participation and adopted a rights-based approach to disability.9 Malaysia went on to ratify the CRPD in 2010 (albeit with reservations to Articles 15 and 18) which, given it has only ratified three of the ten core human rights treaties, at least demonstrates serious intent to promote inclusion.10

6 index mundi, Malaysia Demographics Profile 2019, <www.indexmundi.com/malaysia/demographics_profile.html>7 Eleventh Malaysia Plan 2016–2020: Anchoring growth on people. Speech by the Prime Minister in the Dewan

Rakyat, 21 May 2015. <www.pmo.gov.my/dokumenattached/speech/files/RMK11_Speech.pdf>8 Report of the Office of the United Nations High Commissioner for Human Rights Human Rights Council Working

Group on the Universal Periodic Review Thirty-first session, 5–16 November 2018, p. 2.9 Nurhidayah Abdullah, Haniri Hanafi and Nur Izzati Mohd Hamdi, ‘The rights of persons with disabilities in Malaysia:

the underlying reasons for ineffectiveness of Persons with Disabilities Act 2008’, International Journal for studies on children, women, elderly and disabled. vol.1, January 2017. <www.ijcwed.com/wp-content/uploads/2016/12/IJCWED-165.pdf>

10 Mokhtar, Kharil Azmin and Md Tah, Ikmal Hisham, ‘Malaysia’s ratification of the UN Convention on the Rights of Persons with Disabilities (UN CRPD)’, International Journal of Business, Economics and Law, vol. 11, issue 4, 2017 <http://irep.iium.edu.my/54634>


3

A key deficiency of the Persons with Disabilities Act however, is that it makes no specific provisions for children or young people with disabilities beyond ensuring their right to access education (Article 28) and their right to access recreation, leisure and sport (Article 32).11 In fact, there is no recognition of the impact of age or gender on disability at all. Another significant problem with the Act is the lack of monitoring and redress mechanisms should any party or the Government violate the rights of persons with disabilities. Similarly, Malaysia has yet to sign the Optional Protocol to the CRPD concerning procedures for individual communications and inquiry, which indicates the lack of legal mechanism in the event that the rights of persons with disabilities are violated. This is also reflected in the omissions within the Persons with Disabilities Act.12

The National Council for Persons with Disabilities has the responsibility of overseeing national progress against plans and policies, but there are no penalties for non-compliance with the Disability Act. Malaysians with disabilities are unlikely to be able to bring a legal claim against the Government or service providers for violation of their rights.13 A recent paper on the effectiveness of the 2008 Act14 noted that in its nine years of enactment, no case has referenced it in relation to protecting the rights of persons with disabilities in registered court cases. Therefore, there have been calls for the Persons with Disabilities Act to be abolished, and proposals made for a Disability Discrimination Act as well as the establishment of a commission with enforcement powers to put in place legal measures for non-compliance.15

There is little direct protection of the rights of children with disabilities across the legislative framework. Malaysia enacted the Child Act in 2001 as part of its obligations under the UN Convention on the Rights of the Child (CRC). Although based on non-discrimination, the Federal Constitution and reservations in the CRC do not protect children with disabilities from discrimination.16 Since the Persons with Disabilities Act also fails to specifically reference children and young people with disabilities, there is a significant gap in the protection and promotion of their rights.

1.2.2. Perceptions of children with disabilities in Malaysia The KAP study, commissioned by UNICEF in 2016, revealed that the majority of participants (approximately 60 per cent) were not well informed about disability.17 The lack of awareness and understanding, as well as misconceptions of disability in Malaysia, appear to be key barriers restricting the opportunities available to children with disabilities in comparison with their peers without disabilities.

11 Persons with Disabilities Act 2008, Act 685, Government of Malaysia12 Abdullah, Hanafi and Hamdi, ‘The rights of persons with disabilities in Malaysia’, 2017.13 Section 41 in Part V of Persons with Disabilities Act 2008.14 Abdullah, Hanafi and Hamdi, ‘The rights of persons with disabilities in Malaysia’, 2017.15 Group: Do away with People with Disabilities Act https://www.thestar.com.my/news/nation/2018/07/02/group-do-

away-with-disabilities-act-it-merely-serves- as-an-administrative-document-with-no-enforceme/16 Children with Disabilities in Malaysia: Mapping the Policies, Programmes, Interventions and Stakeholders https://

www.unicef.org/malaysia/UNICEF-Children_with_Disability_in_Malaysia_2014_lowres.pdf17 United Nations Children’s Fund (UNICEF) Malaysia, Childhood Disability in Malaysia: A Study of Knowledge Atti-

tudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017 <www.unicef.org/malaysia/media/281/file/Childhood%20Disability%20in%20Malaysia.pdf>


4

Barriers at family levelAttitudinal barriers have created anxiety among parents of children with disabilities, who hesitate or refuse to register their children as having a disability. Those who do attempt to access services for their children, such as enrolling them in mainstream schools, continue to face resistance or rejection.18 A recent study noted high levels of unmet needs especially among mothers of children with disabilities.This study found that parents of children with disabilities often find themselves quite isolated as they struggle to find appropriate information, financial and social support. Parents reported experiencing a wide range of challenges from lack of accessible transport, increased healthcare costs, inaccessible services and fragmentation of support, to poor coverage of specialist services, lack of experienced professionals and widespread negative attitudes.

Mothers, in particular, are expected to take responsibility for raising children with disabilities, adding considerable levels of psychological pressure to more general financial and logistical difficulties, which often changes the dynamics within families. The same study noted that it was mothers who were most often expected to take their children to hospital or habilitation/rehabilitation appointments or to accompany them to school. The mothers themselves reported feeling very unsupported by their families, communities and service providers, partly as a result of the negative attitudes prevalent towards disability, which they experienced as feelings of shame.19

Barriers at community levelAt the community level, disability is still subject to deep-rooted taboos. Strong religious beliefs that link disability to past misdemeanours can hamper people’s understanding of what causes impairments.20 While the majority of people recognize that disabilities have congenital and genetic causes or result from accidents or disease (77 per cent), a significant minority believe that disability is caused by the ‘will of God’ (10 per cent); the fault of parents (4 per cent); or fate/karma (2 per cent).21

Welfarist-based thinking is also pervasive and reflected in the language used to describe disability: terms such as cacat, a derogatory word meaning handicapped for example, is still widely used and is often synonymous with the official term Orang Kurang Upaya.22 There is still a strong assumption that having a disability implies a state of abnormality, dependency and a need for specialist provisioning. The provision of services implies an act of charity rather than being an entitlement and, as such, can be poorer in quality, regulation and availability.23

The level of discrimination in Malaysia varies for each type of disability. For example, UNICEF Malaysia (2016) found that children with psychosocial or behavioural disabilities, such as hyperactive disorders, face greater stigma compared with those with physical disabilities. There

18 Study shows that Malaysians don’t understand disability https://www.star2.com/people/2017/11/21/disability-prejudice-malaysia/

19 Sukeri Surianti, Raishan Bakar, and Azizah Othman, ‘Barriers to unmet needs among mothers of children with disabilities in Kelantan, Malaysia: A qualitative study’. Journal of Taibah University Medical Sciences, 2017, vol 12(5), p.424-429 <www.ncbi.nlm.nih.gov/pmc/articles/PMC6695062>

20 Islam Rezaul. ‘Rights of Persons with Disabilities and social exclusion in Malaysia’, International Journal of Social Science and Humanity, vol. 5, no. 2, 2015 <www.ijssh.org/papers/447-H10019.pdf>

21 United Nations Children’s Fund (UNICEF) Malaysia, Childhood Disability in Malaysia: A Study of Knowledge Attitudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017, p.42 <www.unicef.org/malaysia/media/281/file/Childhood%20Disability%20in%20Malaysia.pdf>

22 Ibid.23 Islam Rezaul, ‘Rights of Persons with Disabilities and social exclusion in Malaysia’, 2015.


5

is a lack of social acceptance of children with these impairments, even if they manage to enrol in mainstreams schools.24

Nevertheless, there has been some transition in awareness and perception of children with disabilities. Use of the term ‘special’, which is typically used to refer to children with disabilities, is starting to be recognized as potentially segregating and contributing to the overall experience of exclusion.25

1.2.3. Inclusion in education for children with disabilities While the Ministry of Education uses the term ‘inclusive education’ (for example, in the Malaysia Education Blueprint 2013–2025) it does not actually follow an internationally recognized inclusive system. Instead, it operates a special education/integrated education model offering essentially three education pathways for children with disabilities:

•Special needs schools for specific impairments, such as visual and hearing impairments; •Special education integrated programme, which provides specialist classes within

mainstream government or government-aided schools; •Inclusive education programme, where students with and without disabilities are assigned

to the same classes within mainstream schools.

The Education Ministry’s latest target is to have at least 75 per cent of students with disabilities in the Inclusive Education Programme by 2023, with a zero rejection policy.26 However, even if this target is achieved, it still assumes almost a third of students with disabilities are not expected to be in mainstream education programmes and, given there are no disability-disaggregated statistics available for the prevalence of disability in school-age populations, it is a target that will be hard to track. Nevertheless, as the total investment allocated for education is high (RM42.9 billion) the Ministry has plans to carry out mid-term reviews of the Education Blueprint to identify areas for improvement.27

The Inclusive Education Programme is recognized as a model that demonstrates inclusive education is achievable. However, the challenges around implementation are significant, for example, the current shortage of qualified teachers and accompanying professionals such as therapists and psychologists; the lack of an adaptable curriculum; inadequate provision of assistive technology; and inaccessible school facilities. There have been proposals for case studies and data analysis to review existing practices to determine and evaluate any gaps identified. However, it would seem that the Inclusive Education Programme remains difficult to implement.28

24 Malaymail, Malaysians meaner towards children with mental disabilities, Unicef study shows, 16 October 2017 <www.malaymail.com/news/malaysia/2017/10/16/malaysians-meaner-towards-children-with-mental-disabilities-unicef-study-sh/1487855>

25 When we use the word ‘special’ too much https://www.freemalaysiatoday.com/category/opinion/2018/12/03/when-we-use-the-word-special-too- much/

26 Ministry of Education, Malaysia, Malaysia Education Blueprint 2013–2025, Annual Report 2017, MoE Malaysia, Kuala Lumpur, 2018 <www.padu.edu.my/wp-content/uploads/2018/07/AR2017-English-PPPM-.pdf>

27 Inclusive education surpasses target https://www.thestar.com.my/news/education/2018/08/05/inclusive- education-surpasses-target/

28 Of kids and inclusive learning https://www.thestar.com.my/news/education/2014/12/21/of-kids-and- inclusive-learning/


6

Barriers within the education systemIn addition to structural issues around implementation of the Inclusive Education Programme, there are also considerable attitudinal barriers to overcome in the education system. Parents themselves may be reluctant to enrol their child in a mainstream school, preferring instead the perceived security of a special school education. There is a strong sense that mainstream teachers would not be able to provide an appropriate learning environment, especially for children who have hearing or visual impairments. Moreover, when mainstream schools feel under pressure to meet government performance targets, they may refuse to enrol children with disabilities. Teachers themselves may be reluctant to include children with disabilities because of a lack of training, appropriate materials or classroom support.29

Most recently, to promote inclusive education, the Education Ministry produced a directive abolishing streaming in mainstream education (i.e. the placement of students in classrooms based on their academic performance). Theoretically, this should encourage schools to compete based on their level of inclusiveness instead of on academic achievements, but this requires further public awareness and information. The benefits to students both with and without disabilities need to be highlighted as many parents and teachers remain concerned that with the abolishment of streaming, fast learners could be restricted.30

It has been shown that inclusive education benefits students without disabilities, perhaps due to exposure to different ways of experiencing the world, but more awareness-raising is needed to break the deep-rooted stigma attached to disability.31 A systematic review of studies showed that, for the most part, children without disabilities did not experience any negative effects from being in the same classroom as their peers with a disability. In some cases, academic performance overall improved, although this only applied to classrooms that did not have multiple students with severe emotional and behavioural disabilities. Overall, studies suggest that it is the quality of instruction and the collaborative teaching of special and general education teachers that plays a more significant role in educational outcomes for children with disabilities than the type of classroom a student is placed in (inclusive or non-inclusive).32

Inclusion at preschool levelThere have been recent moves to make inclusive education mandatory at preschool level. The benefits of implementing inclusion at this level have been widely promoted, such as to enable early identification and intervention at a critical stage for children with disabilities to receive support and reduce challenges in later years. Early intervention could also be made affordable if inclusive preschool education is compulsory. The Government could subsidize the cost of special education, including private preschools. However, some challenges remain, such as the lack of special needs teachers to play a role as curriculum specialists by providing specialized instructions to supplement the basic curriculum in preschools. More special needs teachers must be trained, as well as training general education teachers.33 Nevertheless, plans are ongoing for

29 United Nations Children’s Fund (UNICEF) Malaysia, Childhood Disability in Malaysia: A Study of Knowledge Attitudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017 <www.unicef.org/malaysia/media/281/file/Childhood%20Disability%20in%20Malaysia.pdf>

30 Abolish streaming in schools leads to caring, inclusive society https://www.nst.com.my/opinion/letters/2018/01/320519/abolishing-streaming-schools-leads-caring- inclusive-society

31 Inclusive education has improved special children’s quality of life https://www.thestar.com.my/metro/metro-news/2017/11/14/inclusive-education-has-improved-special- childrens-quality-of-life/

32 Grindal, Todd, Thomas Hehir and Brian Freeman, A Summary of the Research Evidence on Inclusive Education, December 2016 <www.researchgate.net/publication/312084483_A_Summary_of_the_Research_Evidence_on_Inclusive_Education>

33 Carol Clemant Devaraj et al, SEN children – making preschool inclusion mandatory, 02 July 2018, <www.malaysiakini.com/letters/432301>


7

the Education Ministry’s special needs officers at the district level to train teachers at all public and private preschools.34

Various models of inclusive education have been put forward to raise awareness and demonstrate possibilities for successful implementation in Malaysia, particularly with the increasing number of children with autism. One example is the Integrated Play Group (IPG) model that was developed to encourage socialization and improve autistic children’s communication with other students in natural settings. IPG pilot programmes have shown positive outcomes, such as mutual friendships. The IPG requires individualization of education programmes tailored to each student’s needs and abilities.35

The bi-annual National Early Childhood Intervention Conference is another initiative to increase the awareness and acceptance of children with disabilities. The 2018 event (entitled: One Child, One Family, One Community) was attended by various stakeholders and provided guidance for current services for children with disabilities, which encompassed preschool and school inclusion, parent empowerment, training of professionals and balance to the private sector.36

1.2.4. Official data on children with disabilities in MalaysiaData on the numbers of children with disabilities in all contexts in Malaysia remain unreliable. The main estimates are still taken from the 2011 World Report on Disability, which suggests that 5.1 per cent of children (0–14 years) have a disability, with around 0.7 per cent of those children experiencing ‘severe disability’.37

Available data suggest that one in three children with disabilities is out of school, compared with one in seven of their peers without disabilities. Children with disabilities are likely to experience a range of barriers depending on factors such as degree and type of impairment, age, location, gender and ethnicity. For girls especially, having a disability can seriously impact on participation; globally girls with disabilities are less likely to be successful in education, to benefit from vocational training or to find employment, even compared with young men with disabilities or girls without disabilities.38 For example, estimates of primary school completion rates show just 42 per cent of girls with disabilities complete primary education compared with 51 per cent of boys with disabilities.39

Currently, Malaysia does not collect household-level data on disability, so there is no reliable measure of prevalence in children. Given a current 0–14 years population of around 9 million,40

global prevalence rates would suggest there could be around 440,000 children with disabilities in Malaysia,41 although rates of 10–16 per cent have also been reported.42

34 The Star, MoE vows to help special needs students thrive, 02 September 2018 <www.thestar.com.my/news/nation/2018/09/02/moe-vows-to-help-special-needs-students-thrive/>

35 New Straits Times, Focus on ability, not disability, 2017, <www.nst.com.my/news/2017/03/150095/focus-ability-not-disability>

36 Dr Amar-Singh and Dr Wong Woan-Yiing, The way forward for children with disabilities in Malaysia https://www.malaysiakini.com/letters/364895

37 WHO, World Report on Disability, 2011, accessed June 27 2018, http://www.who.int/disabilities/world_report/2011/en/. 38 Groce, Nora Ellen, ‘Adolescents and Youth with Disabilities: Issues and challenges’, Asia Pacific Disability

Rehabilitation Journal, vol. 15, no. 2, July 2004, pp. 13–32. 39 WHO, World Report on Disability, 2011, p.12, accessed June 27 2018.40 index mundi, Malaysia Demographics Profile 2019, <www.indexmundi.com/malaysia/demographics_profile.html>41 That is 5% of 9,000,00042 Sukeri Surianti, Raishan Bakar, and Azizah Othman, Barriers to unmet needs among mothers of children with

disabilities in Kelantan, Malaysia: A qualitative study. Journal of Taibah University Medical Sciences, 2017, vol 12(5), p.424-429 <www.ncbi.nlm.nih.gov/pmc/articles/PMC6695062>


8

It is worth noting that in 2018 UNICEF commissioned the translation of the Washington Group Child Functioning question set into Bahasa Malaysia to make it easier for the Department of Statistics to collect child disability data in its household survey. The Washington Group data sets have been designed specifically to enable data on disability to be sensitively collected within a rights-based framework; UNICEF has worked on producing the Child Functioning sets to ensure more accurate recording of disability in children.43

The latest statistics on the official website of the Department of Social Welfare report that there were 114,933 persons with disabilities under the age of 18 in 2016.44 The reason the rates appear so low is that this data reflects the numbers of children officially registered as having a disability as part of the OKU identity card system. OKU or Orang Kurang Upaya is the official terminology used to describe people with disabilities and roughly translates to ‘less-abled person’.45

Under the Persons with Disabilities Act (2008), a person is defined as disabled when they ‘.... have long-term physical, mental, intellectual or sensory impairments, which, in interaction with various barriers, may hinder their full and effective participation in society.’ However, for OKU registration, the Government is much more prescriptive and defines seven types of impairments that can be categorized as a disability: hearing (which it further refines by decibels of hearing loss); sight; speech difficulties (which in children have to be assessed at 5 years or above); physical; learning disabilities (which includes Down’s syndrome, autism and other learning difficulties); mental (which includes diagnosed psychiatric conditions); and multiple (where there is more than one significant impairment present).46

Since this is a voluntary process with minimal incentives, and one that can be difficult to navigate or even feel shameful, many parents choose not to register their children. Efforts are underway to encourage registration of children with disabilities47 and to highlight the importance of registering.48 These include initiatives to increase awareness of the benefits of registering such as financial assistance, subsidies, job attachments and places at education institutions.49 However, until there is a national survey using internationally recognized data collection tools (such as the Washington Group question sets) rates of disability in children will remain under-reported.

Consistent with a lack of prevalence data is a general lack of empirical research relating to the voice and lived experiences of children and young people with disabilities in Malaysia. In 2016, UNICEF Malaysia commissioned an important population-based KAP Survey to try to address this knowledge gap; this provides some key observations that are of relevance to this evaluation.50

43 See: http://www.washingtongroup-disability.com/44 Department of Social Welfare Statistics Report 2016 , <www.jkm.gov.my/jkm/uploads/files/penerbitan/Buku%20

statistik%202016.pdf >45 United Nations Children’s Fund (UNICEF) Malaysia, Childhood Disability in Malaysia: A Study of Knowledge Attitudes

and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017, p.3 <www.unicef.org/malaysia/media/281/file/Childhood%20Disability%20in%20Malaysia.pdf>

46 Department of Social Welfare web portal http://www.jkm.gov.my/jkm/index.php?r=portal/left&id=UnN2U3dtUH-hacVN4aHNPbUlPayt2QT09. Worth noting that the government is determined to stop use of the term ‘mental dis-ability’ and adopt the more rights based psychosocial impairment terminology. This has yet to be publicly visible.

47 Register with welfare department to be eligible for benefits, urges deputy minister https://www.thestar.com.my/metro/metro-news/2018/07/24/record-on-the-disabled-inaccurate-register- with-welfare-department-to-be-eligible-for-benefits-urges/

48 The importance of registering a disabled child http://www.theborneopost.com/2013/06/08/the-importance- of-registering-a-disabled-child/

49 Parents urged to register children with disabilities https://www.nst.com.my/news/2015/09/parents-urged- register-children-disabilities

50 UNICEF Malaysia, Childhood disability in Malaysia, 2017.


9

This survey noted that the medical model is still the dominant approach to understanding disability in Malaysia, and has helped maintain a situation where children and young people with disabilities remain: ‘passive recipients of services’51 with no sense of them having individual agency. Generally, people do not focus on barriers to inclusion but instead tend towards viewing the impairment as being the barrier to participation. Therefore, while government policy could be said to align with a rights-based approach, this is not well understood at community level. The underlying drivers for stigma and discrimination remain essentially unchallenged.

1.3. #ThisAbility Flagship Initiative strategic goals Working within and responding to the national context outlined in the previous sections, the strategic direction for UNICEF Malaysia’s #ThisAbility Flagship Initiative is provided in a Working Paper and accompanying Monitoring Framework documents.52 The overall goal of the Initiative is ambitious and forward-looking and underpinned by six key objectives that focus on changing the lives of children with disabilities (see Figure 2).

Figure 2: Goals and objectives of the #ThisAbility Flagship Initiative

Goal

UNICEF commits to making positive changes for CwD [sic] in Malaysia so they are free from discrimination and able to access services to support their growth, development and protection.

Objectives

a) Children with disabilities have increased access to high quality services;b) Children with disabilities do not experience abuse and discrimination;c) Children and adolescents with disabilities are empowered to advocate for themselves;d) General public are aware of children with disabilities; e) Children with disabilities are socially included;f) Children with disabilities can contribute positively in society.

51 Ibid. p. 8552 #ThisAbility Flagship working paper; #ThisAbility Flagship taskforce monitoring framework (2017–2020).


10

The #ThisAbility Flagship Initiative spans the period from 2017 to 2020, therefore this evaluation is akin to a mid-term review. The chronological order of key events is shown in Figure 3:

Figure 3: Summary of key events under the #ThisAbility Flagship Initiative

2017 2018 2019• Launch of #ThisAbility Flagship Initiative

• Disability Actors Database compiled

• Public Advocacy Campaign for Borneo International Marathon

• RAGE videos showcase Paralympians

• ManUnited campaign launch in combination with Borneo International Marathon

• Disability Equality Training• Petrosains – Inclusion Champion

• #ThisAbility Makeathon• Launch of KAP Survey• NGO Impact Academy• Empowering Teachers and Trainers training

• #ThisAbility truck• Disability Equality Training• Public Advocacy Campaign for Borneo International Marathon

• Sime Darby Properties – Inclusion Champion

• Inclusive Playground inception – design

• We are all special inclusive carnival

• What’s so special international day of persons with disability campaign

• What’s so special about Ima? U-Report pick a path story

• National conference on inclusion at National Early Childhood Intervention Council

• What’s so special about special education call to action

Some key events were nationwide, but many were concentrated in Kota Kinabalu and Sabah. A financial budget for the Initiative was not available and, while supported across UNICEF Malaysia, the Initiative was primarily led by a Disability Consultant.

Part of the mandate for this evaluation was to assist UNICEF Malaysia to create a Theory of Change (ToC) for future disability inclusion activities undertaken by the Country Programme. The resulting ToC is presented in Annex F. There was not a results framework available for consideration by this evaluation and therefore the ToC developed by the external evaluators is the first attempt to articulate a logical results chain for UNICEF Malaysia’s disability inclusion work. UNICEF Malaysia will need to decide how to use the ToC.


11

2. EVALUATION PURPOSE, OBJECTIVES AND SCOPE

2.1. PurposeThe primary purpose of this formative, forward-looking evaluation was to draw lessons from a focused analysis of the #ThisAbility Flagship Initiative to identify how UNICEF Malaysia might transition from being an organization that undertakes disability-focused activities, to one that fully embeds disability in its work.53 The evaluation sought to identify and highlight knowledge generated by the Initiative to inform UNICEF Malaysia’s current and emerging programmes.

The relatively nascent nature of the Initiative meant that the evaluation focused on finding evidence of outputs and early-stage outcomes, not on measuring the impact of the Initiative.

The primary users of this evaluation are expected to include UNICEF Malaysia as well as the Government of Malaysia and implementing partners. It is expected that they will utilize the conclusions and recommendations to shape the forthcoming CPD and to inform the development of new partnerships, programmes and initiatives.

Secondary users are expected to include but not be limited to: other agencies working on disability-inclusive programmes for children and young people; UNICEF’s East Asia and Pacific Regional Office; UNICEF headquarters (most especially the Disability and Child Protection Divisions); and the parents of and children with disabilities in Malaysia. Secondary users of this evaluation could utilize the conclusions and lessons learned to strengthen their own disability-inclusion work, identify potential synergies and/or identify opportunities for collaboration.

This evaluation recognizes that the findings can be used to bring UNICEF Malaysia to the forefront of disability inclusion, setting an example for key partners in Malaysia and other countries, as well as providing fresh evidence and impetus to support UNICEF’s strategic focus on equity.

2.2. Evaluation objectivesThe four specific objectives of the evaluation, as defined in the Terms of Reference (ToR) (see Annex A) were to:

•analyse the strategies used to map out the trajectory of the #ThisAbility Flagship, and re-construct the Theory of Change (ToC) behind making positive changes for children with disabilities in Malaysia;

•assess the extent to which the #ThisAbility Flagship has been appropriately designed, efficiently and effectively managed, and integrated across the Country Programme, including the incorporation of equity, gender equality and human rights considerations;

53 During the finalization of this report UNICEF staff emphasized that the #ThisAbility Flagship Initiative was “established as a platform to encourage sections to work collaboratively on disability-related initiatives. It was designed to test out innovative ways of approaching disability in UNICEF’s work. It also intended to build confidence of staff and partners to be disability-inclusive in their approach to designing and implementing initiatives and programmes”.


12

•examine the evolution of UNICEF’s work on children with disabilities in Malaysia, and in particular the intermediate results achieved by UNICEF’s #ThisAbility Flagship, enabling and disabling factors, considering aspects of both reduction of discrimination as well as access to services – including advocacy and awareness-raising both at the national and sub-national level; and its likely sustainability;

•analyse the utility of creating a flagship on specific focus issues to promote cross-sectoral work, including strengths and weaknesses.

2.3. ScopeAt the request of UNICEF’s Evaluation Specialist, the external evaluators paid particular attention to the following 10 activities (see Figure 4) when analysing the effectiveness, relevance, efficiency and sustainability of the aims and objectives in the Initiative Working Paper and when addressing the key evaluation questions (see Section 2.4). A discussion around the utility of the #ThisAbility Flagship Initiative is detailed in Section 4.5.

Figure 4: #ThisAbility Flagship Initiative events analysed by the evaluation team

Event Dates Location StakeholdersDisability Equality Training 2017/18 Kuala Lumpur and

Kota KinabaluIncluded participants from: Petrosains, UNICEF, Tandemic, Sime Darby Property and Sabah Art Gallery

Inclusive Journalism Workshop

2017 Kuala Lumpur, Kota Kinabalu and Kuching

Journalists/members of Malaysian Press Institute

#ThisAbility Public Advocacy Campaign

2017/18 Nationwide General public

Knowledge, Attitudes and Practices Survey launch/dissemination

2017 Kuala Lumpur Children with disabilities

#ThisAbility Makeathon and video

2017 Nationwide, event hosted in Kuala Lumpur

Children with and without disabilities

Non-governmental organization (NGO) Impact Academy

2017 Nationwide, event hosted in Kuala Lumpur

Nationwide disability and women’s rights civil society organizations

The Borneo International Marathon #ThisAbility themed

2017 International event hosted in Kota Kinabalu

General public, children with and without disabilities

Corporate Inclusion Champions

2017/18 Kuala Lumpur and Shah Alam

Petrosains and Sime Darby Property

We Are All Special: A Carnival for All

2018 Kota Kinabalu, Sabah

Children with and without disabilities

Inclusive Play, 2018/19 Shah Alam Children with and without disabilities


13

2.4. Key evaluation questionsThe key evaluation questions (see Figure 5) and the evaluation criteria were described in the original ToR (see Annex A) and are based on UNICEF Malaysia’s priorities. The questions were mapped to relevant data sources in the Evaluation Matrix (see Annex B).

Figure 5: Evaluation questions

Organisation for Economic Cooperation and Development (OECD)/ Development Assistance Committee (DAC) criteria

Sub-questions

Effectiveness of UNICEF’s strategy to children with disability and its implementation considering:

To what degree has UNICEF’s work on disability, and in particular through its #ThisAbility Flagship Initiative, contributed to the creation of positive conditions to reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia? Have there been any unintended results?

•How satisfied are the children and their caregivers with the quality of the services they have received through UNICEF’s partners?

•To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights-based perspective?

•How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice?

•How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included?

•To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities?

•To what extent has the #ThisAbility Flagship Initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities?

•Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship Initiative?


14

Relevance of the programme strategy and approach, considering:

How relevant is UNICEF’s work on disability and its #ThisAbility Flagship Initiative to the developmental challenges in Malaysia, and particularly attitudes and practices with regard to disability?

• How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the Sustainable Development Goal (SDG) roadmap?

• How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050?

• To what extent is #ThisAbility Flagship Initiative aligned with the new UNICEF’s Strategic Plan?

• Has the design of the #ThisAbility Flagship Initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable?

• Have the voices of children and adolescents with disabilities taken into account throughout?

Efficiency of the programme implementation and management, considering:

To what extent and how has UNICEF mobilized and used its resources (human, technical and financial) and improved its cross-sectoral coordination to achieve its planned results for children with disabilities?

• To what degree have UNICEF resources (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility Flagship Initiative?

• How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?

Sustainability in the programme, considering:

To what extent are the benefits and achievements of the #ThisAbility Flagship Initiative likely to continue after 2020 through national ownership and changes in terms of attitudes, knowledge and practices?

• To what degree has the #ThisAbility Flagship Initiative thus far contributed to the generation of sustainable capacities at the national and sub-national levels?

• What are the enabling as well as constraining factors that are likely to influence future cross-sectoral work on focus issues?

During the inception phase, in which an initial round of key informant interviews (KIIs), internal document review, and an on-site visit took place, it became clear that the evaluation would not be able to assess efficiency as it relates to financial management. This was because UNICEF Malaysia did not hold financial or output monitoring information that could be used as the basis for such an assessment. Therefore, the evaluation process did not carry out a value for money assessment.

Moreover, the evaluation’s approach to assessing the effectiveness and sustainability of the Initiative was also relatively limited. This was partly due to the early-stage nature of the Initiative, but mostly because of the disconnect between the stated aims of the Initiative – as a mechanism for providing UNICEF Malaysia with a platform for developing its internal capacity for work on disability inclusion – and the goals and objectives as outlined in the #ThisAbility Flagship Initiative Working Paper (see Figure 2). This issue is explored in full in Section 4. While being aware of the above constraints, the evaluation team made no changes to the evaluation objectives (or evaluation questions) following a request by UNICEF to use the same objectives and evaluation questions as those in the ToR to the fullest extent possible.


15

3. EVALUATION METHODOLOGY

3.1. Design and methodsThe evaluation followed a mixed-methods approach, non-experimental design to addressing the key evaluation questions. This evaluation drew upon three key strands of data collection:

I. Three focus group discussions (FGDs) were conducted with 11 pairs of children and young people with disabilities and their parents/carers. The groups consisted of five male and six female children with disabilities (one participant aged 0–5 years, five participants aged 6–12 years and five participants aged 13–19 years). The participants included those with cognitive, mobility and hearing impairments. A key aim of this research strand was to provide insights into the experiences and barriers faced by children with disabilities and their families. Focus groups were conducted in Klang Valley, Sabah, and Sarawak, where UNICEF Malaysia activities have been most prominent, or where there is intention to do more. Each session took around 120 minutes and was based on a topic guide agreed with UNICEF Malaysia during the inception phase.

II. Forty-two key informant interviews (KIIs) were completed with staff of partner organizations (including mainstream organizations and disability organizations), UNICEF staff (including staff working on any disability-related programming, finance, and human resources) and three Government representatives. The interviews were conducted face-to-face, or via telephone or Skype; all interviews were guided by semi-structured interview questions that reflected the evaluation questions and were aligned with the evaluation matrix, utilizing a topic guide agreed with UNICEF Malaysia during the inception phase. Each interview was recorded, transcribed and translated to English (when necessary) to aid further analysis.

III. A desk review of internal (UNICEF) documents provided by UNICEF Malaysia enabled the external evaluators to familiarize themselves with all aspects of the #ThisAbility Flagship Initiative. A desk review of external documents (including national and international strategies) supplemented the review of internal documents, providing greater context around disability inclusion in Malaysia and the relevance of the #ThisAbility Flagship Initiative. The document review was guided by the evaluation matrix (see Annex B), which identified the potential use of each document for addressing the evaluation questions. In total, the evaluation team reviewed more than 180 documents.

The evaluation team agreed the above three strands of data collection with UNICEF through an inception report that went through several rounds of review and revision; this was agreed with UNICEF’s Evaluation Specialist before data collection.

3.2. SamplingParticipants for the FGDs were selected from a sample frame of Initiative participants provided by UNICEF Malaysia. The external evaluators used purposive sampling of children and their parents/carers to ensure diverse perspectives on the Initiative, using pre-defined criteria agreed with UNICEF Malaysia. This ensured a mix of participants across gender, age and type of impairment. Participant children’s ages ranged from 5–19 years; overall 5 female and 6 male children participated; and types of children’s impairments included cerebral palsy, hearing


16

impairment, attention deficit hyperactivity disorder (ADHD), autism, and Down’s syndrome. All participants were living in, or close to, the respective market centres (Klang Valley, Kota Kinabalu and Kuching); UNICEF selected these areas because they were where the key activities that were the focus of this evaluation took place (see Section 2.3) and/or they were the three main areas of focus communicated by UNICEF.

UNICEF’s Evaluation Specialist and UNICEF Malaysia’s Disability Specialist provided a sampling frame for the KIIs. The external evaluators used purposive sampling to ensure different viewpoints were sufficiently captured from a range of programme implementers, partners and other stakeholders. The external evaluators sought to interview more Government stakeholders but they were not available, despite support from UNICEF Malaysia to contact them and an extension to the data collection period. The full list of interviewees is available in Annex D.

3.3. Data management and handlingThe external evaluators put a series of processes and systems in place with the specific intent of ensuring high quality data collection and analysis:

• Triangulation of information: In preparing this evaluation report, the external evaluators triangulated data collected. FGDs and KIIs were recorded and the transcribed data was analysed by using an evaluation matrix (see Annex B) derived from the main evaluation questions. The external evaluators also held regular team discussions to jointly discuss and triangulate information, validate findings, and identify areas for further exploration throughout the data collection process.

• Quality assurance: Interviews and FGDs were undertaken by two team members and were immediately followed by debrief sessions. A Director from Equal International and from Ipsos MORI reviewed all evaluation tools and outputs. The evaluation approach and associated tools were finalized following multiple reviews by UNICEF’s Evaluation Specialist. UNICEF Malaysia also established a Reference Group consisting of selected UNICEF staff and external partners to support the delivery of the evaluation.

3.4. Approach to equity, gender and human rights The external evaluators followed the CRPD in promoting a human rights-based understanding of disability. Children and young people with disabilities are defined as those aged 18 years and under:

‘.... who have long-term physical, mental, intellectual or sensory impairments which in interaction with various barriers may hinder their full and effective participation in society on an equal basis with others.’

– (Article 1, CRPD).

This concept of disability moves away from the traditional individual, medical-based perspective characterised by a focus on physical deficits (impairments), to one that encompasses the attitudinal, environmental and institutional barriers that limit or exclude children and young people with impairments from participation. In the context of this evaluation process, it meant a rights-based approach to the analysis was taken, starting from the premise that children and young people with disabilities have the right to participate in and benefit from all development actions.


17

To fully examine the #ThisAbility Flagship Initiative, data was analysed against key barriers to inclusion, those that are attitudinal, environmental and institutional. This analysis undertook to identify barriers experienced by children and young people with disabilities in the context of the Initiative, and was used to help determine whether or not the Initiative addressed issues of relevance to programme participants.

Taking a rights-based approach encompasses not just disability but also issues around gender, age and other contextual factors. The focus of this evaluation was to understand the situation in terms of barriers, rather than needs, and to look at the situation from the context of both the rights holders and the duty bearers. Conclusions and lessons learned stem from a focus on equity and equality rather than needs and problems and are set within the context outlined by the CRPD.

3.5. Ethical issues and considerationsThis evaluation was completed with reference to the United Nations Evaluation Group (UNEG) Norms and Standards (2016), with particular attention given to Norm 6 on ethics and Norm 9 on human rights and gender equality. Reference was also made throughout this evaluation to the Revised Evaluation Policy of UNICEF (2018); UNEG Ethical Guideline for Evaluation (2008); Ethical research Involving Children (2013); UNICEF Procedure for Ethical Standards and Research, Evaluation and Data Collection and Analysis (2015); UNICEF-adapted UNEG Evaluation Reports Standards (2017); the Global Evaluation Reports Oversight System Handbook (2017) and UNEG Guidance on Integrating Human Rights and Gender Equality in Evaluation (2014). Ipsos is also a member of the Market Research Society, which has a Code of Conduct governing the design and implementation of all its research. In drawing upon the above guidance documents, the external evaluation upheld the following key principles:

• Independence: The evaluation is free from bias.• Impartiality: The external evaluators have given a balanced view of both strengths and

weaknesses observed. • Credibility: The evaluation is based upon reliable data and observations. • Conflicts of interest: The external evaluators avoided any conflicts of interest that threatened

the credibility of the evaluation.• Accountability: External evaluators recognize their responsibilities and have accurately

represented themselves and their data, and exercised prudence throughout the evaluation.

Informed consent protocols ensured that all participants received a comprehensive briefing on the evaluation aims and how the data collected would be used/stored, and they were then required to provide explicit consent to proceed. Information sheets and consent protocols, including a privacy note, were prepared in collaboration with UNICEF’s Evaluation Specialist before data collection.

The external evaluators ensured strict data confidentiality throughout the evaluation. For example, all KIIs were given a unique code and this final evaluation report contains only anonymized data that cannot be attributed to specific individuals. Personal data files held by the external evaluators will be destroyed upon formal completion of this evaluation. Primary data was collected by Ipsos Malaysia, and Ipsos MORI ensured compliance of all research and analysis processes with its Business Excellence System.


18

3.6. Methodological limitationsThe completion of this evaluation was subject to a series of challenges and limitations that are important to note:

•Absence of strategic and monitoring information in programme documentation: There was an absence of baseline information, a lack of systematic monitoring data and no impact indicators related to the objectives. The internal documents provided by UNICEF for review gave no sense of scope or scale to the #ThisAbility Flagship Initiative, no geographical information or description of the target population and no disaggregation by intersectional markers (such as gender, ethnicity, geographical location or age). This made it challenging to fairly measure the progress of the Initiative against the recommended Development Assistance Committee (DAC) criteria of relevance, effectiveness, efficiency and sustainability, as stipulated in the ToR. In particular, the Initiative documentation included no overall programme-based outcome or impact reporting, nor baseline statistics against which this evaluation could track progress of the Initiative. Instead, the activity-level focus of the Monitoring Framework meant this evaluation had to focus on the delivery of individual activities.

•Lack of financial data: Financial data, including an overall budget for the Initiative, were not available to measure efficiency or value for money.

•Reliance on qualitative data collection: The absence of programme-based outcome or impact reporting meant that this evaluation relied upon the collection of primary data through KIIs and FGDs to assess progress. A reliance on research participant information for programmatic analysis can be restricting insofar as participants tend to have limited experiences. The external evaluators sought to mitigate this through saturation of potential participants and triangulating the data. A reliance on research participant information also hinges upon the availability of participants (as already stated many Government representatives were not available for interview despite multiple attempts by the external evaluators and UNICEF Malaysia) and a sufficient number of participants (resource constraints particularly limited the number of FGD participants).

•Objectivity: At UNICEF Malaysia’s request, a member of UNICEF’s Evaluation Team joined the external evaluators in the FGDs. The proximity of the UNICEF staff member to a key strand of data collection increased the risk that participants’ responses would be biased and therefore objectivity could be compromised; this risk was discussed with UNICEF Malaysia. In ensuring participants’ informed consent, it was explained throughout recruitment protocols and during the FGDs that a representative from UNICEF Malaysia’s Evaluation Management Team was present as an observer. The UNICEF staff member did not participate in the discussion; they were there only to observe. The approach adopted by the external evaluators to triangulate information and validate findings did not include the member of UNICEF’s Evaluation Team.

While not a methodological limitation, it should also be noted that it is the firm belief of the external evaluators that UNICEF’s request for this evaluation to stick rigidly to the evaluation questions and criteria in the ToR has restricted its potential value. Recommendations made by the external evaluators at the end of the inception stage and informally accepted by UNICEF Malaysia at that time, were abandoned at the request of UNICEF’s Evaluation Specialist. Had the external evaluators been afforded the scope to conduct this evaluation as an evaluability assessment, they could have taken the iterative and loose construction of the Initiative into account more and focused analysis on providing strategic guidance for UNICEF Malaysia’s future work around disability inclusion, and an associated monitoring and evaluation strategy. As it is, the evaluation was constrained by the summative nature of the key evaluation questions, which required assessment of the Initiative against its intended long-term objectives.


19

4. FINDINGS

4.1. EffectivenessBox 1: Effectiveness questions

How satisfied are the children and their caregivers with the quality of the services they have received through UNICEF’s partners?

To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights-based perspective?

How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice?

How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included?

To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities?

To what extent has the #ThisAbility Flagship Initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities?

Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship Initiative?

Summary: Most of those directly involved in activities implemented under the #ThisAbility Flagship Initiative found them to be interesting, exciting and of benefit to the children with disabilities who took part. Since the activities were all one-off events, however, there has so far been little opportunity for effecting lasting attitudinal change and no clear evidence of effectiveness in reducing discrimination and enabling access to services by children and adolescents with disabilities. The lack of an advocacy strategy, behavioural change activities, follow-on activities, and an absence of monitoring structures (including the collection of disability-disaggregated data), mean that UNICEF Malaysia is currently unable to determine whether its interventions are having an impact on changing attitudes. The disconnect between the #ThisAbility Flagship Initiative goals and objectives and the intent for the Initiative to promote mainstreaming of disability programming within the UNICEF Malaysia office has limited the effectiveness of this Initiative in all respects.


20

This evaluation concluded that activities carried out under the Initiative cannot be fairly evaluated against effectiveness as anticipated by the evaluation questions. The evaluation questions require a focus on the Initiative’s anticipated results, which are to ‘reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia’. The objectives in the Working Paper54 are entirely focused on systemic-level outcomes requiring changes to the way services are designed and delivered.

Initiative activities were not focused on addressing service design and delivery and instead appeared to focus on a narrow range of attitudinal and behavioural changes. UNICEF had not collected data that would enable it to demonstrate how each of the activities contributed to any of these higher-level objectives. Staff and partner informants confirmed that individual activities were not monitored at outcome or impact level, with the absence of this level of reporting leading to some degree of frustration among staff respondents. A key point to emphasize here is that there was no baseline and no programme targets or milestones against which to measure the degree of effectiveness achieved by UNICEF Malaysia in reducing discrimination and access to services more broadly, as stated in the evaluation questions.

4.1.1. Satisfaction levels and public awarenessEvaluation question: Summary of findings:How satisfied are the children and their caregivers with the quality of the services they have received through UNICEF’s partners?

• Events were fun, exciting and of personal benefit• Lack of follow through resulted in a loss of

momentum for effecting longer-term change

Because the children with disabilities and their families that participated in various Initiative activities were not in receipt of any direct services provided by UNICEF Malaysia or its partners, this part of the evaluation question could not be answered. Therefore, rather than ‘access to services provided by UNICEF’, this evaluation focused on satisfaction with the activities delivered under the Initiative, and identification of any outcomes resulting from these activities.

Across all regions, parents of children with disabilities were found to have some general awareness of Initiative activities. It has to be noted, however, that this is partly because the participant referral process was done through UNICEF Malaysia and doesn’t necessarily indicate awareness in general.

SabahIn Sabah, the Borneo International Marathon (BIM) and the We are all special Carnival were the activities that generated the most responses in FGDs. However, in FDGs in Kuala Lumpur and Kuching, these Sabah-based events were not well known, suggesting the impact was geographically limited.

The 2017 BIM was the first opportunity for UNICEF to launch a public awareness campaign under the #ThisAbility Flagship Initiative (see Box 2). Several key disability organizations in Sabah were involved, including Cheshire Homes Services, Special Olympics, Child Sabah and Sabah Society for the Deaf. Their engagement was primarily focused on promoting the participation of children with disabilities from within their constituencies. Several key informants noted that while they had been asked to provide inputs to the organising committee around positive messaging, their suggestions were not taken up.

54 #ThisAbility Flagship working paper, p.2


21

Box 2: Case study illustration of a public awareness event

Borneo International Marathon The Marathon first took place in 2008 and since then has grown in size and reputation. Around 10,000 people now take part in this annual event. Part of the mandate of the Marathon is to raise money for local charities, including those supporting disability. UNICEF has been linked to the Marathon since its inception, but it was not until 2017 that it formally sponsored the event with a #ThisAbility disability-inclusive theme. Primarily a public awareness campaign event, UNICEF also sought to showcase this as an inclusive event through the addition of a new 3km race for children with and without disabilities and their families. Between March and the event in May, mainstream and social media exposure activities helped promote inclusion and challenge biases against children with disabilities in sport. On the day, there were themed t-shirts for runners and speeches and medals for all the child participants. Several local disability organizations mobilized their stakeholders, resulting in around 300 children with disabilities taking part.

Key informants from Sabah reported positive experiences of the BIM. Parents commented that the Marathon had made a difference to them personally, as they got to know other parents who also had children with disabilities. In general, people thought the event was fun, well organized and a good opportunity for children with disabilities to do something outdoors. Parents regarded a lack of outdoor accessible space as a significant barrier to inclusion for children with disabilities; in this regard, they saw the BIM as a positive opportunity for their children to get involved in outdoor activities.

UNICEF/UN068401/Ho


22

In terms of impact, however, the consensus among all groups of informants was that this was a one-off activity that had minimal impact on challenging deep-rooted attitudes. Parents expressed particular disappointment that there had been no ongoing attempts to engage them or to actively follow up on the BIM with any tangible programmes for children with disabilities. There was also concern that people with experience of disability (people with disabilities or the parents of children with disabilities, for example) were not actively engaged in the design or monitoring of the aware-ness-raising elements of this event. In commenting about the impact of the BIM, one focus group participant explained:

“Running is the only thing about this event. Other than that, there is nothing... we could not share the knowledge.”

Key informants were not convinced that the BIM had any impact on changing social attitudes beyond those individuals who participated in the event. One organization felt quite strongly that the event had not challenged stereotypes or removed barriers sufficiently for them to consider taking part again.

It is worth noting that the BIM itself has continued to offer the 3km race for children with disabilities and has in more recent years expanded this to include children with and without disabilities. Several accessibility audits have been carried out that have resulted in gradual improvements to the route and race facilities (for example, including provision for a toilet accessible to persons with disabilities). UNICEF continues to sponsor the event with new themes each year.

The 2018, the We are all special Carnival in Sabah was another awareness-raising event that brought together a wide range of partners to run various events (see Box 3). The Carnival included the Sabah State Library, Sabah Art Gallery, the Jesselton Philharmonic Orchestra and several local disability organizations. Given that working with children with disabilities was new to a number of these agencies, UNICEF provided staff representatives from Sabah State Library, Sabah Art Gallery, and the Jesselton Philharmonic Orchestra with a one-day Disability Equality Training (DET) session.

Box 3: Inclusion in the Sabah Carnival

We are all Special – Sabah CarnivalIn November 2018, Sabah Welfare Services Department, Sabah Community Services Council and UNICEF ran the first ‘Carnival for All’ in Sabah. Another key activity under the #ThisAbility Flagship Initiative, this one-day public event was an opportunity to promote social inclusion, showcase the abilities of children with and without disabilities, challenge negative attitudes and help increase levels of self-esteem and self-confidence among children with disabilities. While the Carnival itself was a regular annual event run by local government, until 2018 it had always been exclusively designed for children with disabilities. This time, the Carnival was purposively designed to include children with and without disabilities participating together in a variety of different activities. Throughout the day, there were fun activities for the children to take part in including arts and crafts workshops; music workshops; a reading corner; team games; a chess competition; and a selection of information stands.


23

Feedback from informants in relation to the Carnival was positive, with the impact felt particularly by those who hosted events. In many instances, this was the first contact people had had with disability, and it led to some level of awareness-raising, especially of improving physical accessibility and in realizing the potential of children with disabilities.

Parents provided positive feedback that their children had the opportunity to mix with their peers without disabilities. However, while parents were very enthusiastic about all the different activities their children could take part in, because things were taking place in different locations (unlike the BIM), they felt they didn’t have the same opportunity to meet and mix with other parents. One significant barrier all parent focus groups raised repeatedly was the lack of support services for them as key carers of children with disabilities. When UNICEF runs inclusive events or activities, usually parents get to meet each other, providing a type of support key carers say they lack; parents commented that UNICEF events have proved beneficial in helping them to gain peer support, learn more about their children’s disabilities and feel slightly less isolated. Unfortunately, because events have not been systematically followed-up with ongoing engagement activities, opportunities to assist parents of children with disabilities have been missed.

Opinions were also mixed as to the Carnival’s impact on improving the social inclusion of children with disabilities more broadly. While it had some effect on raising awareness among the public who attended, it appears to have done little to change perceptions more broadly. Since many people who had been invited already had children with disabilities, the impact on changing wider social attitudes was agreed to have been limited. Parents suggested greater impact could be gained by hosting events in more public spaces, such as in shopping malls, where people without children with disabilities can be more easily reached.

There was also a call for more activities in which children with and without disabilities participate equally. One example raised was the possibility for more outdoor activities like camping or hiking, where there was a genuine effort to be inclusive of all children and where their children’s specific requirements could be accommodated and not treated separately.

In summary, what parents expressed gaining the most from these two key events was being able to come together and share experiences and to see their children taking part in community events which were relatively barrier-free. Given the isolation that families with children with disabilities can face, the value of community participation should not be underestimated. However, these were not the primary stated aims of the activities themselves; rather, these represented positive unintended outcomes.

Kuala LumpurThe Initiative activities most frequently encountered by parents in Kuala Lumpur were the 2017 launch of the KAP Survey, the Makeathon (see Box 4), and participation in the design of an inclusive playground with Sime Darby Property. On the whole, parents had received fewer opportunities to engage directly with the Initiative in Kuala Lumpur because there had been limited events.


24

Box 4: Disability-inclusive Makeathon

The MakeathonIn 2017, UNICEF Malaysia trialled a new activity called a ‘Makeathon’, which was aimed at all children and young people. This was a nationwide competition calling for innovative ideas to assist children with disabilities to participate more fully in their schools, homes and public spaces. Around 20 teams including children with and without disabilities, parents and others submitted ideas that were shortlisted, so that eventually eight teams were invited to participate in the Makeathon event. Petrosains, a well-known national science and education centre in Kuala Lumpur hosted the event, with each team being paired up with private sector experts (including engineers, product designers, 3D modellers and occupational therapists). A range of ideas, from a remote-controlled wheelchair-mounted 360-degree camera, to a wheelchair accessory bag with integrated table-top and a portable computer keyboard underwent prototype design during the Makeathon event itself; the most successful was then developed over a further two-month period. The eventual winning design (a finger-grip device) has since been tested for use by occupational therapists in hospitals and playgroups, although only around 10 products have so far been produced.

As an event, the Makeathon was a popular activity among those who were involved (either as organizer, host, technical specialist or participant). Parents whose children had participated welcomed the activity because they felt it really engaged their children’s creative skills, while at the same time it raised awareness of some of the challenges their children face. Key informants who had been part of the design and delivery of the activity also expressed how much they had learned from engaging directly with young people with disabilities; overall it was viewed as an exciting experience.

The extent to which this activity has been successful in increasing public awareness and challenging negative attitudes, however, was questioned by all groups of informants. The one-off nature of the activity was a key issue in limiting its impact. With no subsequent campaign or follow-up series of events, the momentum both the Makeathon and the launch of the KAP study generated was considered to be lost.

Overall, informants felt that the Makeathon had real potential, but that its effectiveness was diminished due to its one-off nature. Parents, in particular, expressed that while activities like these are exciting, unless there is sustained follow up, there is little chance they will have any effect on changing perceptions or making life more inclusive for their children.

A broad range of key informants and parents also perceived a lack of communication and transparency from UNICEF, with questions raised about how decisions had been made during the Makeathon process. Some of those who participated reported being left wondering what, if anything, might happen next. It was clear from discussions with a range of different stakeholders that UNICEF Malaysia’s communication strategies were not enabling stakeholders to feel engaged.

Informants also reported some frustration with communication regarding UNICEF Malaysia’s work with Sime Darby Property. The Sime Darby Property partnership was a much broader Initiative activity aimed at influencing a significant property development company to build more disability-inclusive spaces (including playgrounds). UNICEF took advantage of Sime Darby’s openness to explore ideas around inclusive spaces and the fact they were willing to set aside an area of land within one of their new developments to trial an inclusive playground. As part of this process, UNICEF invited several disability stakeholders and children with disabilities to take part in a design


25

workshop. This resulted in the submission of plans which could, if approved by local government, become the blueprint for Malaysia’s first inclusive playground. While a potentially significant achievement, informants expressed disappointment that they did not receive feedback and were not kept informed of the process, which made them think that nothing had resulted from their discussions, given the playground had not been built at the time of this evaluation.

Some informants had taken part in discussions and dissemination events in relation to the KAP Survey and gave the study itself a high rating. Once again, the informants raised concerns about what had happened since it was launched. There was a sense among informants that, while the messages and recommendations were excellent; nothing seemed to have been done to follow them up at time of the evaluation.

SarawakUNICEF Malaysia didn’t undertake any Initiative activities in Kuching following a strategic decision in 2019 to focus its interventions in one sub-national geographical location. The intention was to maximize the impact of limited resources and to create evidence that could support changes at both national and sub-national levels across the country.

Parents were aware of Initiative activities that had taken place in other parts of the country, but they pointed to the limited geographic reach of the Initiative and any spillover benefits.

“I don’t see any programme by UNICEF in Kuching other than donation drive.... In our national support group, we know that they have a lot of programme in Kuala Lumpur.”

Parent groups in the area have organized activities for children with disabilities themselves. They described coming together to run an early intervention programme for children under the age of six and a youth group for young people, which includes an element of ‘job empowerment’. Parents in Kuching expressed a degree of dissatisfaction and were somewhat resentful that UNICEF Malaysia had not yet made any commitments to support the area under the Initiative. They nevertheless described UNICEF Malaysia as a good organization that is committed to helping to improve the situation for children (information they had gleaned from fundraising material). There was agreement that UNICEF Malaysia had the power to effect change, but surprise that it took the organization until 2016 to start talking about children with disabilities (suggesting no awareness of UNICEF Malaysia’s work regarding disability outside of the Flagship Initiative). As one participant highlighted:

“Another thing I want to know is why UNICEF just decided to start focusing on disabilities in 2016? There is disabilities many years back, but why in 2016? Your organization is so big, but why you started so late?”

4.1.2. The effectiveness of Disability Equality TrainingEvaluation question: Summary of findings:To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights-based perspective?

• Good quality delivery, appropriately targeted

• Lack of focus on children with disabilities resulted in a missed opportunity for increasing knowledge around children’s rights


26

An expert disability trainer with a disability provided DET. Key informant interviews concurred that the quality of the training was high, and it was well received. Records showed that 103 certificates were awarded to participants in 2017, and 74 awarded in 2018; indicating some 177 people have benefitted from DET. Certificates were awarded to personnel in Petrosains, UNICEF, Tandemic, and Sime Darby Property. In addition, over 90 teachers in Sabah took part in a two-day workshop on ‘Empowering teachers to teach children with disabilities’ in November 2017, during which DET was provided; and KIIs revealed DET had taken place at the Sabah Art Gallery. Over 250 individuals have therefore likely benefitted from DET provided under the Initiative since 2017.

In general, staff and partners found the DET to be very useful in helping them to understand disability from a different perspective. People felt more comfortable and confident in how to manage, collaborate and interact with people with disabilities. There were some who felt that without any follow-up activities the impact of the training was limited. Some also felt that it could more usefully be broadened to take place in schools so that children can learn more about disability.

Some also sensed that the DET was too focused on the needs of persons with physical disabilities, while others noted that it was at quite a basic level, and therefore may not be appropriate for all stakeholders. From the DET slides, the focus of the training appears to have been on disability in general; it was advertised as being especially useful to employers who might want to make their workplace more accessible for employees with disabilities.55

Participants also felt the training was not specifically tailored towards children, with the lead trainer having no previous experience with children with disabilities. This means that while DET was focused on moving people towards a rights-based understanding of disability in general, its role in raising awareness or increasing knowledge of children with disabilities has been limited. This seems to have been a missed opportunity for UNICEF.

This evaluation considers that UNICEF has been ambitious in its expectations around the effect of one-off training on attitudinal change, since this type of activity works best as part of a wider strategy with more targeted messaging and support on how to implement inclusion. It is a good approach, but it needs to be implemented as part of a wider programme where behaviour change expectations are mapped out and monitored. UNICEF Malaysia has not articulated clearly enough how these initial training activities will be used to promote longer-term attitudinal and behaviour change.

In their engagement with different stakeholders, the evaluation team noted the extent to which the language they used was often still charity/medical focused. It was reasonably common for participants to talk in terms of ‘disabled children’ and ‘normal children’ (rather than children with or without disabilities), or to consistently refer to children with disabilities as ‘special’ (rather than equal). Other phrases included use of ‘the handicapped’ and ‘the disabled’ or even ‘PWDs’ (to mean People with Disabilities). These stakeholders are the ones who have had the most direct contact with UNICEF’s messaging, suggesting that there is still considerable work to be done to promote a rights-based approach to disability.

In summary, the DET appears to have been an effective way to introduce people to a social-model approach. However, since the Initiative did not specify a target group for influencing knowledge and understanding of a rights-based approach, nor establish a baseline measure, it is not possible for this evaluation to comment on the degree to which the training has been more broadly effective.

55 DET only flyer for UNICEF 2017.pdf.


27

4.1.3. The effectiveness of partnershipsEvaluation question: Summary of findings:How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice?

• Creation of corporate inclusion champions was a well-conceived strategy with plenty of future potential

• Limited evidence (at this point in the partnerships) of the partnerships resulting in the adoption of inclusive strategies

Partnerships are regarded as being ‘at the heart of the UNICEF mandate’ and are defined as ‘voluntary and collaborative relationships between various parties, both public and non-public, in which all participants agree to work together to achieve a common purpose or undertake a specific task...’.56 UNICEF Malaysia refers to partnerships in different ways. One activity under the Initiative was to establish partnerships with two corporate ‘inclusion champions’, with the objective of ensuring children with disabilities are socially included.57 UNICEF signed a Memorandum of Understanding (MoU) with Petrosains in 2017 and another with Sime Darby Property in July 2018, which formalized their relationship as corporate inclusion champions. Both of the agreements were two years in duration and were non-financial.

While not formalized relationships, UNICEF Malaysia also refers to organizations such as Sabah Art Gallery, Sabah State Library, Jesselton Philharmonic Orchestra and the various disability-focused NGOs they have worked with as ‘partners’. Such organizations described their own relationship with UNICEF as ‘a partnership’, even though no formal relationship was established. In addition, Tandemic is a corporate organization contracted as an implementation partner under a Partnership Cooperation Agreement and has been linked with UNICEF since 2016. Its partnership was based on a set of key deliverables linked to the Makeathon event with a fee attached. This section only focuses on the #ThisAbility Flagship Initiative corporate inclusion champions.

The MoU with Petrosains included DET for Petrosains staff; support for the launch of the KAP Survey report; hosting of a Makeathon event; use of the innovations lab and pilot-testing of inclusive tours and communications materials.58 This partnership was considered to have undergone quite a long ‘courtship’ process, but both sides were clearly in agreement once the relationship was formalized with an MoU in 2017. Since becoming a corporate inclusion champion, Petrosains reported gaining moral and technical support (in the form of DET) from UNICEF and felt UNICEF had been a good ‘sounding board’.

The Sime Darby Property MoU was signed in July 2018 and included DET for Sime Darby Property staff; joint work on an inclusive playground; and a joint awareness campaign focused on the importance of inclusion, with the UNICEF-Sime Darby Property partnership utilized as an example of good business practice.59

The Petrosains partnership has resulted in increased staff awareness around disability with a reported ‘change of mindset’, but there is less confidence over the extent to which this has changed the way Petrosains engages with the public more generally. Internally, Petrosains has plans for some of the new science space to be used to help raise the profile of disability; it is

56 E/ICEF/2012/18 Report on the implementation of the strategic framework for partnerships and collaborative relationships, UNICEF, 2012,

57 Disability Flagship Working Paper (n.d.) p.258 Memorandum of Understanding between Petrosains SDN BHD and UNICEF, Sept 2017, p.2.59 Memorandum of Understanding between Sime Darby Property Berhad UNICEF, July 2018, p.2


28

exploring the development of a sign language app to provide signs in real time, which will be tested at Petrosains once the prototype becomes available. Overall, there is a sense that the MoU with UNICEF has helped raise the profile of disability, but that disability inclusion still needs to be institutionalized, including changes to organizational policies, which will inevitably take time and require a much broader buy-in from across the company.

The longer-term impact on the ability of Petrosains to be more disability-inclusive will inevitably take more time, although with the positive relationship that UNICEF Malaysia has established, things should continue to improve. Petrosains is currently working on making changes to improve its overall accessibility, but at this stage it is too early to assess the partnership’s effect on improving their ability to model inclusive good practice.

When UNICEF signed an MoU with Sime Darby Property, the latter was already developing a relatively inclusive new township. Sime Darby Property referred to this development as ‘multi-generational’; houses were being designed with step-free access, extra wide doors on the ground floor, and some extra-large bathrooms, making the houses wheelchair friendly. UNICEF was able to influence the design of the township further by suggesting Sime Darby designs and builds an inclusive playground space. Through the consultation process, UNICEF was able to guide the design so that it became genuinely inclusive (rather than just accessible) drawing on the ideas and interests of young people with disabilities themselves.

Currently, there is not enough information on the partnership with Sime Darby Property to assess whether the inclusive playground work will affect disability-inclusive practices more broadly. Given Sime Darby Property is already implementing elements of Universal Design into their housing developments, there is a good chance the inclusive playground design will be utilized in the future.

4.1.4. Impact of partnerships on empowerment and self-advocacyEvaluation question: Summary of findings:How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included?

• Some evidence to suggest interactions between children with disabilities and corporate inclusion champions have increased

• No evidence to show the partnerships have resulted in the empowerment of children and adolescents to undertake self-advocacy

This evaluation found no evidence that partnerships had been effective in empowering children and adolescents with disabilities to advocate for themselves and in ensuring they are socially included. Partly this is due to the nature of the partnerships: the MoUs were not designed specifically to encourage or enable corporate inclusion champions to work with children and young people with disabilities on advocacy issues. There is no clear linkage between empowerment and self-advocacy results and the partnership activities. Moreover, UNICEF Malaysia itself does not yet work in a way that promotes self-advocacy by children and young people with disabilities and, therefore, the activities that it currently supports are not yet providing the kind of platform that could act as a model for corporate inclusion champions.

Nevertheless, what these partnerships have achieved via the key activities (the Makeathon and the inclusive playground) is to enable consultations to happen between children and young people with disabilities and the corporate partners, which has helped the children to express their needs more directly. These events have provided a very small number of children and young people


29

with disabilities the opportunity to engage in activities where their views, experiences and ideas were given prominence, and some of the children were supported to express themselves in media articles and short films.

4.1.5. The effectiveness of public advocacy campaignsEvaluation question: Summary of findings:To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities?

• Public awareness-raising has been central to the work of the Initiative with good use made of existing public events

• The one-off nature of events and the lack of UNICEF’s own monitoring data means their effectiveness at increasing public awareness cannot be evaluated

Public awareness-raising campaigns and behaviour change activities (rather than advocacy) have been a core component of UNICEF’s #ThisAbility Flagship Initiative. All the activities carried out under this Initiative were focused on raising awareness, with the assumption that by promoting positive messages and imaging of children and young people with disabilities, the attitudes and behaviour of persons without disabilities would change. Specific events such as the BIM, the Sabah Carnival and the Makeathon were used as vehicles around which to build public campaigns that helped to raise the profile of children with disabilities and challenge assumptions by showcasing what children and young people with disabilities can do.

Considerable efforts were put into events such as the Sabah Carnival, which involved many different sections within UNICEF Malaysia working together to create a positive experience for children and young people with disabilities, as well as promoting the inclusion and visibility messaging of the #ThisAbility Flagship Initiative (see Box 3).

The Man United awareness campaign provides a key example of a well-designed messaging platform. This was commissioned by UNICEF and completed before the #ThisAbility Flagship Initiative, but was used quite extensively throughout 2017 and 2018 to support UNICEF’s messaging around inclusion. This two-minute film uses sport as the entry point, promoting a shared interest in football among young people with and without disabilities. Children and young people with disabilities were a key part of the design of the video, and were identified through members of the Malaysia Partnership for Children with Disabilities (established with the support of UNICEF pre-2017).

As many key informant groups highlighted, however, UNICEF Malaysia lacks systematic monitoring of awareness-raising events. Without pre- and post- event data it is not possible to monitor the impact of campaigns on the public’s awareness of children with disabilities. The KAP Survey Report actually provides UNICEF Malaysia with a baseline from which it could design a behaviour change strategy, and there are three key indicators that the Report suggests could be used to monitor the impact of interventions (see Box 5).60 However, while the Disability Flagship Working Paper references the KAP Survey Report, it does so only in terms of identifying a broad set of key messages, ‘...designed to fill the knowledge gap outlined in the KAP Survey Report findings and recommendations.’61 These key messages do not reference the indicators suggested by the KAP Survey Report, which would seem like a missed opportunity for UNICEF Malaysia.

60 Moore K., and Bedford, J., Childhood disability in Malaysia: a study of knowledge, attitudes and practices. UNICEF Malaysia, Kuala Lumpur, (n.d), p.82.

61 Disability Flagship Working Paper,( n.d.), p.3.


30

Box 5: Suggested KAP indicators for UNICEF Malaysia to monitor

Baseline indicatorsQ.26: ‘How well informed are you about children with disabilities?’ 15.9% of the population surveyed reported that they were ‘Not at all informed’

about children with disabilities. The least informed answer options (i.e. answer options one, two and three on the Likert scale) account for 58.4% of the population surveyed.

Q.34: ‘Is a child with disabilities equal to a child without disabilities?’ 49.7% of the population surveyed reported that a child with disabilities is not equal

to a child without disabilities.

Q.44: ‘ Are children with disabilities more likely to be abused?’ 59.7% of the population surveyed reported that children with disabilities are more

likely to be abused.

In general, focus group participants and key informants had varying levels of awareness of specific campaigns. However, there was not a great deal of support for the campaigns overall, as one participant commented:

“...because it is always joined by the same group of people.”

Most external key informants sensed that the campaigns had not had much impact on people who did not already have an interest in children with disabilities and that they had limited effects on mainstream audiences. Informants had more support for seeking opportunities to target and change the mindsets of young people, especially through school-based events or campaigns. They felt that this is the target group where the most impact could be had, since attitudes were still relatively flexible.

Responses from parent informants suggested that, while big events like the BIM and the Carnival did offer the opportunity to affect behaviour, their lives had not changed as a result. They felt their children were still quite isolated and that significant barriers remained; where there were specific events that focused on inclusion their children could participate, but beyond that little had changed. All key participant groups repeatedly mentioned the lack of follow through: events happened in isolation with little collaboration; decisions were made with no subsequent actions or activities to help reinforce messaging. UNICEF staff also noted the lack of follow through, with one commenting that events are run and there is a focus on disability, but afterwards:

“…we go back to our daily routine until another event gains our attention.”

Feedback regarding the Carnival event also suggested to the evaluation team that the messaging around disability was not as rights-based as it could have been. One informant who was involved (and who had undergone DET), talked about the impact of the event in terms of eliciting people’s sympathy:

“Well I think everybody was quite touched with it. Of course, the families, but also the public who were there because we noticed the response when they performed.”


31

One of the issues seemed to be that UNICEF had not developed a specific behaviour change model, and was therefore not sure what effect its awareness-raising messages might be having. The sense from staff informants within UNICEF Malaysia was that there had not been enough direct engagement with communities and people with disabilities, and it was therefore not possible to understand exactly what efforts might be most successful in effecting behavioural changes.

The Man United campaign was the only example where young people with disabilities had been part of the development and implementation of an awareness-raising activity. Unfortunately, this happened prior to the #ThisAbility Flagship Initiative and, while it was used extensively as a resource, the process was not repeated. It is also notable that UNICEF does not appear to have a relationship with the disability movement in Malaysia. The organizations UNICEF has worked through are not Disabled People’s Organizations and there have not been any active efforts to link the Initiative’s activities with disability-focused advocacy priorities such as implementation of the CRPD or updating of the Disability Act (2008).

4.1.6. Evidence of cross-sector work Evaluation question: Summary of findings:To what extent has the #ThisAbility Flagship Initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities?

• The Initiative successfully promoted collaborative working in different ways

• No evidence was found that this collaborative work has led to an increase in high quality services

On the basis of FGDs and KIIs, it was evident that many of the individual activities have been collaborative. For example, the Makeathon was run as an active collaboration between Tandemic, technical experts, UNICEF and Petrosains. This resulted in a well-planned event that was enjoyed by those who took part. Likewise, the Sabah Carnival was the result of multiple stakeholders (public, private and UNICEF) collaborating actively to provide the venue, funding, expertise, and other material inputs to make the event possible. The effectiveness of the collaborations directly contributed to the success of key activities, which played a role in helping to increase access and inclusion for children with disabilities and their families (albeit on a one-off basis).

Within the UNICEF office too, a key feature of the #ThisAbility Flagship Initiative taskforce, which is made up of representatives from several different departments, is that it has created a collaborative feeling among staff working on disability.

Beyond the UNICEF office and one-off events, however, this evaluation did not find any examples of where cross-sector work has led to increased access to high quality services for children with disabilities. The partnership with Sime Darby Property has the potential to lead to changes in the way playgrounds are designed in the future, however. But given the first inclusive playground has yet to be built (despite obviously very positive progress) it is too early to say whether or not this will prove to be effective.


32

4.1.7. The extent of gender inclusionEvaluation question: Summary of findings:Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship Initiative?

• There is no evidence to indicate that gender equality has been taken into consideration in the design, planning or implementation of activities

Despite providing all key informants with opportunities to talk about gender and other intersectionality issues, this evaluation could find no evidence that gender equality considerations have played any role in the planning or implementation of #ThisAbility Flagship Initiative activities. UNICEF Malaysia does not collect disability-specific monitoring data that is disaggregated by gender or any other metric. The Working Paper and Monitoring Framework do not reference gender markers or gender equity activities. Children with disabilities are treated as a homogenous group in all programmatic references with no considerations given to any intersectional identities (age, gender, ethnicity, migration status, socio-economic status, location, impairment etc.). Anecdotal evidence suggests efforts were made to try and balance gender and impairment representation in events, but this falls short of anything that could be described as transformative from a gender and social inclusion perspective.

4.2. RelevanceBox 6: Relevance questions

How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the Sustainable Development Goal (SDG) roadmap?

How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050?

To what extent is #ThisAbility Flagship Initiative aligned with the new UNICEF’s Strategic Plan?

Has the design of the #ThisAbility Flagship Initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable?

Have the voices of children and adolescents with disabilities been taken into account throughout?

Summary: UNICEF Malaysia has signalled a strong desire to work on disability inclusion within its CPD and country strategy documents. In turn, the Government has also made public its intentions to create more equal and inclusive services. Given that the Government of Malaysia has ratified the CRPD but has struggled with implementation, UNICEF’s intention to work on disability is highly relevant. UNICEF Malaysia has missed opportunities to better align its activities under the #ThisAbility Flagship Initiative with broader influencing around the rights of persons with


33

disabilities, because it has not made explicit use of international frameworks such as the CRPD in its campaigning. The lack of an overall advocacy strategy has limited the success of individual events to have a broader impact.

4.2.1. Alignment to Agenda 2030Evaluation question: Summary of findings:How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the SDG roadmap?

• Intrinsically, the work on disability is well aligned to Agenda 2030 and the SDGs

• No explicit link has been made by the Initiative to Agenda 2030 or any international development frameworks

Currently, the #ThisAbility Flagship Initiative makes no mention of Agenda 2030 and it is not immediately obvious how the Initiative is assisting the Malaysian Government to realize its full responsibilities around Agenda 2030 and the SDGs. Key informants were generally in agreement that the Initiative is indirectly contributing towards Agenda 2030 given its focus on inclusion and its visibility.

It is a key issue that while the messaging has indirectly supported the SDGs, the lack of overt rights-based campaign messaging on Agenda 2030 has meant UNICEF Malaysia has missed opportunities to link up its activities with wider development issues.

4.2.2. Alignment to Malaysia Vision 2020Evaluation question: Summary of findings:How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050?

• The work on disability is relevant to the Malaysian Government’s longer-term development plans but the Initiative does not align itself to these plans

Within its CPD, UNICEF commits to supporting Malaysia to make the transition to high-income status (a focus of Malaysia’s Vision 2020 plan) through three cross-sector focal areas: enhanced child and adolescent well-being; social inclusion and disparity reduction; and enhanced engagement and partnerships for child rights. These focus areas are reflected in more detail as ‘outcomes’ in the CPAP and the CPMP.

Within the CPD, disability features in ‘enhanced child and adolescent well-being’ in relation to early learning opportunities; and in ‘social inclusion and disparity reduction’. This indicates that at the country programme level, UNICEF Malaysia has the intention of using its relationship with the Government to help promote inclusive growth.

Clearly, disability is an issue that has been flagged within the Government’s development plans and at country programme level, and UNICEF Malaysia is responding to this within its own country agreement. Overall, the #ThisAbility Flagship Initiative can be said to be of relevance since it is an attempt by UNICEF Malaysia to operationalize its commitment to support inclusion.

The main issue is that it is difficult to make any strategic link between the Initiative, the CPD and the Government’s development plans. Neither the Initiative Working Paper nor the Monitoring Framework (2017–2020) mentions Malaysia’s Vision 2020 plan. Malaysia’s National Transformation


34

2050 is also absent from the Working Paper and Monitoring Framework, perhaps because the preparation phase for Malaysia’s National Transformation 2050 only began in 2017. It is also hard to link the Initiative directly to the CPD. There is not an obvious link between the focus of the Initiative’s activities and the Government’s commitments towards improving childcare facilities, education and employment opportunities, although awareness-raising activities will undoubtedly help reduce the high levels of stigma and discrimination that are in part leading to exclusion.

4.2.3. Alignment to UNICEF’s Strategic PlanEvaluation question: Summary of findings:To what extent is #ThisAbility Flagship Initiative aligned with the new UNICEF Strategic Plan?

• There is implicit alignment to UNICEF’s Strategic Plan however, there is no clear results chain guiding implementation of disability inclusion work that obviously links it to the CPMP (or the CPD)

The UNICEF Malaysia CPMP states that disability is a key focus area for UNICEF Malaysia, particularly in relation to education, although this has not been the subject of any significant activities implemented under the Initiative. UNICEF Malaysia has nevertheless restructured its Programme Section to ensure the Education Officer position reports directly to the Head of Programme/Deputy Representative with a view to improving cross-sectional collaboration. This was also designed in part so that a ‘…focus on children with disabilities will be strengthened through increased coordination of the work done by both education and child protection.’62 There is recognition here too that the Communications Section requires a strong element of communication for development (C4D) in order to address the deep-rooted stigma associated with disability, among other issues which should be brought out across the programme.

The CPD Results Framework includes two disability-related indicators: • Output 1.1 – percentage of population surveyed with positive perceptions and attitudes

towards children with disabilities, with a target increase of 30 per cent, based on baseline data gathered from a disability KAP Survey commissioned by UNICEF in 2016.

• Output 1.4 – existence of Ministry of Education improved strategy for inclusive education for children with disabilities, based on modelling, with measurement of this reliant on reporting by Ministry of Education.

What is notable about the disability indicators themselves is that they do not relate directly to the experiences of children with disabilities which forms the main focus of the #ThisAbility Flagship Initiative objectives. The indicators are also not reported against on an annual basis. For example, UNICEF does not intend to carry out another KAP Survey until 2025, which is some way beyond the current CPMP.63

The introduction of the #ThisAbility Flagship Initiative in 2017 can be seen as a determined effort to find a mechanism through which to operationalize disability work. Key UNICEF informants suggested that the benefits of having a Flagship Initiative included the ability to work more proac-tively on disability despite the lack of concrete direction within the CPMP. The Initiative therefore provides a helpful point of reference around which Sections can engage. It appears, however, that UNICEF Malaysia does not yet have the mechanisms in place to monitor progress or measure success at this level.

62 UNICEF Malaysia 2016–2020 Country Programme Management Plan, p6.63 Personal correspondence, August 2019.


35

4.2.4. Intersectionality and the voice of children with disabilitiesEvaluation questions: Summary of findings:Has the design of the #ThisAbility Flagship Initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable?

• No explicit attempts have been made to address intersectional issues in the design of the Initiative, with children and young people with disabilities being treated as a homogenous group

• Impairment needs were generally well accommodated during activities


• Some children and young people with disabilities have had opportunities to actively participate in the design of some events but there is no evidence of systematic engagement

This evaluation could find no evidence that the diversity of children’s experiences had been taken into consideration during the design of the #ThisAbility Flagship Initiative. KIIs suggested there was no explicit mechanism through which children and young people could feed into the design of the Initiative. In fact, there was a general sense among informants that UNICEF Malaysia had not done very well in meaningfully engaging children and young people with disabilities in any of its planning processes.

KIIs and especially FGDs revealed a mixed reaction in relation to whether or not people felt UNICEF had considered diversity issues. In the main, participants agreed that both girls and boys were freely able to join activities, but there was no sense that things had been designed in ways that could challenge norms.

Some participants noted that it would be good to have higher participation levels from girls with disabilities because they are generally more isolated. Most participants believed that neither ethnicity nor other social identities would be a barrier to participation, although a few respondents highlighted an absence of participation by some ethnic groups. Some participants felt that there could be financial barriers to participation:

“... if you don’t have a wheelchair, you can’t take your child out; if you don’t have a car, it is almost impossible to bring them...”

Overall, UNICEF Malaysia appeared to have tried to consider a wide range of impairment needs in the design of individual activities. Many of the non-disability organizations tasked with implementing activities during the Sabah Carnival revealed how much UNICEF had supported them to think about different accessibility issues. Several staff informants reported that they appreciated being prompted by the Disability Consultant to think about accessibility, with the inclusive events checklist proving especially helpful.

Several informants referred to the inclusive playground consultation as being especially inclusive. One informant commented that the proposed design had even considered how to make the playground a safe place for deaf children with the use of visual prompts to help communication between deaf and hearing children.

The main challenge in evaluating the extent to which intersectionality has been considered in the programme is the fact that the Initiative does not collect or collate data that can be disaggregated by any diversity indicator. For UNICEF Malaysia to be confident it is creating inclusive activities that take account of the diverse range of identities among children with disabilities, it is necessary to monitor participation.


36

4.3. EfficiencyBox 7: Efficiency questions

To what degree have UNICEF resources (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility Flagship Initiative?

How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?

Summary: The activities delivered under the #ThisAbility Flagship Initiative were generally observed by stakeholders to have been well resourced and well managed with a high degree of technical support provided. Internally, concerns were raised that the prioritization of resources, especially human resources, was insufficient to effect organization-wide change and that a more strategic approach to disability is required. The lack of systematic monitoring has made it difficult for UNICEF Malaysia to assess whether its resources have been used in the most efficient way (see Section 4.4 for more details).

4.3.1. Human, technical and financial resourcingEvaluation question: Summary of findings:To what degree have UNICEF resources (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility Flagship Initiative?

• Resourcing of individual activities appears to have been sufficient

• The stand-alone nature of activities means UNICEF Malaysia cannot measure their efficiency overall

This evaluation was not able to assess efficiency to any great extent because the evaluation team were not provided with a #ThisAbility Flagship Initiative programme budget; there is no accountability process against which to measure dispersal of resources. KIIs with staff and partners suggested that resources were sufficient for the effective implementation of planned activities. Indeed, several partners commented that UNICEF had been a good organization to work with in relation to resources. Informants expressed some concern that the Initiative had fallen short in terms of how funding had been prioritized. Informants suggested that the #ThisAbility Flagship Initiative had not been given enough of a priority status in resourcing terms rather than a lack of funds per se:

“I think there could have been more that could’ve been done to support us as a flag-ship and the person driving that forward.”

Informants raised specific concerns about whether the #ThisAbility Flagship Initiative was sufficiently staffed to achieve its ambitious objectives. Given this was a Flagship, some staff were surprised it was being led by a consultant (rather than a full-time member of staff). However, the consultant provided both excellent technical support and a much-needed driving force, for example simply being available to attend meetings, address forums or to follow up with different sections on their disability activities. Without that centralised support role, participants were sure less progress would have been made in actually implementing such a range of successful activities. Informants expressed caution that this does not necessarily equate to a working model that is efficient or sustainable


37

Several informants also suggested that contributing to the #ThisAbility Flagship Initiative was not perceived as mandatory or something that all parts of the organization must do. While teams contributed to the Initiative and engaged in related activities, several informants perceived the Initiative to be driven by one person and not as the collective responsibility of more UNICEF Malaysia team members.

“The weakness is that it is only by one person...”

This reinforced the point that the #ThisAbility Flagship Initiative Working Paper and Indicators Framework are insufficiently strategic to be able to influence and embed disability work into the Country Programme more broadly.

4.3.2. The adequacy of partners’ capacity

Evaluation question: Summary of findings:How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?

• UNICEF Malaysia has a reputation for delivering good technical support to partners

• An overreliance on organizations that work for and on behalf of people with disabilities rather than with organizations of people with disabilities is limiting opportunities for UNICEF Malaysia to advance a rights agenda

Mainstream partners generally commented that having UNICEF on board in matters relating to children with disabilities did give them more confidence and ‘mileage’ to engage with disability issues. Mainstream organizations were especially keen to highlight the value of UNICEF support in helping them understand how best to meet the needs of children with disabilities. For example, all partners commended the DET arranged by UNICEF because it had given them a better understanding of how to work with people with disabilities.

In terms of events, all the organizations UNICEF Malaysia engaged have done well to accommodate the specific requirements of children with disabilities and, as noted extensively in this evaluation, feedback from those who participated in activities was generally very positive. Beyond the activities, however, it is clear from KIIs and FGDs that most partners have not made progress in taking up issues on the rights of children with disabilities as a result of the Initiative. In fact, what has been striking across all key informant groups is the lack of reference to the rights of children with disabilities in the language used to describe the Initiative.

As noted in Section 4.1.4, it is also notable that UNICEF Malaysia has not so far engaged directly with the disability movement in Malaysia.


38

4.4. SustainabilityBox 8: Sustainability questions

To what degree has the #ThisAbility Flagship Initiative thus far contributed to the generation of sustainable capacities at the national and sub-national levels?

What are the enabling as well as constraining factors that are likely to influence future cross-sectoral work on focus issues?

Summary: Due to the fact the #ThisAbility Flagship Initiative is intended to be primarily an internal management device to increase disability work across the organization, it currently lacks the national ownership required for sustainability. While many events and campaigns have been run with a view to changing attitudes, knowledge and practices, as noted in previous sections, the lack of an advocacy strategy and any systematic monitoring data meant this evaluation could not determine their impact or sustainability.

4.4.1. Sustainability at national and sub-national levelEvaluation question: Summary of findings:To what degree has the #ThisAbility Flagship Initiative thus far contributed to the generation of sustainable capacities at the national and sub- national levels?

• The design of the #ThisAbility Flagship Initiative does not currently enable UNICEF Malaysia to have an impact at national and sub-national levels

The evaluation was unable to address the question of the #ThisAbility Flagship Initiative’s sustainability at national and sub-national levels because its activities were not designed to effect changes in capacity at national and sub-national level.

Most activities were one-off events with limited geographical reach (e.g. Klang Valley and Sabah), therefore it is challenging to gauge the benefit of the Initiative on a wider scale.

Without a baseline or any systematic monitoring of outcomes and impact, the evidence available to assess sustainability at this level was also insufficient. Efforts to engage Government stakeholders (referred to the evaluation team by UNICEF Malaysia) in the evaluation process proved extremely difficult, with most of the potential Government key informants declining to be interviewed. Of the officials who did take part, there was limited direct experience with UNICEF Malaysia on issues related to children with disabilities, again despite UNICEF Malaysia suggesting these officials to the external evaluators. The most important benefits that Government informants reported in relation to disability was the willingness to fund inclusive events and the quality of UNICEF’s press releases and communications materials.


39

4.4.2. Enabling and constraining factors for future work Evaluation question: Summary of findings:What are the enabling as well as constraining factors that are likely to influence future cross-sectoral work on focus issues?

• The design of the #ThisAbility Flagship Initiative does not allow for this question to be answered

The evaluation was also unable to address the question of the enabling and constraining factors likely to influence future cross-sectoral work because of the current design of the #ThisAbility Flagship Initiative activities. The one-off nature of events with limited geographical reach meant such factors could not be identified.

4.5. Utility of a Flagship InitiativeIn the absence of a formal definition of what constitutes a ‘flagship initiative’ within UNICEF Malaysia, this evaluation has taken the views expressed by staff and partners to assume that it means a programme or set of activities which are somewhat ‘off-plan’ and cross-cutting; effectively, activities created during an existing planned period to provide UNICEF Malaysia with a platform for developing innovative working relationships across sections, in a new area of work; in this case, disability.

The absence of an agreed definition of a ‘flagship initiative’ inevitably means it is difficult for people to evaluate how successful it has been and whether or not it is a mechanism they would seek to utilize again. This point was highlighted in a 2016 Evaluability Assessment in which it was noted that the programme had been running two flagships, on Child Marriage and Out of School Children, but that: ‘(I)t is not clear what a ‘flagship’ denotes’.64

In summary, evidence collected through this review suggests there are three key strengths to having a flagship initiative:

• Enables cross-sectional working to grow, creating new synergiesMany informants expressed that having the opportunity to work together on activities was a key strength, including within UNICEF; between UNICEF and partners; and in some cases, between partners. For example, civil society organizations in Sabah commented that the opportunity to work on events like the BIM and the Sabah Carnival was instrumental in helping them to engage with each other in ways they had not done in the past. Within UNICEF, the working relationship between C4D, Innovations, CSR and the Disability Consultant seemed particularly productive (see Box 9).

64 Vine Management Consultanting (2016) Evaluability Assessment: UNICEF Malaysia Country Programme 2016-2020. P.11


40

Box 9: Case study illustration on positive inter-office collaboration

‘We are all special’ CarnivalThis event involved a wide range of UNICEF Sections. C4D worked closely with the Disability Consultant around partnership building and advocacy; Education provided some advice and messaging around inclusive education; Private Fundraising and Partnerships mobilized street fundraisers to use face-to-face encounters to raise awareness and get people to sign up to the Innovations Section U-Report mechanism to take an online attitudes survey; Communications created a press release, event speeches, engaged the media and filmed numerous short videos.

The digital assets created at this event were used the following month to underpin a mini-awareness campaign – ‘What’s so special?’ – coinciding with International Day of Persons with Disabilities 2018. Opportunities were taken to showcase videos, disseminate information, education and communication materials and display billboards in Kuala Lumpur and Selengor.

• Helps raise the visibility of disabilityWithin UNICEF, informants were clear that by running events under the #ThisAbility Flagship Initiative, awareness of disability had increased. During 2017 and 2018, around 26 UNICEF staff and 25 FACERS undertook a one-day DET session and it’s likely that more were involved in sessions provided to partners. The session itself helped to raise awareness but, with the ongoing series of activities, staff and partners were asked to focus on children with disabilities more explicitly than had happened in the past. Several staff informants described how their own knowledge and experience of engaging directly with children with disabilities through these events had changed the way they approach disability. They felt more confident around children with disabilities and no longer saw the need to treat them differently or ‘in a special way’. Among the non-disability focused partner organizations, this was one of the benefits most often cited of working with UNICEF on Initiative activities.

• Good quality technical guidanceMany staff and partner informants commented on the quality of the support provided through the Initiative. Of particular note was the Disability Consultant’s skill in bringing people together, motivating and prompting where necessary. Staff informants noted the Consultants’ rich experience in disability, which was readily shared in ways that encouraged participation. From the perspective of partners, many informants described UNICEF as providing good technical support during the planning and implementation of activities, which particularly helped non-disability organizations to become more familiar with requirements around accessibility and accommodation for children and young people with a variety of different impairments. The work around the Inclusive Playground Initiative is a good illustration of how appropriate and timely technical support resulted in what promises to be a good quality inclusive design (see Box 10).


41

Box 10: Case study illustration on good use of technical expertise

Inclusive Playground InitiativeDuring the latter part of 2018, UNICEF engaged Sime Darby Property as an Inclusive Champion. One of the key areas of collaboration was the design of what could be Malaysia’s first Inclusive Playground for children. As a property development company, Sime Darby was a good stakeholder to influence and was especially open to the idea of setting aside an area of land in a new development. Given the innovative nature of the activity and the fact that UNICEF was keen that end-users had the opportunity to input into the design, an architectural company (Jeavons Landscape Architects), with experience in inclusive playgrounds, was brought in to lead the design process. A key part of the design process was a half-day workshop hosted by Sime Darby in which 45 end-user participants took part (including children and young people with disabilities, parents and grandparents). At this workshop, UNICEF used interactive play-based sessions to help children express their ideas of what they would want from a playground. This helped the architects to appreciate not just accessibility, but also what children with disabilities were interested in doing in an outdoor space. While the playground has yet to be constructed, the plans are now undergoing the approval process with the local municipal council.

In terms of weaknesses of the Flagship approach, three key areas of concern emerged, which are summarized below:

• Driven by one personMost staff informants and some partners noted that the work seemed to be driven by one person – an external consultant – and not a permanent member of staff, which many perceived as indicating that the initiative might not have had full support across the organization.65

While all informants were keen to acknowledge the passion and expertise provided by the Consultant, they were nevertheless concerned that the Initiative was not a sustainable model and had not helped them to mainstream disability into their work. Several informants highlighted this and noted that, since disability was not reflected in key performance indicators, it sometimes made it hard for people to provide the time required. Moreover, it had the effect that this was seen as ‘additional work’, leaving the consultant having to ask for support.

Others also sounded a note of caution, saying that simply working together on events and attending joint meetings did not necessarily mean that cross-section engagement was happening. Often the motivation for coming together was provided by the Disability Consultant rather than something that was driven by a common objective.

• No clear strategy with a lack of follow up or follow throughWhile the #ThisAbility Flagship Initiative did have a Working Paper and Monitoring Framework outlining a strategy, UNICEF staff reported experiencing the Flagship as a series of unconnected activities or events. In fact, many informants from all stakeholder groups noted that there seemed to be an over reliance on large-scale one-off events that were not seen to be part of a wider narrative.

65 It must be noted that the consultancy position was made into a staff position during the period of the evaluation. As a result, this may have weakened the perception that the initiative might not have total support across the organization because it was ‘driven’ by an external consultant.


42

Most staff informants and many partners perceived the main weakness of the Initiative to be a lack of strategic direction, with a lack of a narrative or journey. Several informants feared that disability could be an issue that, although it had now been raised, would soon be superseded by something else as priorities shifted. Without an overarching narrative to which all staff are making a contribution, some informants were concerned that disability could be ‘dropped’. To address this, many expressed considerable support for a disability-focused ToC or long-term strategy, with targets for each section that are regularly monitored; in this way, disability could be more effectively mainstreamed across the organization.

• No outcome or impact reportingStaff informants expressed some level of frustration about the lack of formal analysis of events or activities. Several noted that while they were in receipt of anecdotal evidence for what people thought of events, there was often no actual reporting. As one staff informant noted:

“...I spoke to parents at the events as well they are also very happy (sic) but I can’t really identify what are you really happy about what does your child think...”

There was some concern that as an organization, UNICEF was not properly tracking what was going well or what was not working because there were no regular evaluations happening. In the words of one staff informant:

“...sometimes it can be a bit all over the place...”

Staff informants suggested that this distanced staff from the work; taking away some of the agency and ownership because without feedback on the impact of the work it became difficult for them to build upon it in future plans.

To a large extent, this view was echoed during FGDs and partner interviews, when informants who had participated in events noted that they had not been asked to provide feedback on the events or activities. The lack of formal monitoring has not only deprived UNICEF of important data on whether its strategies are having an impact (intended or unintended), but it also has the effect of closing off information-sharing opportunities and accountability to partners and other stakeholders. Focus group participants particularly felt they were unsure what had happened since events like the Makeathon and the Inclusive Playground, despite having invested time in supporting them.


43

5. CONCLUSIONSBased on the analysis outlined in each section of this report, the evaluation team makes the following conclusions:

In terms of effectiveness:Conclusion 1On the whole, parents and children who participated in the various events implemented under the #ThisAbility Flagship Initiative found them to be fun, exciting and of personal benefit. The lack of direct engagement with parents, children and young people with disabilities in the design and evaluation of Initiative activities has been a missed opportunity for UNICEF Malaysia. The lack of follow up on Initiative activities with participants has also presented missed opportunities for UNICEF Malaysia.

Conclusion 2The DET was of good quality and has played a role in challenging participants’ preconceptions around disability: in many cases helping them to be more confident in engaging with people with disabilities. However, the DET lacked a focus on the rights of children with disabilities and the specific barriers they face.

Conclusion 3Establishing the two formal partnerships with corporate ‘inclusion champions’ was a well-conceived strategy that has had a degree of impact on promoting inclusive practice. However, there is no evidence to suggest these partnerships have empowered children and adolescents with disabilities to undertake self-advocacy; nor is there evidence of the potential of the partnerships to do so based on the objectives and design of the partnerships.

Conclusion 4Public awareness-raising has been central to the work of the #ThisAbility Flagship Initiative. UNICEF Malaysia did well to utilize existing public events and to influence them to become more inclusive. The effectiveness of the public awareness-raising could not be analysed because of the lack of monitoring data and the nature of these one-off activities. The medical/charity model approach to disability appears to still be widely used.

Conclusion 5The #ThisAbility Flagship Initiative successfully promoted collaborative working. Through participation in events, civil society organizations, corporate agencies, local government and media agents gained experience in working together to promote disability inclusion. Within UNICEF Malaysia too, individuals within different Sections worked well together to implement Initiative activities. There is no evidence yet of this collaboration leading to better quality services for children and young people with disabilities, although the inclusive playground initiative with Sime Darby Property could lead to significant improvements in the way outdoor spaces like this are designed in future.

Conclusion 6The evaluation could find no evidence that gender equality, or any other intersectional issue had been taken into consideration in the design, planning or implementation of #ThisAbility Flagship Initiative activities. The Initiative appeared to treat children and young people with disabilities as a homogenous group. A transformative approach to gender and social inclusion had not been taken,


44

although attempts have been made to balance gender and impairment representation in activities. This is a serious gap that UNICEF Malaysia will need to address in its future work.

In terms of relevance:Conclusion 7Intrinsically, the #ThisAbility Flagship Initiative is well aligned to Agenda 2030 and the fulfilment of the SDGs. However, the Initiative strategy makes no explicit link to Agenda 2030 and there is a lack of reference to international frameworks in UNICEF Malaysia’s messaging.

Conclusion 8UNICEF Malaysia is not actively aligning the #ThisAbility Flagship Initiative to the Malaysian Government’s longer-term development plans, despite the link having been identified in the CPD. Until a clearer strategic plan for disability is in place and embedded within the reporting structures of the CPD, the Flagship nature of the initiative will mean it remains an internal mechanism for promoting collaborative work, rather than a strategy for promoting disability- inclusive development.

Conclusion 9While the ideas about the need to include children and young people with disabilities are positive, there is not a clear results chain guiding implementation of disability inclusion work that obviously links the #ThisAbility Flagship Initiative to the CPMP (or the CPD). The lack of a clear framework for disability-inclusive work evidences a gap at this strategic level that has not so far been alleviated by the Initiative.

In terms of efficiency:Conclusion 10UNICEF Malaysia seems to have provided enough resources to implement activities to a good standard. The standalone nature of #ThisAbility Flagship Initiative events made it difficult to assess efficiency overall, as does the lack of systematic monitoring.

Conclusion 11UNICEF Malaysia has a reputation for delivering good technical support to its partners. However, an overreliance on engaging with organizations that work for and on behalf of people with disabilities, rather than with organizations of people with disabilities, is limiting opportunities for UNICEF Malaysia to advance a rights agenda.

In terms of sustainability:Conclusion 12The evaluation was unable to determine the extent to which the Initiative’s campaigns are having an impact on generating sustainable changes at national and sub-national levels.

In terms of the utility of a Flagship Initiative:Conclusion 13This Flagship Initiative enabled cross-sectional collaboration, raised the visibility of disability inclusion, and strengthened UNICEF staff’s access to good quality technical guidance. To further strengthen flagship initiatives, there is a need for UNICEF Malaysia to be explicit in providing a clear definition of what constitutes a ‘flagship initiative’ before developing such mechanisms in the future.


45

6. LESSONS LEARNEDLesson learned 1 – Deep-rooted attitudes towards disability require strong, coordinated rights-based messaging to effect change. Evidence gathered from the desk review, KIIs and FGDs indicates that UNICEF staff and key external partners focused primarily on promoting inclusion and increasing visibility of children with disabilities, to the detriment of promoting the rights of children with disabilities. By not explicitly promoting the rights of children with disabilities or linking their exclusion and lack of visibility to the barriers that exist within communities and systems, it is possible for disability to continue to be considered an individual’s problem, undermining efforts to achieve systemic and sustainable change.

Developing messaging that is explicit about the rights of children with disabilities, and framed in terms of reducing some of the barriers within communities (such as communities becoming more welcoming, so that the children and parents feel more respected) would be more compliant with the CRPD. Disability inclusion would be better framed as the result of removing barriers and upholding rights. This approach would also support mainstreaming because whatever child rights issues were being raised, messaging around children with disabilities could always be incorporated.

Lesson learned 2 – A Theory of Change and strong monitoring systems are required to help mainstream disability at an organizational-level. An objective of the evaluation was to assess the extent to which the #ThisAbility Flagship has been appropriately designed. Observations throughout the evaluation indicated that the system for monitoring the Initiative was creating uncertainty and a lack of cohesion. The lack of a ToC has inhibited monitoring at all levels. Each UNICEF Section appears guided by its own set of actions, for which it has output indicators. If these do not include disability-related outputs, then they are unlikely to measure the impact of their activities on children with disabilities, and there is no disaggregation of data by disability at impact level for any Section-based intervention. Furthermore, while a Flagship Taskforce was created, key informants explained that its main purpose was to update everyone on progress towards upcoming events, rather than analysing the outcomes and progress against the strategy. These observations underscore the importance of some key components that, in the external evaluators’ experience, are essential to disability-inclusion programming:

a. Explicit organizational commitment – the organization as a whole makes it clear why it is committed (its overall values); and what it seeks to achieve through this commitment (its overall purpose).

b. Inclusive programmes and services – the organization will ensure that all its direct services and programmes are delivered in inclusive, equitable and non-discriminatory ways and do not create barriers or reinforce negative stereotypes. This means ensuring all programmes are accessible to people with disabilities, regardless of whether or not people with disabilities are an intended target.

c. Disaggregated data – all programme data is disaggregated by disability using an international standard definition of disability (for example the Washington Group Question sets - www.washingtongroup-disability.com).

Lesson learned 3 – A short-term mindset can present safeguarding concerns and possible challenges for future engagement. Several of the assistive tools that were created during the Makeathon event were broken by the time of this evaluation, leaving children who participated in the event without their new gadgets. While this was a predictable outcome, the children and their families may not have been fully cognisant of this when they agreed to participate. For the We are


46

all special Carnival in Sabah, the Jesselton Philharmonic Orchestra supported a young deaf girl to play the violin and perform on the day of the Carnival. The Orchestra’s engagement with the girl reportedly stopped at the end of the Carnival, requiring the girl to hand the instrument back with no follow-up actions. The disappointment and frustration that participants reported regarding these two examples raises serious questions around the coordination of one-off events where children have participated in something potentially life-enhancing, but are left without the products they helped to create or continued opportunities to develop their new skills. UNICEF Malaysia may have inadvertently crossed the ‘do no harm’ threshold with these activities because they were relatively high profile but did not adequately consider or prepare for the experiences of the child participants whose expectations were raised and then not fulfilled. Safeguarding issues also arise from such activities; it is necessary to adequately consider the well-being of vulnerable young children when exposing them to such experiences without an obvious plan for follow up.


47

7. RECOMMENDATIONSThe following recommendations are based upon the key findings, the conclusions reached by the evaluation team, and the lessons learned; they have been developed to reflect the views of key informants and with the input of key stakeholders that formed a Reference Group for this evaluation.

Recommendation 1 To whom: Country Representative, Deputy Representative, Planning Monitoring and

Evaluation ChiefPriority: HighTimeline: 6-months

UNICEF Malaysia needs to make a strategic commitment to disability inclusion by fully embedding disability into key planning documents with the expectation that children with disabilities will be included and monitored across all areas of programming. We encourage UNICEF Malaysia to start doing this in the short-term, beginning with the forthcoming strategic planning.

Considerable support has been generated within UNICEF Malaysia for working on disability. The high-quality technical support provided during the initial phase of the #ThisAbility Flagship Initiative and the opportunities for staff within sections such as C4D, communications and innovations to work together has increased capacity and generated positive momentum within the office. For sustained progress to be made from the Initiative, this work needs to be mainstreamed across all Sections. The ToC constructed as part of this evaluation (see Annex F) provides a single narrative that can frame UNICEF Malaysia’s future disability inclusion work. Embedding the ToC (or one similar) in strategic plans and establishing a monitoring system that can track progress toward the strategic goal (including the capture of disability- disaggregated data) at all monitoring points will significantly strengthen UNICEF Malaysia’s work in this area.

Recommendation 2 To whom: Country Representative, Deputy Representative, Communications Chief, Private

Funding and Partnerships Chief, C4D SpecialistPriority: MediumTimeline: 12-months

UNICEF Malaysia, starting immediately, needs to be much more explicitly rights focused in its messaging around disability and ensure its campaigns fully align with key international frameworks, including the CRPD.

Any future public engagement needs to promote the rights of children with disabilities, with a greater emphasis placed on removing barriers to achieve equality and inclusion. This evaluation recommends that UNICEF Malaysia develops an advocacy strategy that incorporates behaviour change targets that can be used to track the effectiveness of its messaging. More explicit reference to and use of the CRPD is highly recommended.


48

Recommendation 3 To whom: Country Representative, Deputy Representative, Communications Chief,

Programme StaffPriority: MediumTimeline: 12-months

UNICEF Malaysia, starting immediately, needs to provide more opportunities to meaningfully engage children and young people (with and without disabilities) during the design of campaigns and messaging.

While the #ThisAbility Flagship Initiative has provided some limited opportunities for children and young people with disabilities to voice their needs, it has not attempted to systematically engage them in discussions around how to shape advocacy priorities and activities. It is also evident that UNICEF Malaysia does not yet have a particularly strong relationship with the disability movement, with a tendency to rely on working with and through organizations that work on behalf of people with disabilities. This means that it is missing opportunities to support capacity building of the disability movement and to support a new generation of disability advocates. There are some precedents that UNICEF Malaysia might usefully take advantage of including a previously highly successful youth advocacy programme called ‘Young Voices’ which ran in Sabah with support from Cheshire Services.66 While this programme is no longer running, there is continued interest in using its experiences to develop more opportunities for young people with disabilities to come together to work on rights-promotion activities.

Recommendation 4 To whom: Country Representative, Deputy Representative, Communications Chief,

Programme Chiefs and Programme StaffPriority: HighTimeline: 6-months

UNICEF Malaysia urgently needs to develop robust and rigorous safeguarding review processes for all activities involving children and young people with disabilities.

As a matter of priority, UNICEF Malaysia should initiate measures to ensure that all activities involving children and young people with disabilities undergo a safeguarding assessment to ensure that the well-being of children will not be put at risk during or after events. Consulting with parents and older children with disabilities would help UNICEF Malaysia gain a better understanding of what mitigating actions could be put in place to ensure activities are fun and safe as well as protecting long-term well-being.

66 Sabah Cheshire Home, <http://www.sabahcheshirehome.org/prog_youngvoices.html>.


49

Recommendation 5 To whom: Country Representative, Deputy Representative, Gender Focal Point,

Communications Chief, Programme Chiefs and Programme StaffPriority: MediumTimeline: 12-months

UNICEF Malaysia needs, starting as soon as possible, to routinely assess new and existing programmes and activities to ensure gender and social inclusion norms and unequal power relations are considered in full.

A key finding from this evaluation was the absence of any consideration of gender and social inclusion norms and unequal power relations in the design and delivery of activities. Whilst there has been a conscious effort towards ensuring environmental accessibility during events for participants with disabilities, which has resulted in improved awareness of the need to provide reasonable accommodation, the #ThisAbility Flagship Initiative has not yet reached a level which could be assessed as being either gender and social inclusion (GESI) accommodating or transformative. There is considerable scope now for moving beyond the assumption that children and young people with disabilities can be treated as a homogenous group with just their individual impairment’s needs accommodated. As UNICEF Malaysia develops its disability-inclusive ToC it would be useful for it to undergo a Gender, Equality and Social Inclusion (GESI) assessment process to determine the extent to which all new programmes and activities can help move them towards becoming GESI transformative.


50

ANNEX A: EVALUATION TERMS OF REFERENCE

1. IntroductionThis Terms of Reference (ToR) document outlines the purpose and scope of a formative, forward-looking and learning-oriented Evaluation of UNICEF’s #ThisAbility Flagship Initiative in Malaysia. Further to the UNICEF’s State of the World’s Children on Children with Disability in 2013 that pointed to the need to reduce the vulnerability of children with disabilities, UNICEF Malaysia launched a number of initiatives to bring disability into the political and social discourse. Through partnerships with organizations and individuals UNICEF Malaysia created momentum to engage in dialogue on the rights of children with disabilities, and in 2016 #ThisAbility became a flagship priority within the new UNICEF Country Programme 2016-2020. During the flagship period UNICEF committed to make positive changes for children with disabilities in Malaysia so that they are free from discrimination and able to access services to support their growth, development and protection, in line with the United Nations (UN) Agenda 2030 principle of “leaving no one behind”.

With this backdrop, this evaluation seeks to examine #ThisAbility Flagship Initiative from 2016 until its current implementation, but it will also look at the evolution of UNICEF’s work on disability in Malaysia since 2013. The ToR present thus methodological options and operational modalities for an institutional contract of a team of two evaluation consultants (one international and one national). This independent evaluation is commissioned by UNICEF Malaysia; hence, UNICEF is looking for institutions with deep commitment to, and strong background in, the evaluation of social programmes, and particularly disability. Findings and recommendations from this evaluation will inform the direction #ThisAbility should take in the course of the remaining part of the Country Programme, including future evaluation of the flagship. Lessons from the evaluation may inform further cross-sectoral work and integration of disability across all UNICEF programmes, policy-advocacy for children with disability and the application of innovations. The evaluation is expected to being October 2018 and to be completed by January 2019.

2. Background and Rationale2.1. UNICEF Malaysia and its Focus on DisabilityUNICEF’s work in middle-income countries such as Malaysia has transitioned in recent decades to focus on upstream policy and advocacy interventions, capacity development and evidence generation. Malaysia ratified the Convention on the Rights of the Child (CRC) in 1995 and the Convention on the Rights of Persons with Disabilities (CRPD) in 2010, albeit with reservations, and implemented a Persons with Disabilities Act in 2008. Whilst progress has been made in improving the lives of children with disabilities (right holders), critical issues remain including equitable access to services and participation and overcoming widespread negative perceptions towards persons with disabilities. Such discriminatory views have a profound effect at all levels, from the development and life-course of children with disabilities, through to the shaping of laws and policies at the national level. In their concluding observations to Malaysia’s first periodic report, the Committee on the Rights of the Child made three key recommendations: that the country combats stigma and discrimination of children with disabilities; collects adequate statistical data on children with disabilities; and ensures that such data is used in the development of policies and programmes for children with disabilities.


51

A key component of UNICEF Malaysia’s 2016-2020 Country Programme67 is to support the country to realise the rights of all children, reduce disparities and foster social inclusion. This is in line with the UN General Assembly High Level Meeting on Disability and Development (2013) which urged increased understanding, knowledge and greater social awareness about persons with disabilities to eliminate discriminatory social and attitudinal barriers so that they participate fully in society, and the UN Agenda 2030 that promotes a world where “no one is left behind”.

UNICEF has an important role to play in supporting national and sub-national authorities and other partners to create an enabling environment for children with disabilities in Malaysia, and an environment in which they are included into society and have the right and ability to participate without fear of discrimination or marginalisation. UNICEF’s Malaysia recent knowledge, attitudes and practices (KAP) study68 noted, for instance, that respondents have limited knowledge about the actual causes of disability, attitudes are poor and practices are sporadic for children with disabilities. In addition, children with disabilities, their parents and families face stigma and discrimination at home, in schools and many other public environments. Having said that, children and adolescents are more tolerant and accepting than the general adult population, acting thus as effective agents of change by taking a stand against discrimination, promoting diversity and equal opportunities, and helping to ensure the social inclusion and participation of children and adolescents with disability.

In 2016 disability hence became a flagship priority around which the whole new Country Programme can gather. UNICEF’s work on disability is spread across four programme outputs as follows:

a) Output 1.1: Duty bearers have increased knowledge and demonstrate positive attitude towards protecting all children from harmful practices and discrimination (incl. disability);

b) Output 1.4: Improved equal learning and development opportunities for vulnerable children;

c) Output 2.4: Government ministries and research institutions have strengthened institutional capacities for undertaking research-based social and behavioural change communication (on issue related to disability, among others); and

d) Output 3.1: The Malaysian private sector has increased knowledge and commitment to its responsibility to respect children’s rights, and can access quality technical support and capacity for implementing the CRBP (incl. children with disabilities).

In addition, UNICEF developed a #ThisAbility Flagship Working Paper and Monitoring Framework that outlines the objectives, activities and measures that UNICEF committed to implement until 2020 to make positive changes for children with disabilities in Malaysia so that they are free from discrimination and able to access services to support their growth, development and protection. These objectives include:

• Children with disabilities have increased access to high quality services;• Children with disabilities do not experience abuse and discrimination;• Children and adolescents with disabilities are empowered to advocate for themselves;• General public are aware of children with disabilities; • Children with disabilities are socially included; and• Children with disabilities can contribute positively in society.

67 https://www.unicef.org/about/execboard/files/2015-PL24-Malaysia_CPD-ODS-EN.pdf 68 https://www.unicef.org/malaysia/Final_File_Childhood_Disability_in_Malaysia_Book_(Interactive_PDF).

compressed_3_Oct_2017.pdf


52

#ThisAbility Flagship Change Commitments and Monitoring Framework are presented in Annex I and II.

2.2. RationaleThis evaluation is expected to be formative, forward-looking and learning-oriented in nature and to produce credible, reliable and useful evidence on the results achieved to date by UNICEF’s #ThisAbility Flagship in Malaysia – including through the cross-sectoral coordination. The evaluation will also help identify strengths and weaknesses to inform the future direction of UNICEF’s work on children with disabilities in Malaysia, and it will be followed by an outcome evaluation by 2020. It will be shared with UNICEF’s Regional Office for East Asia and the Pacific (EAPRO) and other offices to promote learning on the utility of creating a flagship on disability.

3. Purpose, Objectives and Scope of Work3.1. PurposeThe main purpose of this independent, formative evaluation is to foster learning and improvement within the #ThisAbility Flagship initiative. The evaluation will document the evolution of UNICEF’s work on children with disability in Malaysia and assess both programme design and management, and assess its programmatic relevance, appropriateness, effectiveness, efficiency and sustainability against regional and international benchmarks. The evaluation will also assess the utility of creating a flagship on specific focus issues to mobilize cross-sectoral work among programme components in Malaysia.

The evaluation will be used to inform the future direction UNICEF’s work on children with disability should take in the remaining part of the Country Programme. It will help identify good practices, lessons learned, as well as inform cross-sectoral work and integration of disability across all UNICEF programmes, policy-advocacy for children with disability and the application of innovations. The evaluation will also make recommendations as to the type of evaluation that should be conducted at the end of the Country Programme.

The primary users of the evaluation include UNICEF Malaysia, the government and implementing partners (duty bearers). Secondary users include other agencies that work on children with disabilities in Malaysia, EAPRO, UNICEF Headquarters (i.e., Disability and Child Protection Divisions) (secondary duty bearers) and children with disabilities themselves (right holders).

3.2. Objectives and Scope of WorkThe objectives of the evaluation include the following:

• Analyse the strategies used to map out the trajectory of the #ThisAbility Flagship and re-construct the theory of change behind to make positive changes for children with disabilities in Malaysia;

• Assess the extent to which the #ThisAbility Flagship has been appropriately designed, efficiently and effectively managed, and integrated across the Country Programme, including the incorporation of equity, gender equality and human rights considerations;

• Examine the evolution of UNICEF’s work on children with disability in Malaysia, and in particular the intermediate results achieved by UNICEF’s #ThisAbility Flagship, enabling and disabling factors, considering aspects of both reduction of discrimination as well as access to services – including advocacy and awareness raising both at the national and sub-national level; and its likely sustainability; and


53

• Analyse the utility of creating a flagship on specific focus issues to promote cross-sectoral work, including strengths and weaknesses.

The evaluation is expected to provide reasonable conclusions based on the findings and substantiated by evidence, and clear, specific and actionable recommendations for informing the future direction of UNICEF’s work on children with disability in Malaysia. It should also identify good practices, lessons learned and innovations throughout the evaluation process.

The evaluation will not be an impact evaluation, but it will cover the evolution of UNICEF’s work on disability in Malaysia since 2013, with a primary focus on the #ThisAbility Flagship initiative since 2016 to date. Data collection will focus on states where UNICEF is operating through partners, notably Kuala Lumpur in Peninsular Malaysia and Sabah and Sarawak in East Malaysia. To the extent possible, the evaluation should be participatory and include the views of children and adolescents with disabilities that participated in the Flagship as well as those of their caregivers.

Evaluation evidence will be judged using modified Organisation for Economic Co-operation and Development (OECD) / Development Assistance Committee (DAC) criteria of relevance, efficiency, effectiveness and sustainability, as well as equity, gender equality and human rights considerations. Key evaluation questions (and sub-questions) include the following: Effectiveness of the UNICEF’s strategy to children with disability and its implementation considering: To what degree has UNICEF’s work on disability, and in particular through its #ThisAbility Flagship initiative, contributed to the creation of positive conditions to reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia? Have there been any unintended results?

Sub-questions:• How satisfied are the children and their caregivers with the quality of the services they have

received through UNICEF’s partners?

• To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights based perspective?

• How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice?

• How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included?

• To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities?

• To what extent has the #ThisAbility Flagship initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities?

• Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship?


54

Relevance of the programme strategy and approach, considering: How relevant is UNICEF’s work on disability and its #ThisAbility Flagship initiative to the developmental challenges in Malaysia, and particularly attitudes and practices with regard to disability? Sub-questions:

• How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the SDG road-map?

• How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050?

• To what extent is #ThisAbility Flagship initiative aligned with the new UNICEF’s Strategic Plan?

• Has the design of the #ThisAbility Flagship initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable?

• Have the voices of children and adolescents with disabilities taken into account throughout?

Efficiency of the programme implementation and management, considering: To what extent and how has UNICEF mobilized and used its resources (human, technical and financial) and improved its cross-sectoral coordination to achieve its planned results for children with disabilities?

Sub-questions:• To what degree have UNICEF resources (human, technical and financial) been sufficient in

implementing the activities under the #ThisAbility Flagship initiative?

• How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?

Sustainability in the programme, considering: To what extent are the benefits and achievements of the #ThisAbility Flagship initiative likely to continue after 2020 through national ownership and changes in terms of attitudes, knowledge and practices?

Sub-questions:• To what degree has the #ThisAbility Flagship initiative thus far contributed to the generation

of sustainable capacities at the national and sub-national levels?

• What are the enabling as well as containing factors that are likely to influence future cross-sectoral work on focus issues?

One of the key tasks to be initiated at the proposal stage will be to interrogate these questions and criteria and determine if all key issues have been given due prominence. Bidders are required to propose appropriate evaluation criteria (e.g., OECD/DAC criteria for evaluating development programmes, including sub-criteria such as equity, gender equality, human rights). Improvements


55

and/or refinements to the draft questions may be offered at the proposal stage; however, the expectation is that the inception process will yield the final set of questions.69

4. Evaluation Approach and MethodologyBased on the objectives of the evaluation, this section indicates a possible approach, methods, and processes for the evaluation.70 Methodological rigor will be given significant consideration in the assessment of proposals. Hence bidders are invited to interrogate the approach and methodology proffered in the ToR and improve on it, or propose an approach they deem more appropriate. In their proposal, bidder should clear refer to triangulation, sampling plan and methodological limitations and mitigation measures. Bidders are encouraged to also demonstrate methodological expertise in evaluating initiatives related to social inclusion and in particular the rights of children with disabilities.

It is expected that the evaluation will employ both a theory-based and a mixed methods approach drawing on key background documents and the monitoring framework for guidance. The timing of the evaluation is such that it will take mainly a formative approach, identify and assessing the effectiveness of UNICEF’s work on children with disabilities and to the extent possible children’s satisfaction to date to inform the future of the #ThisAbility Flagship initiative. To this end the evaluation will provide continuous and rapid feedback to primary users in the course of the evaluation process. The evaluation should consider throughout issues of equity, gender equality and human rights, in line with UNICEF’s Evaluation Policy (2013) and the UN Evaluation Group (UNEG) Norms and Standards (2016). Given the nature of the programme, equity, gender equality and human rights-based approach to development will be central in the conduct of the evaluation.

At minimum, the evaluation will draw on the following methods: • Desk review of background documents and other relevant data, including review and

analysis of secondary quantitative data;• Key Informant Interviews (KIIs) with disability actors in Malaysia (existing and potential),

non-governmental organizations participating in the NGO Impact Academy, journalists, other stakeholders such as teachers, medical practitioners, community-based rehabilitation workers, inclusion champions and UNICEF staff;

• Focus Group Discussions (FGDs) with children and adolescents with disabilities and their caregivers;

• Case studies of children and adolescents participating in the #ThisAbility Flagship initiative;• Cost-effectiveness analysis of the #ThisAbility Flagship initiative; and• Surveys.

69 The actual final decisions on the detailed questions will be taken in the inception phase, based on the following principles: 1) Importance and priority: the information should be of a high level of importance for the various intended audiences

of the evaluation;2) Usefulness and timeliness: the answer to the questions should not be already well known or obvious, additional

evidence is needed for decision;3) Answerability and realism: all the questions can be answered using available resources (budget, personnel)

and within the appropriate timeframe; data and key informants are available and accessible, and performance standards or benchmarks exist to answer the questions; and

4) Actionability: the questions will provide information which can lead to recommendations that be acted upon to make improvements.

70 The proposed methodology is just indicative, and based on internal experience in conducting similar evaluations. The will be a need to develop a detailed design, analytical methods and tools during the inception phase based on additional literature review and in consultation with UNICEF Malaysia.


56

The data collected should be disaggregated by age, gender, ethnicities, religion or faith, disability category, state, etc. where relevant and focus on the #ThisAbility Flagship activities both at the national and sub-national level. Sampling of KIIs and FGDs should be done in consultation with UNICEF. Baseline data will be provided based on the KAP survey conducted in 2016. Additionally, secondary data sources can be used. Other secondary data specific to the #ThisAbility Flagship will be provided by UNICEF, such as monitoring reports.

Likewise, conventional ethical guidelines are to be followed during the evaluation. Specific reference is made to the UNEG Norms and Standards and Ethical Guidelines, as well as to the UNICEF’s Evaluation Policy, the UNEG Guidance on Integrating Human Rights and Gender Equality in Evaluation, the UN SWAP Evaluation Performance Indicator,71 and the UNICEF Procedure for Ethical Standards in Research, Evaluation and Data Collection and Analysis and UNICEF’s Evaluation Reporting Standards.72 Good practices not covered therein are also to be followed. Any sensitive issues or concerns should be raised with the Evaluation Manager as soon as they are identified.

5. Management and Conduct of the Evaluation5.1. Evaluation Management StructureThe evaluation will be conducted by an external evaluation team to be recruited by UNICEF Malaysia. The evaluation team will operate under the supervision of the Evaluation Specialist at UNICEF Malaysia that will act as evaluation manager and therefore be responsible for the day-to-day oversight and management of the evaluation and for the management of the evaluation budget. The evaluation manager will assure the quality and independence of the evaluation and guarantee its alignment with UNEG Norms and Standards and Ethical Guidelines and other relevant procedures, provide quality assurance checking that the evaluation findings and conclusions are relevant and recommendations are implementable, and contribute to the dissemination of the evaluation findings and follow-up on the management response. Additional quality assurance will be provided by the Regional Evaluation Adviser. The final report will also be approved by the Country Representative at UNICEF Malaysia. A reference group will be established, bringing together the Chief of Child Protection at UNICEF Malaysia, the Disability Consultant, the Chief of Communication for Development, the Innovation Specialist, the Education Specialist and representatives from UNICEF EAPRO (Regional Disability Focal Point), UNICEF Headquarters (Disability and Child Protection Divisions), JKM (Department for Persons with Disabilities/National Council for Persons with Disabilities), civil-society organizations, Disabled People’s Organisations/Disability Activist and Academia (Malaysian University academics/Study Centres that are specialists on Disability Studies). The reference group will have the following role: contribute to the preparation and design of the evaluation, including providing feedback and comments on the inception report and on the technical quality of the work of the consultants; provide comments and substantive feedback to ensure the quality – from a technical point of view – of the draft and final evaluation reports; assist in identifying internal and external stakeholders to be consulted during the evaluation process; participate in review meetings organized by the evaluation management team and with the evaluation team as required; play a key role in learning and knowledge sharing from the evaluation results, contributing to disseminating the findings of the evaluation and follow-up on the implementation of the management response.

71 Please refer to: http://www.unicef.org/evaluation72 https://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF


57

5.2. Evaluation Team ProfileThe evaluation will be conducted by engaging an institution. The proposed team consists of one (1) international senior-level consultant (Team Leader) to conduct the evaluation that will be supported by at least one (1) national consultant (Team Member/Technical Expert).

The Team Leader should bring the following competences:• Having extensive evaluation experience (at least 15 years) with an excellent understanding

of evaluation principles and methodologies, including capacity in an array of qualitative and quantitative evaluation methods, and UNEG Norms and Standards.

• Having extensive experience on social inclusion interventions – planning, implementing, managing or monitoring and evaluation.

• Holding an advanced university degree (Masters or higher) in international development, public policy, public health, sociology, anthropology or similar, including sound knowledge of policy and systemic aspects; familiarity with the rights of children with disabilities.

• Bringing a strong commitment to delivering timely and high-quality results, i.e., credible evaluations that are used for improving strategic decisions.

• Having in-depth knowledge of the UN’s human rights, gender equality and equity agendas.• Having a strong team leadership and management track record, as well as excellent

interpersonal and communication skills to help ensure that the evaluation is understood and used.

• Specific evaluation experience of disability programming is strongly desired, but is secondary to a strong mixed-method evaluation background, so long as the disability expertise of the other team member (see below) is harnessed to ensure the team’s collective understanding of issues relating to disability programming from a UN or NGO perspective.

• Previous experience of working in an East Asian context is desirable, together with understanding of Malaysia context and cultural dynamics.

• The Team Leader must be committed and willing to work independently, with limited regular supervision; s/he must demonstrate adaptability and flexibility, client orientation, proven ethical practice, initiative, concern for accuracy and quality.

• S/he must have the ability to concisely and clearly express ideas and concepts in written and oral form as well as the ability to communicate with various stakeholders in English.

The Team Leader will be responsible for undertaking the evaluation from start to finish, for managing the evaluation, for the bulk of data collection, analysis and consultations, as well as for report drafting in English and communication of the evaluation results.

One (1) national Team Member/Technical Expert:• Holding advanced university degrees (Masters-level) in international development, public

policy or similar. • Hands-on experience in collecting and analysing quantitative and qualitative data, but this

is secondary to solid expertise in social inclusion interventions.• Strong expertise in equity, gender equality and human rights based approaches to

evaluation and expertise in data presentation and visualisation. • Be committed and willing to work in a complex environment and able to produce quality

work under limited guidance and supervision.


58

• Having good communication, advocacy and people skills and the ability to communicate with various stakeholders and to express concisely and clearly ideas and concepts in written and oral form.

• Excellent Bahasa Malaysia and English communication and report writing skills.The Team Member will play a major role in data collection, analysis and presentation, and preparation of the debriefings and will make significant contributions to the writing of the main evaluation report.

The evaluation team is expected to be balanced with respect to gender to ensure accessibility of both male and female informants during the data collection process. Back-office support assisting the team with logistics and other administrative matters is also expected. It is vital that the same individuals that develop the methodology for the RFPS will be involved in conducting the evaluation. In the review of the RFPS, while adequate consideration will be given to the technical methodology, significant weighting will be given to the quality, experience (CV’s and written samples of previous evaluations) and relevance of individuals who will be involved in the evaluation.

5.3. Evaluation DeliverablesEvaluation products expected for this exercise are:

1) An inception report (in English), including a summary note in preparation for data collection (in both English and Bahasa Malaysia);

2) A report of the initial evaluation findings from primary data collection (in English), including a desk review analysis and a PowerPoint presentation to facilitate a stakeholder consultation exercise;

3) A draft and final report (in English and Bahasa Malaysia) that will be revised until approved (incl. a complete first draft to be reviewed by the evaluation management team and UNICEF; a second draft to be reviewed by the reference group and Regional Evaluation Adviser within UNICEF EAPRO, and a penultimate draft). The final report will need to be developed in several media to maximize its utility and accessibility including in text and audio versions;

4) A PowerPoint presentation (in both English and Bahasa Malaysia) to be used to share findings with the reference group and for use in subsequent dissemination events; and

5) A four-page executive summary (in both English and Bahasa Malaysia) that is distinct from the executive summary in the evaluation report and it is intended for a broader, non-technical and non-UNICEF audience. The executive summary should also be produced both in text and audio versions.

Other interim products are: • Minutes of key meetings with the evaluation management team and the reference group;

and• Presentation materials for the meetings with the evaluation management team and

the reference group. These may include PowerPoint summaries of work progress and conclusions to that point.

Outlines and descriptions of each evaluation products are meant to be indicatives, and include: • Inception report: The inception report will be key in confirming a common understanding

of what is to be evaluated, including additional insights into executing the evaluation. At this stage, evaluators will refine and confirm evaluation questions, confirm the scope


59

of the evaluation, further improve on the methodology proposed in the ToR and their own evaluation proposal to improve its rigor, as well as develop and validate evaluation instruments. The report will include, among other elements: i) evaluation purpose and scope, confirmation of objectives and the main themes of the evaluation; ii) evaluation criteria and questions, final set of evaluation questions, and evaluation criteria for assessing performance; iii) evaluation methodology (i.e., sampling criteria), a description of data collection methods and data sources (incl. a rationale for their selection), draft data collection instruments, for example questionnaires, with a data collection toolkit as an annex, an evaluation matrix that identifies descriptive and normative questions and criteria for evaluating evidence, a data analysis plan, a discussion on how to enhance the reliability and validity of evaluation conclusions, the field visit approach, a description of the quality review process and a discussion on the limitations of the methodology; iv) proposed structure of the final report; v) evaluation work plan and timeline, including a revised work and travel plan; vi) resources requirements (i.e., detailed budget allocations, tied to evaluation activities, work plan) deliverables; v) annexes (i.e., organizing matrix for evaluation questions, data collection toolkit, data analysis framework); and vi) an evaluation briefing note for external communication purposes. The inception report will be 15-20 pages in length (excluding annexes), or approximately 15,000 words, and will be presented at a formal meeting of the reference group.

• Initial evaluation findings: This report will present the initial evaluation findings from primary data collection, comprising the desk-based document review and analysis of the disability inclusion flagship programme. The report developed prior to the first drafts of the final report should be 10 pages, or about 8,000 words in length (excluding annexes, if any), and should be accompanied by a PowerPoint presentation that can be used for validation with key stakeholders.

• Final evaluation report: The report will not exceed 40 pages, or 25,000 words, excluding the executive summary and annexes; it will be produced both in text and audio versions.73

• PowerPoint presentation: Initially prepared and used by the evaluation team in their presentation to the reference group, a standalone PowerPoint will be submitted to the evaluation management team as part of the evaluation deliverables.

• An evaluation briefing note, data and a four-page executive summary for external users will be submitted to the evaluation management team as part of the evaluation deliverables, in both text and audio versions.

• Reports will be prepared according to the UNICEF Style Guide and UNICEF Brand Toolkit (to be shared with the winning bidder) and UNICEF standards for evaluation reports as per GEROS guidelines (referenced before). All deliverables must be in professional level standard English and they must be language-edited/proof-read by a native speaker.

• The first draft of the final report will be received by the Evaluation Manager and UNICEF who will work with the team leader on necessary revisions. The second draft will be sent to the reference group for comments. The Evaluation Manager will consolidate all comments on a response matrix, and request the evaluation team to indicate actions taken against each comment in the production of the penultimate draft.

73 UNICEF has instituted the Global Evaluation Report Oversight System (GEROS), a system where final evaluation reports are quality assessed by an external company against UNICEF/UNEG Norms and Standards for evaluation reports. The evaluation team is expected to reflect on and conform to these standards as they write their report. The team may choose to share a self-assessment based on the GEROS with the evaluation manager.


60

Bidders are invited to reflect on each outline and effect the necessary modification to enhance their coverage and clarity. Having said so, products are expected to conform to the stipulated number of pages where that applies.

An estimated budget has been allocated for this evaluation. As reflected in Table 1, the evaluation has a timeline of four months from October 2018 to January 2019. Adequate effort should be allocated to the evaluation to ensure timely submission of all deliverables, approximately 12 weeks on the part of the evaluation team.

Table 1: Proposed Evaluation Timeline74

ACTIVITY DELIVERABLE TIME ESTIMATE RESPONSIBLE PARTY

1. INCEPTION, DOCUMENT REVIEW AND ANALYSIS

3 weeks, concurrent(Oct, 2018)

1. Inception meeting by Skype with the Evaluation Specialist, Chief, Child Protection and Disability Consultant

Meeting minutes Week 1 Evaluation team, evaluation manager

2. Inception visit (incl. initial data collection and desk review; development of evaluation matrix, methodology and work plan, data collection material, drafting of the inception report)

Draft inception report

Week 1 Evaluation team

3. Present draft inception report to the reference group

PowerPoint presentation

Week 2 Evaluation team, evaluation manager, reference group

4. Receive inception report and feedback to evaluation team

Evaluation commenting matrix

Week 3 Evaluation manager, reference group

5. Present inception report, confirm planning for field visit

Final inception report


2. DATA COLLECTION5 weeks, consecutive(Oct to Nov, 2018)

1. Pilot data collection tools and conduct field-based data collection

- Weeks 4-7 Evaluation team

2. Prepare initial evaluation findings report and prepare presentation for validation workshop to validate data collection results

Initial evaluation findings report (incl. desk review), PowerPoint presentation, meeting minutes


74 Please note that the timing of the data collection may change depending on the possibility of carrying out KIIs and FGDs and other contextual factors.


61

3. REPORTING AND COMMUNICATION OF RESULTS

4 weeks, consecutive(Dec, 2018 – Jan 2019)

1. Prepare and submit first draft of evaluation report

Draft report Week 9 Evaluation team

2. Receive first draft and feedback to evaluation team


Week 10 Evaluation manager

3. Prepare and submit second draft of evaluation report


4. Receive second draft and feedback to evaluation team


Weeks 12-13 Evaluation manager, reference group

5. Prepare and submit penultimate draft of evaluation report


6. Submit and present final report to reference group and prepare presentation and other materials

Final report, executive summary, PowerPoint presentation, meeting minutes


6. Payment ScheduleUnless the proposers propose an alternative payment schedule, payments will be as follows:

• Approved inception report: 25% of the contractual amount;• Approved initial evaluation findings report: 30% of the contractual amount; • Approved final report: 30%; and • Approved final presentation and other materials: 15%.

7. Application ProcessEach proposal will be assessed first on its technical merits and subsequently on its price. In making the final decision, UNICEF considers both technical and financial proposals. The Evaluation Team first reviews the Technical Proposals followed by review of the Financial Proposals of the technically compliant firms. The proposal obtaining the highest overall score after adding the scores for the technical and financial proposals together, that offers the best value for money, will be recommended for award of the contract.

The Technical Proposal should include but not be limited to the following:a) Request for Proposals for Services Form (provided above).b) Presentation of the Bidding Institution or institutions if a consortium (maximum two

institutions will be accepted as part of the consortium), including:• Name of the institution;• Date and country of registration/incorporation; • Summary of corporate structure and business areas;


62

• Corporate directions and experience;• Location of offices or agents relevant to this proposal;• Number and type of employees;• In case of a consortium of institutions, the above listed elements shall be provided for

each consortium members in addition to the signed consortium agreement; and• In case of a consortium, one only must be identified as the organization lead in dealing

with UNICEF.

c) Narrative Description of the Bidding Institution’s Experience and Capacity in the following areas: • Evaluation of social inclusion interventions; • Process evaluation of social inclusion interventions, ideally implemented by government

institutions;• Previous assignments in developing countries in general, and related to social inclusion

programmes, preferably in East Asia; and• Previous and current assignments using UNEG Norms and Standards for evaluation.

d) Relevant References of the proposer (past and on-going assignments) in the past five years. UNICEF may contact references persons for feedback on services provided by the proposers.

e) Samples or Links to Samples of Previous Relevant Work listed as reference of the proposer (at least three), on which the proposed key personnel directly and actively contributed or authored.

f) Methodology. It should minimize repeating what is stated in the ToR. There is no minimum or maximum length. If in doubt, ensure sufficient detail.

g) Work Plan, which will include as a minimum requirement the following: • General work plan based on the one proposed in the ToR, with comments and proposed

adjustments, if any; and• Detailed timetable by activity (it must be consistent with the general work plan and the

financial proposal).

h) Evaluation Team: • Summary presentation of proposed experts;• Description of support staff (number and profile of research and administrative assistants

etc.);• Level of effort of proposed experts by activity (it must be consistent with the financial

proposal); and• CV of each expert proposed to carry out the evaluation.


63

The Technical Proposal will be submitted in hard copy and electronic (PDF) format.

Please note that the duration of the assignment will be from October 2018 to January 2019, and it is foreseen that the Team Leader and the Team Member will devote roughly half of their time to the evaluation. The presence of a conflict of interest of any kind (e.g., having worked for or partnered with UNICEF on the disability inclusion flagship programme in the design or implementation phase will automatically disqualify prospective candidates from consideration).

The Financial Proposal should include but not be limited to the following:Bidders are expected to submit a lump sum financial proposal to complete the entire assignment based on the different sample sizes (as per instruction given under Section 8 below). In order to arrive at the lump-sum offer for a given sample size, the firm may work out the budget detail as below:

a) Resource Costs: Daily rate multiplied by number of days of the experts involved in the evaluation.

b) Conference or Workshop Costs (if any): Indicate nature and breakdown if possible. c) Travel Costs: All travel costs should be included as a lump sum fixed cost. For all travel

costs, UNICEF will pay as per the lump sum fixed costs provided in the proposal. A breakdown of the lump sum travel costs should be provided in the financial proposal.

d) Any Other Costs (if any): Indicate nature and breakdown. e) Recent Financial Audit Report: Report should have been carried out in the past 2 years

and be certified by a reputable audit organization.

Bidders are required to estimate travel costs in the Financial Proposal. Please note that: i) travel costs shall be calculated based on economy class fare regardless of the length of travel; and ii) costs for accommodation, meals and incidentals.

The financial proposal must be fully separated from the Technical Proposal. The financial proposal will be submitted in hard copy and electronic (PDF) format. Costs will be formulated in US$ and free of all taxes.

8. Evaluation Weighting CriteriaThe proposals will be evaluated against the two elements: technical and financial. The ratio between the technical and financial criteria depends on the relative importance of one component to the other. Cumulative Analysis will be used to evaluate and award proposals. The evaluation criteria associated with this ToR is split between technical and financial as follows:

• Weightage for Technical Proposal = 70%• Weightage for Financial Proposal = 30%• Total Score = 100%

a. Technical Proposal:The Technical Proposal should address all aspects and criteria outlined in this Request for Proposal.


64

Table 2: Evaluation of Technical Proposal

The Technical Proposals will be evaluated against the following:REF CATEGORY POINTS1 Overall response:

• Completeness of response • Overall concord between RFP requirements and proposal

23

2 Company/key personnel/Individual Consultant: • Range and depth of experience with similar projects • Samples of previous work • References• Key personnel: relevant experience and qualifications of the

proposed team for the assignment

85514

3 Proposed methodology and approach: • Detailed proposal with main tasks, including sound methodology to

achieve key outputs• Proposal presents a realistic implementation timeline

20

13

Total Technical 70

Only proposals which receive a minimum of 55 points will be considered further.

b. Financial ProposalThe total amount of points allocated for the price component is 30. The maximum number of points will be allotted to the lowest price proposal that is opened and compared among those invited firms/institutions which obtain the threshold points in the evaluation of the technical component.

All other price proposals will receive points in inverse proportion to the lowest price, e.g., Max. score for price proposal * Price of lowest priced proposalScore for price proposal X = ------------------------------------------------------------------------------

Price of proposal

ANNEX B: EVALUATION MATRIXOverarching questions Sub-questions Indicators

Effectiveness of the UNICEF’s strategy to children with disability and its implementation considering: To what degree has UNICEF’s work on disability, and in particular through its #ThisAbility Flagship initiative, contributed to the creation of positive conditions to reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia? Have there been any unintended results?

How satisfied are the children and their caregivers with the quality of the services they have received through UNICEF’s partners?

Percent satisfied

Average level of satisfaction

Qualitative evidence of satisfaction

To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights based perspective?

Evidence of training materials incoprorating a social/rights-based perspective

Percent reporting change in understanding

Qualitative evidence of change in understanding

How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice?

Qualitative evidence of change in awareness

Documented evidence of changes in strategies

Qualitative evidence of change in strategies

How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included?

Interviewee assessments of outcomes of private sector activities

Private sector interviewee assessments of role of UNICEF in fostering partnerships

To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities?

Alignment of UNICEF Malaysia’s strategy with barriers prioritized by children/YP or evidenced in literature

Alignment of UNICEF Malaysia’s strategy with international best practice/rights-based approach

Interviewee perspectives on appropriateness of strategy and inclusion of relevant stakeholders in programme design

Quantitative evidence of increased public awareness

To what extent has the #ThisAbility Flagship initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities?

Interviewee assessments of outcomes of cross-sectoral workDocumented evidence of changes in programming in all sections

Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship?

Evidence of gender parity in disaggregated output and outcome indicators (by gender)

Interviewee perspectives on inclusion of gender-sensitive approaches

Evidence of gender mainstreaming approaches in strategy documents

Sources of information Key documents Main informants

(qual)

Internal document review

FGOs

Feedback forms/MI - if available from UNICEF or partners (TBC)

Disabled young peopleParents of disabled children


FGOs

KIIs

DET documentationPeter Tan - UNICEF DET 2017Worksheet DET UNICEF 2017Outcomes-Outputs and Indicator Status by Regional / Business Area seep.9.29Feedback forms/MI - if available (TBC)

Disabled young peopleParents of disabled childrenUNICEF staff


External document review

KIIs

Activities matrixPartnership agreementsCompany documentation2016 COAR Disability Summary AnalysisUNICEF Rolling Workplan 2018 FinalThisability activity matrix 2017-2019 (project 3,10)Company documents including websites

UNICEF staffPartners


FGOs

KIIs

Activities matrixPartnership agreementsChild Rights in Business PrinciplesMI - if available

Disabled young peopleParents of disabled childrenUNICEF staffPartners


External document review

FGOs

KIIs

MI (including social media)_thisability Flagship MonitoringDisability KAPCountry Programme Action PlanCountry Programme DocumentCountry Programme Management PlanRAMEN_X-Sectoral Programme Guidance Disability FinalStrategic PlanUNSPFR

Disabled young peopleParents of disabled childrenUNICEF staffPartnersGovernment


FGOs

KIIs

Task Force meeting minutesRAM ReportsAnnual ReportsDisability Flagship meeting minutesMI information

Disabled young peopleParents of disabled childrenUNICEF staff


FGOs

KIIs

EN_Annex A_Intervention by Focus Area_thisability Flagship Monitoring Framework#thisability Flagship Working PaperAnnex B_Key Messaging DisabilityFinal File Childhood Disability in Malaysia


Relevance of the programme strategy and approach, considering: How relevant is UNICEF’s work on disability and its #ThisAbility Flagship initiative to the developmental challenges in Malaysia, and particularly attitudes and practices with regard to disability?

How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the SDG road-map?

Alignment of UNICEF Malaysia’s strategy/TOC with barriers prioritized by children/YP and their familiesInterviewee perspectives on appropriateness of strategy and inclusion of relevant stakeholders in programme designDisaggregation of evidence by subgroup (gender, age, region, impairment, etc.)

How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050?

Alignment of UNICEF Malaysia’s strategy/TOC with Malaysia Vision 2020 and National Transformation 2050Interviewee perspectives on alignment of UNICEF’s work with national goals and activities

To what extent is #ThisAbility Flagship initiative aligned with the new UNICEF’s Strategic Plan?

Alignment of UNICEF Malaysia’s strategy/TOC with Malaysia Vision 2020 and National Transformation 2050Interviewee perspectives on alignment of UNICEF’s work with national goals and activities

Has the design of the #ThisAbility Flagship initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable?

Evidence of programme strategies and documents disaggregated by sub-groupInterviewee perspectives on inclusiveness of UNICEF’s approachChildren/family perspectives on alignment of UNICEF’s approach with needs


Evidence of processes for engagement with children being in placeInterviewee perspectives on the effectiveness of processesAlignment of strategy with priorities and needs expressed by children and their families

Efficiency of the programme implementation and management, considering: To what extent and how has UNICEF mobilized and used its resources (human, technical and financial) and improved its cross-sectoral coordination to achieve its planned results for children with disabilities?

To what degree have UNICEF resources (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility Flagship initiative?

Interviewee perspectives on capacity to implement activities in TOCDocumented evidence of budget and staff resources to carry out activities in TOCEvidence of effective governance structures/ proceduresBenchmarking of resources against comparator programmes

How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?

Number of FTE UNICEF and partner staff working on the rights of children with disabilities (disaggregated by years of experience in disability)Interviewee perspectives on capacity

Sustainability in the programme, considering: To what extent are the benefits and achievements of the #ThisAbility Flagship initiative likely to continue after 2020 through national ownership and changes in terms of attitudes, knowledge and practices?

To what degree has the #ThisAbility Flagship initiative thus far contributed to the generation of sustainable capacities at the national and sub-national levels?

Evidence of capacity developedInterviewee perspectives on risks to sustainability and likely sustainabilityEvidence of mitigation strategies for risks identified

What are the enabling as well as containing factors that are likely to influence future cross-sectoral work on focus issues?

Interviewee perspectives on key factorsLessons learnt from existing activities

UNICEF strategy docsUNICEF reportsSDG documentationExternal documents

Disability KAPIts_About_Ability FinalUNICEF Children_with_ DisabilityUNICEF Childhood Disability in MalaysiaSWCR2013_ENG_ Lo_res24AprilCountry Programme Action PlanCountry Programme DocumentCountry ProgrammeSDG documents


Internal document reviewExternal document reviewFGOsKIIs

Disability KAPIts_About_Ability FinalUNICEF Children_with_ DisabilityUNICEF Childhood Disability in MalaysiaSWCR2013_ENG_ Lo_res24AprilCountry Programme Action PlanCountry Programme DocumentCountry Programme Government documents


Internal document reviewFGOsKIIs

_thisabilty Flagship Monitoring Framework#thisabilty Flagship Woring PaperAnnex B_Key Messaging DisabilityFinal File Childhood Disability in MalaysiaAnnex 4-2018-03-16 PRIME 2018-2019Annual Management Plan 2018MCO-CPAP 2016-2020 SignedFINAL CPMP 11052015 MLY

Disabled young peopleParents of disabled children UNICEF staff


EN_Annex A_Intervention by Focus Area_thisabilty Flagship Monitoring Framework#thisabilty Flagship Woring PaperAnnex B_Key Messaging DisabilityFinal File Childhood Disability in Malaysia



Final File Childhood Disability in Malaysia2017-2018-Final Result Framework-activities 4,5,11Thisability activities matrix 2017-2019

UNICEF staffDisability stakeholdersDisabled young peopleParents of disabled children

Internal document reviewKIIs

Annual Management PlanUNICEF Rolling WorkplanRAMOrganigramEvaluations of other UNICEF flagships

UNICEF staffPartners


Data request to UNICEF and partnersOrg chart


FGOsKIIs


KIIs UNICEF staff


67

ANNEX C: RESEARCH TOOLSThis annex contains the research tools for the key informant interviews (KIIs) and focus group discussions (FGDs). These tools were developed in line with the evaluation matrix. Each KII had a duration of approximately 120 minutes.

Tool 1: Key Informant Interview – Topic GuideNote to InterviewerThese consultations aim to inform the development of a Theory of Change (ToC) for the UNICEF Malaysia #ThisAbility Flagship Initiative, as well as to address the specific questions in the evaluation framework. Equal International and Ipsos will complete consultations with representatives from UNICEF, its partners, and government stakeholders.

This topic guide asks stakeholders about: • their experiences of the Initiative;• inputs, activities and outputs of the Initiative, intended outcomes, long-term impacts and key

risks and assumptions underpinning the ToC;• wider contextual factors to be aware of;• views on the strategy of the Initiative and its alignment with the needs of its intended

beneficiaries.Consultations are semi-structured; interviewers should be aware that not all consultees will be able to provide responses to all questions, and should therefore tailor interviews accordingly by selecting relevant questions, probes and according to previous involvement in the programme.

Introduction (All)Interviewer Note: Interviewers should give the following introduction before the start of each interview (although this should be tailored for each consultee):

My name is … and I am from Ipsos, the market and social research company. Your contact details were provided to my organization by UNICEF. As you know, Ipsos and our partners Equal International have been commissioned by UNICEF Malaysia to complete an evaluation of the #ThisAbility Flagship Initiative. The evaluation aims to assess the implementation of the Initiative to date and its future strategy.

The purpose of this consultation is to support our understanding of the Initiative. This includes discussing:

• the rationale for the Initiative, its objectives and anticipated results;

• your experience of the Initiative;

• the wider context for the Initiative.

Information collected in consultations will be used to evaluate the relevance, effectiveness, efficiency, impact and sustainability of the #ThisAbility Flagship initiative. Personal Data shall only be processed for the purposes of UNICEF’s #ThisAbility Flagship Initiative Evaluation.


68

The study team will not attribute quotes to any consultees. However, given the specific roles of consultees, complete anonymity cannot be guaranteed. Please be assured that you can request the interview to be stopped at any time. You are not obliged to provide responses to any question during the interview.

Interviewer Note: Please obtain the following consultee consent prior to starting the interview:• Obtain verbal consent from the consultee that they are happy to complete the interview. Note

their response. • If a recorder is used, request permission to record the interview prior to starting the recording.

Additionally, obtain consent from the consultee to complete the interview once the recording has started. State that the recording will be securely destroyed upon completion of the evaluation.

• Close the interview if consent is not provided by the consultee. • Check that the consultee has any questions about the consultation before moving on.

Overview (All)- Please could you give me a brief overview of your role?

o Is your role disability-focussed? If not, to what extent does your work relate to disability?o How long have you been in this role?

- For government and partners:

o Please describe your organization and its key activities.o Is the organization’s remit regional or national?o What is the nature of the relationship of your organization with UNICEF?

- For disability partners:

o Please describe to what extent and in what ways your organization works on behalf of children and young people with disabilities.

o To what extent are your organization’s activities informed by: Agenda 2030/the SDG agenda?Malaysia Vision 2020, National Transformation 2050, and any other national goals

or activities?The needs or priorities identified by children/young people with disabilities and/or

their families?• Have you undertaken any direct consultations with children and young people with

disabilities as part of your activity planning?o Do you work in collaboration with any other organizations/agencies in relation to

disability?o What services does your organization provide that are inclusive for children and

young people with disabilities?o Are there any specific gender issues that you take into consideration when providing

these services? Are there any other ways in which you take into account the different needs of children, especially the most vulnerable?


69

- For mainstream partners:

o Prior to becoming involved in the #ThisAbility Flagship Initiative, did your organization have a disability inclusion strategy (whether formally or informally)?

- How have you been involved in the UNICEF #ThisAbility Flagship Initiative?o When did you become involved?o Probe on any involvement in: design, implementation, or management of the initiative.

When, how and why did you become involved?o Probe on involvement in/awareness of specific activities (see box A).

Box A: Specific activities delivered by the #ThisAbility Flagship Initiative

Programme design related activities, for example:(1) Inclusive Playground design collaboration

(2) ‘Inclusion Champions’ – identification of private sector partners

Capacity building activities, e.g.:(3) NGO Impact Academy – 15-week course for NGO partners

(4) Disability Equality Training (DET) – workshops for UNICEF staff, partners, and government

(5) Media Disability Training – training for journalists

(6) Disability-Inclusion Teacher Training – pilot project in Sabah

(7) Design and development of Inclusive Ethos training for Petrosains

Activities with partners, e.g.:(8) National launch of the Childhood Disability: KAP Study – with Petrosains

(9) #ThisAbility Calendar – with MPH Book Store

(10) Disability Theory of Change Exercise with MCO

Public activities, for example:(11) #ThisAbility Makeathon – team-based activities for children

(12) #ThisAbility public advocacy campaigns – awareness-raising campaigns, including Borneo International Marathon, ASEAN Para Games, Manchester United PSA campaign, What’s So Special? Campaign

(13) National launch of the Childhood Disability: KAP Study

(14) National Conference on Inclusion ‘One Child, One Family, One Community’

(15) Disability Inclusion Communication for Development (C4D) Ground Activation – Carnival in 2018, and planned activity in 2019

(16) #ThisAbility Truck – branding of truck with messaging


70

Programme design (UNICEF staff, disability partners, and government)

Rationale for the Initiative- In your own words, how would you describe the aims of the #ThisAbility Flagship Initiative?

What is the rationale for the Initiative?- In your own words, how would you describe the overall strategy for the Initiative? How does

the Initiative intend to achieve its expected aims?o How was this strategy developed?o Could you please describe how UNICEF Malaysia first began to work on disability

issues? What kind of things were happening at that time?o In your view, what prompted this work to start?

- In your view, how relevant and appropriate has the Initiative been to the following goals and strategies, and (if you are aware) how were these considered in the design of the programme:

o Agenda 2030/the SDG agenda?o Malaysia Vision 2020, National Transformation 2050, and any other national goals or

activities?o UNICEF’s Strategic Plan?o Are there any other goals or strategies that are relevant to the Initiative?

- What do you consider are the main barriers facing children and young people in Malaysia in the sector that you work in?

- In what ways do you see disability as having an impact in your area of work?- In your view, how relevant and appropriate has the Initiative been to the following:

o The barriers prioritized by children/young people and their families?o The different needs of boys and girls?o The different needs of children of different ages?o The different needs of children with different ethnicities, religions or faiths?o The different needs of children across disability categories?o The different needs of children with different legal status (e.g. migrants) or other social

identities (such as working status, education level and socio-economic status)?o The most vulnerable children?

- Who would you consider the key stakeholders for the Initiative? To what extent were relevant stakeholders involved in the design of the Initiative?

o To what extent have the voices of children and adolescents with disabilities been taken account in the design and implementation of the programme?

o Have you undertaken any direct consultations with children and young people with disabilities as part of programme planning? If yes, can you please describe what kind of accommodations were made for children of different ages/gender/impairments.

o What processes are in place for engaging with children throughout implementation of the programme, and do you consider these to be effective?


71

Implementation of the Initiative- How do you think the way UNICEF Malaysia approaches disability has changed during your

time working with/at UNICEF?- In your view, what are the strengths and weaknesses of the UNICEF team working on the

#ThisAbility Flagship Initiative?o Probe on: knowledge of disability, design of the Initiative, management of the

Initiative, cross-sectoral engagement, financial managemento Does the UNICEF team have sufficient human, technical and financial resources to

achieve its mission?

Implementation and management (UNICEF staff)UNICEF processes

- From your perspective, how is UNICEF Malaysia’s work on disability inclusion coordinated? What guidelines or guidance have you been following? Do you have a strategy that you are using? (Please provide any documents you might have that outline your strategic-level approach to disability.)

- To what extent has the #ThisAbility Flagship Initiative enabled cross-sectoral work? In what ways have you collaborated with other sectors in the development and implementation of disability-inclusive activities/programmes? Can you provide any specific examples where you have collaborated?

- How do you currently measure the inclusion and participation of children and young people with disabilities in your programmes? Please provide details including the extent to which this information is further disaggregated by age, gender and other vulnerability variables.

Internal results- To what extent has the Initiative changed your knowledge of children with disabilities?

o What was your level of knowledge prior to being engaged in the Initiative?o What were your key learnings?o What do you understand are the differences between taking a charity/medical-

based approach to disability inclusion and a human rights-based approach?- Has cross-sectoral work resulted in changes in programming across sections, or in the way

programmes have been delivered? If so, how was this achieved?- Has cross-sectoral work resulted in increased access to high quality services for children

with disabilities? If so, how? Why was the cross-sectoral collaboration important?- What factors were important for success? What internal or external factors may have

constrained success?- To what extent are these results likely to be sustained over time? Why (or why not) might

the results be sustained? How could sustainability of results be improved?


72

Capacity building activities (Respondents who have been involved in capacity building activities)

Overview- You said that you have been involved in capacity-building activities under the #ThisAbility

Flagship Initiative. In your own words, what is the rationale for these capacity building activities?

- How effective do you consider the capacity building activities to have been in achieving their intended aims? Why/why not?

Results- For UNICEF staff:

o To what extent have capacity building activities (especially Disability Equality Training) been effective in increasing knowledge of children with disabilities? What evidence of this result exists? How was this achieved?

o To what extent have capacity building activities (especially Disability Equality Training) been effective in changing understanding of disability from an individual to a social/rights based-perspective? What evidence of this result exists? How was this achieved?

- For all others:

o Please describe in what ways you believe your organization has benefitted?Probe on: increased awareness, new programmes or services, new

partnerships.o To what extent have capacity building activities changed your knowledge of children

with disabilities? What was your level of knowledge prior to completing the activity/activities? What were your key learnings?

o What do you understand are the differences between taking a charity/medical-based approach to disability inclusion and a human rights-based approach?

- All:

o What factors were important for success? What internal or external factors may have constrained success?

o To what extent are these results likely to be sustained over time? Why (or why not) might the results be sustained? How could sustainability of results be improved?

Activities with partners (Partners and other respondents who have been involved in partnerships)

Overview (Partners)- Please describe to what extent and in what ways your organization works to include

children and young people with disabilities.o What services do you provide that are inclusive for children and young people with

disabilities?


73

o Are there any specific gender issues that you take into consideration when providing these services?

o When did your organization first begin to work on disability inclusion? Do you know what prompted that?

- Does your organization follow any guidelines or guidance in regards to your partnership with UNICEF on disability? Do you have a strategy that you are using or have any overarching aims/objectives? Would you be able to share any documents you have which outline your approach to disability?

- Has UNICEF provided your organization with any training or support in the area of disability rights/disability inclusion? If yes, please describe the training/support you have received. What were the benefits? (Probe: increased awareness, new programmes or services, new partnerships.)

Rationale/Aims- From your perspective, what do you think UNICEF Malaysia’s overall objective is when it

comes to developing partnerships (with organizations like yours)?- How effective do you consider the partnership/partnership activities to have been in

achieving the intended aims? Why/why not?- For partners:

o Has your organization undertaken any direct consultations with children and young people with disabilities as part of your activity planning?

Results- For partners:

o How does your organization currently measure the inclusion and participation of children and young people with disabilities in your programmes/activities/services? Are any of these measures further disaggregated by gender or any other vulnerability variables?

o Do you have an organizational policy on the inclusion of people with disabilities in your workforce? Has this been influenced in any way by your organization’s involvement with UNICEF?

- All:

o To what extent have partnerships resulted in empowerment of children and adolescents? Has empowerment resulted in children advocating for themselves to ensure they are socially included?

- For partners:

o To what extent has your participation in the Initiative changed your knowledge of children with disabilities? What was your level of knowledge prior to being engaged by UNICEF? What were your key learnings?

o What do you understand are the differences between taking a charity/medical-based approach to disability inclusion and a human rights-based approach?


74

- For UNICEF staff:o What, in your experience, is the general response from partners towards the inclusion

of children and young people with disabilities?- All:

o What factors were important for success of partnership activities? What internal or external factors may have constrained success?

o To what extent are these results likely to be sustained over time? Why (or why not) might the results be sustained? How could sustainability of results be improved?

Public activities (Partners and other respondents who have been involved in public activities)

Rationale/Aims- You said that you have been involved in campaigns or other public activities under the

#ThisAbility Flagship Initiative. In your own words, what is the rationale for these activities?- To what extent were relevant stakeholders involved in the design of public activities?

o To what extent does the strategy align with wider strategies? With the needs prioritized by children or their advocates?

o Was a rights-based approach incorporated in the design of public activities? How?- How effective do you consider the public activities to have been in achieving their intended

aims? Why/why not?

Results- To what extent have these activities resulted in increased public awareness of children with

disabilities? What evidence is available to demonstrate this?- What factors were important for success? What internal or external factors may have

constrained success?- To what extent are these results likely to be sustained over time? Why (or why not) might

the results be sustained? How could sustainability of results be improved?

Other (All)- Have there been any unintended results from the Initiative, whether positive or negative?

How did these results come about?- In your view, what types of activities would you like to see UNICEF undertake in the future?

Are there any activities that UNICEF is currently undertaking through the #ThisAbility Flagship Initiative that are less valuable?

- Do you have any other comments or suggestions for the #ThisAbility Flagship Initiative which this evaluation should consider?

Contextual factors (Disability partners)- What, in your experience, is the general response from the public towards the inclusion of

children and young people with disabilities? Does this vary when you take factors like age, gender or impairment into consideration?


75

- How prepared do you believe government is to incorporate the rights of children with disabilities into the delivery of services and programmes?

- In your experience, what are the key needs or priorities for children with disabilities or their families? How are these needs currently being met by your organization, the government, UNICEF, and others?

- What are the greatest challenges your organization faces in working inclusively with children and young people with disabilities? Does this vary when you take factors like age, gender or impairment into consideration?

- What have been the biggest benefits gained from working inclusively with children and young people with disabilities?

Close (All)Interview Note: To finish the interview, check through the list of activities below:

- Request permission to re-contact interviewee to clarify information- Check consent to use the consultation to support the evaluation- Reminder that participation is anonymous- Thank and close.

Tool 2: Focus Group Discussion Overview

Below is a list of topics for inclusion in focus group discussions with children with disabilities and their parents/carers. These topics have been selected in line with the evaluation matrix. Appropriate tools and any necessary stimuli will be developed as part of the research phase following confirmation of the Inception Report. Each FGD will have a duration of 90–120 minutes. Prior to the FGDs, a consent form will be collected from each respondent (sample in Annex G)

Discussion topics- Experiences and needs of children with disabilities

o Consideration of different needs according to:GenderAgeEthnicityReligionDisability categoryOther social identities75 (such as working status, education level and socio-

economic status)o Priorities for services and support

75 Social identity is defined as a person’s sense of who they are based on their group membership(s). In this regard, we will be capturing information such as their working status, education level and socio-economic status, which is represented by their income level. This information is obtained from a set of questions administered during the recruitment period.


76

- Experiences of participation in UNICEF programmingo Activities participated ino Perceptions of programme

Overall perceptionsRelevance of the programme to children’s needsSatisfaction with quality of services provided/activitiesConsideration of different needs according to:

• Gender• Age• Ethnicity• Religion• Disability category• Other social identities (such as working status, education level and socio-

economic status)Satisfaction with staff, including knowledge, skills, and responsiveness to needsSatisfaction with opportunities to provide feedback

o Effects of the programme on:Increasing knowledge of children/parentsChanging understanding of children/parents to a rights-based perspectiveAccess to services (and quality of services)Changing behaviour of children/parents

o Likelihood of sustainability of effects- Experiences of participation in UNICEF partner programming

o Activities participated ino Perceptions of programming

Overall perceptionsRelevance of the programme to children’s needsSatisfaction with quality of services provided/activitiesConsideration of different needs according to:

• Gender• Age• Ethnicity• Religion• Disability category• Other social identities (such as working status, education level and socio-

economic status)


77

Satisfaction with staff, including knowledge, skills, and responsiveness to needsSatisfaction with opportunities to provide feedback

o Effects on:Feelings of empowermentChanges in children’s behaviour: advocating for themselvesSocial inclusion of children with disabilitiesAccess to services (and quality of services)

o Likelihood of sustainability of effects- Public advocacy campaigns

o Awareness of campaignso Perceptions of campaigns

Relevance of campaigns to children’s needso Effects on public awareness of children with disabilitieso Likelihood of sustainability of effects

- Future strategyo Types of activities children/carers would like to see UNICEF carry outo Availability of other services, and UNICEF’s unique role


78

ANNEX D: STAKEHOLDERS INTERVIEWED

In total, 42 interviews were conducted; five interviewees were interviewed twice. 1. Manager of Cheshire Homes Services2. Race Director, Secretariat of UNICEF Borneo Marathon3. Coordinator, Secretariat of UNICEF Borneo Marathon 4. Director of Programmes and Services at Seri Mengasih Centre5. Manager at Sabah Art Gallery6. Co-Founder of Persatuan Child Sabah7. Co-Founder of Persatuan Child Sabah8. Principal at Sabah Society for the Deaf9. Director of Sabah State Library10. Manager of Special Olympics Sabah11. President of Special Olympics Sabah12. Executive Council for Sabah Council of Social Services13. Associate #1, Tandemic14. Associate #2, Tandemic15. Director of Research and Innovation, Petrosains16. U Report lead; UNICEF consultant17. Partnerships and Programming Consultant in Sabah, UNICEF18. CSR Officer, UNICEF19. Innovation Specialist, UNICEF20. Communications for development specialist, UNICEF21. Chief of Communication, UNICEF22. Country Representative/ Head of Office, UNICEF23. Deputy Country Representative, UNICEF24. Chief of Child Protection, UNICEF25. Disability Consultant, UNICEF26. Director, Malaysian CARE27. President, National Early Childhood Intervention Council28. Chair, Special Olympics29. President, Malaysian Federation of the Deaf30. Founder, Jesselton Philharmonic Orchestra31. Co-founder, WeCare Journey32. Trainer, Disability Equality Training33. Operations Manager, UNICEF34. Communications Officer, UNICEF35. Education Officer, UNICEF36. C4D Deputy Officer, UNICEF37. Principal Assistant Secretary, Ministry of Women, Family and Community Development


79

ANNEX E: DOCUMENT ANALYSIS MAP

The following table is intended to provide an overview of the documents UNICEF shared with the Evaluation Team for the desk review. The table is designed to reflect the structure of the file sharing system, to help track the documents included to date. Folders names are bolded.

Document Name Date Finalized

Owner Description Date Added

00 Essential Documents01 Documentations#ThisAbility MakeathonJudging Kit Judging Criteria and

Overview of team invention on creating solution for their everyday challenges

Makeathon Info Sheet Aina July,2017 UNICEF Malaysia

Aina’s Team made a wheel organizer and back pouch

20 Nov 2018

Makeathon Info Sheet Amir July,2017 UNICEF Malaysia

Amir’s team made a series of communication boards

20 Nov 2018

Makeathon Info Sheet Braden

July,2017 UNICEF Malaysia

Braden’s team developed a grip- assistive device

20 Nov 2018

Makeathon Info Sheet Ethan July,2017 UNICEF Malaysia

Ethan’s team developed an activity pack

20 Nov 2018

Makeathon Info Sheet Janna July,2017 UNICEF Malaysia

Janna’s team made a multi-purpose rear mounting frame for wheelchair

20 Nov 2018

Makeathon Info Sheet Junsen


Junsen’s team made a prosthesis and belt holder

20 Nov 2018

Makeathon Info Sheet Nikhil July,2017 UNICEF Malaysia

Nikhil’s team made camera/iPad remote control assistive device

20 Nov 2018

Winner-Branden Lim Magic HolderMakeathon Funds Release Letter

May,2018 UNICEF Malaysia

Fund release letter to the winners of the #ThisAbility Makeathon 2017

21 Nov 2018

Braden Lim July,2017 UNICEF Malaysia

Image of the winner with his invention tool

21 Nov 2018

FB_Image1 July,2017 UNICEF Malaysia

Closer Image of Winner with his invention tool

21 Nov 2018

Institutional Contract Evaluation Form Kaki DIY


Overview on performance evaluation institution contractor 2017

21 Nov 2018

Magic Holder July,2017 UNICEF Malaysia

Image of winner’s invention – a gripping assistive device

21 Nov 2018


80

Makeathon Device Plan_V2 July,2017 UNICEF Malaysia

Overview Excel spreadsheet on device plan

21 Nov 2018

Re_UNICEF Makeathon MagicHolder


21 Nov 2018

Winner Image 1 July,2017 UNICEF Malaysia

Image of Winner- Braden Lim

21 Nov 2018


Image of Braden Lim with his family

21 Nov 2018


Image of Braden 21 Nov 2018


Image with Braden with his brother

21 Nov 2018


Image of Braden with volunteers from PetroSains

21 Nov 2018


Image of tools used to make the devise by Braden’s team

21 Nov 2018


Image on one of the members from technical support

21 Nov 2018


Image of a representative from technical support team

21 Nov 2018



21 Nov 2018



21 Nov 2018


Image of Braden with physicians

21 Nov 2018


Image of Braden painting 21 Nov 2018


Image of Braden presenting his painting

21 Nov 2018

Expert Onboarding Kit July,2017 UNICEF Malaysia

An overview handbook listing team and programme information on #ThisAbility Makeathon event

21 Nov 2018

Makeathon Compilation of Participants Ideas During the Making- Part 1


PowerPoint slides on Makeathon

21 Nov 2018

Makeathon Overview_CP Innovation


Overview recommendations to support Children with Disabilities to develop innovative solutions to the challenges they face

21 Nov 2018


81

Makeathon User Handbook July, 2017 UNICEF Malaysia

Overview introduction report of the event

21 Nov 2018

Participant Kit Agenda + Map + Team


General guideline to the participants, volunteers and partners on the event

21 Nov 2018

NGO Impact Academy

Challenges Young Inclusive Journalist Project (CYIJP)


General introduction essay to the event

20 Nov 2018

NGO Impact Academy_Overview_CP Innovation

April,2017 UNICEF Malaysia

A list of recommendations by CP Innovation ThisAbility programs

20 Nov 2018

Parent Manual - 25218 July,2017 Inclusive Space Services

An overview proposal of a community project called “The Inclusive Outdoor Classroom”

20 Nov 2018

Programme Brief_NGO Impact

July,2017 Tandemic A programme brief on NGO Impact Academy

20 Nov 2018

ThisAbility Flagship Monitoring


ThisAbility Flagship Monitoring Framework of 2017-2020

15 Nov 2018

DET Only Flyer for UNICEF 2017

2017 UNICEF Malaysia

Event flyer of DET programme under Peter Tan Training

14 Nov 2018

Peter Tan – UNICEF DET 2017


PowerPoint Slides layout of DET for UNICEF Malaysia

14 Nov 2018

ThisAbility Activities Matrix 2019-2019

Nov,2018 UNICEF Malaysia

A list of conceptual notes and resource materials of all the ThisAbility activities

22 Nov 2018

Worksheet DET UNICEF 2017


Overview worksheet of DET UNICEF on attitudes and awareness

14 Nov 2018

02 Meeting Minutes_thisabilityflagshipmtg_minutes-040718


Minutes of ThisAbility Taskforce Meeting follow-up

20 Nov 2018

_thisabilityflagshipmtg_minutes-230518

May,2018 UNICEF Malaysia

Minutes of ThisAbility Taskforce Meeting

20 Nov 2018


June,2017 UNICEF Malaysia

Minutes of ThisAbility internal meeting

20 Nov 2018


Sept,2017 UNICEF Malaysia

Minutes of ThisAbility internal taskforce meeting

20 Nov 2018


Oct,2017 UNICEF Malaysia

Minutes of ThisAbility internal taskforce meeting

20 Nov 2018

01. Evaluation Designs and Stakeholders

1. Terms of Reference


82

- Annex C ToR Evaluation November, 2018

UNICEF Malaysia

An overview of Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative in Malaysia

2 November 2018

2. Technical Proposal- Technical Proposal_

LRPS20189139940June, 2018 Equal

InternationalAn overview of Procurement of consultancy services to undertake a formative evaluation of UNICEF’s #ThisAbility Flagship Initiative

1 November 2018

- UNICEF Malaysia – IPSOS CVS

November, 2018

EqualInternational

A list of CVs of IPSOS members

1 November 2018

3. Evaluation Design- 2019 Official UN

HolidaysNovember, 2018

UNICEF Malaysia

A list Public Holidays in UN Malaysia

13 November 2018

- 2018 Official UN Holidays Malaysia

November, 2017

UNICEF Malaysia

A list of Public Holidays in Malaysia

16 November 2017

- Key Informant-Contact List

October, 2018 UNICEF Malaysia

Spreadsheet list of Informant Contact List

31 October 2018

- 2018-10-25-Agenda Meeting with Equal

November, 2018

UNICEF Malaysia

Meeting Agenda with Equal 1 November 2018

- 2018-11-02- Agenda Meeting with Equal

November, 2018

UNICEF Malaysia

Meeting Agenda with Equal 1 November 2018

- 2018-11-02 Key Informant Contact List

November, 2018

UNICEF Malaysia

Spreadsheet list of Informant Contact List

1 November 2018

- 2018-11-02 Members Mgmt-Reference Group

November, 2018

UNICEF Malaysia

A list of Members Management Reference Group

2 November 2018

4. Evaluation Guidelines and Standards

- 4.1 UNEGUNEG Code of Conduct March, 2008 UNEG Overview of Code of

Conduct for Evaluation in the UN system

27 June 2018

UNEG Ethical Guidelines for Evaluation

March, 2008 UNEG Overview on ethical guidelines for evaluation

27 June 2018

UNEG Human Rights and Gender Quality

March, 20111 UNEG Overview on Human Rights and Gender Equality guidance

27 June 2018

UNEG Norms and Standards 2

June, 2016 UNEG PowerPoint slides brief on norms and standards for evaluation

27 June 2018

UNEG Norms and Standards

June, 2016 UNEG Overview on norms and standards for evaluation

27 June 2018


83

- 4.2 UNICEFBrand Book UNICEF August, 2017 UNICEF Brand manual version

1.0 on brand strategy in compliance with the Global Regulatory Framework

27 June 2018

Gender Responsive Evaluation Handbook

2015 UN Women Highlighting evaluation process on gender equality, women’s empowerment and human rights

27 June 2018

GEROS Feedback Matrices UNICEF Feedback form on evaluation quality assurance tool with grades and explanations

15 October 2018

GEROS Final Report Template

UNICEF Feedback form on evaluation quality assurance tool with grades and explanations

14 October 2018

GEROS Handbook UNICEF Summary UNICEF staff handbook orient to GEROS tool and GEROS system more generally

27 June 2018

GEROS Inception Report Template

UNICEF Evaluation Inception Report assessment form

15 October 2018

How to Design Equity-Focused Evaluations

2011 UNICEF Guideline on strategic evaluation findings, innovative approaches and methodologies

27 June 2018

UNICEF Guidance on Ethics in Research

April, 2015 UNICEF Overview on UNICEF procedure for ethical standards in research, evaluation, data collection and analysis

27 June 2018

UNICEF Revised Evaluation Policy

June, 2018 UNICEF Outline the evaluation principles and procedures, developed accordance with Executive Board decision

16 July 2018

UNICEF Style Book August, 2017 UNICEF Guideline on writing style in consistent terminology

27 June2018

UNICEF – Adapted UNEG Evaluation Reports Standards

June, 2017 UNICEF Guideline on report quality assessment tool

27 June 2018

- 4.3 Other DocumentsDe UNICEF 01.18.18 January, 2018 USAID Presentation slides on

developmental evaluation overview

18 January 2018

Write-up LL De UNICEF Lesson learned in conducting developmental evaluation within UNICEF Myanmar

7 November 2016


84

02. Desk Review

1- UNICEF Documents- UNICEF Malaysia

Documents

o ThisAbility Flag-ship Documents

Essential Documentation

Documentations_thisability Flagship Moni-toring


Overview monitoring framework on new activities of 2017-2018 and 2018-2019 under thisability flagship

14 Nov 2018

DET Only Flyer for UNICEF 2017


Event flyer of DET programme under Peter Tan Training

14 Nov 2018

DET_UNICEF-LIBRARY_SAG_PRO


Inaccessible 14 Nov 2018

Peter Tan – UNICEF DET 2017


PowerPoint Slides layout of DET for UNICEF Malaysia

14 Nov 2018

Worksheet DET UNICEF 2017


Overview worksheet of DET UNICEF on attitudes and awareness

14 Nov 2018

CwD MY KAP Study Dissemination

UNICEF Overview of Children with Disabilities in Malaysia Knowledge, Attitudes and Practice Study Dissemina-tion Plan 2017-2019

10 November 2018

Disability KAP November, 2016

UNICEF PowerPoint slides on childhood disabilities in Malaysia

10 November 2018

Disability Newsletter 2018 June, 2018 UNICEF Overview on UNICEF Children with Disabilities Newsletter in Word version

10 November 2018

Final File Childhood Disability in Malaysia

September, 2017

UNICEF A study report of knowl-edge, attitudes and practic-es on childhood disabilities

10 November 2018

Its_About_Ability Final April, 2008 UNICEF An overview explanation of the Convention on the Rights of Persons with Disabilities

10 November 2018

SWCR2013_ENG_Lo_res24April

May, 2013 UNICEF Overview of the State of the World’s Children highlighting children with disability

10 November 2018


85

UNICEF Childhood Disability in Malaysia

September, 2017

UNICEF PowerPoint slides on childhood disabilities in Malaysia by Dr Amjad Rabi

10 November 2018

UNICEF Children_with_Disability

May, 2014 UNICEF Overview of mapping the policies, programme, interventions and stakeholders

10 November 2018

Useful Links for CwD Data or Resources

UNICEF List of useful links for CwD or Resources

10 November 2018

o Situation of Children in Malaysia

2017 NHMS Infographic April 2018 Ministry of Health Malaysia

Infographic booklet of key findings from the adolescent health and nutrition surveys

16 October 2018

2013-002 Profile of Children in Malaysia

November 2013

UNICEF Overview on implementation children’s rights with equality

20 April 2015

EIU Country Report Malaysia

December, 2015

EIU Overview on the forecast and highlights of Malaysia

17 December 2015

MDG 2015 Summary UNICEF Presentation slides on the summary report of Millennium Development Goals Report

17 February 2016

UNDP_MDG Report 2015 UN Malaysia

Millennium Development Goal Report on Malaysia

28 February 2016

UNICEF Children in Sabah 150507 Finalized

2014 UNICEF Situational Analysis of children in Sabah Handbook

31 May 2015

o Country Programme

Annual Management PlanAnnex 1- Table of Delegation of Old Functions

March, 2018 UNICEF Malaysia

A list of members of old functions from Country Office

11 October 2018

Annex 2- MCO Organogram 2016-2020


Organization chart of Malaysia Country Office 2016-2020

11 October 2018

Annex 3- Draft Revised Malaysia CO


A list of Focal Points in Country Office 2017

11 October 2018

Annex 3- Revised Malaysia CO Working Group / Team


Overview of Working Divisions in the Country Office

11 October 2018

Annex 4- 2018-03-16 PRIME 2018-2019

March 2018 UNICEF Malaysia

Overview on 2018-2019 PRIME

11 October 2018

Annex 6- 2018HACT Assurance


Overview of HACT Micro Assessment and Assurance Plan 2018

11 October 2018


86

Annex 8- BCP ICT DRR Plan


Overview of Business Continuity Plan of Country Office

11 October 2018

Annex 9- Consolidate MCO 2018


Excel version of Consolidate 2018

11 October 2018

Annual Management Plan 2018


Finalised overview of annual management plan in country office

11 October 2018

Country Programme Action Plan Annex01-UNICEFMALAYSIA-GOM-CPAP-IRRF

December, 2015

UNICEF Malaysia

Overview Table on Results and Resources Framework

07 December 2015

Annex02-UNICEFMALAYSIA-GOM-CPAP-IMEP

December, 2015

UNICEF Malaysia

Overview Table on Integrated Monitor and Evaluation Plan

07 December 2015

MCO-CPAP 2016-2020 Signed

December, 2015

UNIEF Malaysia

Overview Report on Country Programme Action Plan

23 March 2016

MCO CPAP 2016-2020 Standard Indicator template consolidated - Final 16June2016

December, 2015

UNIEF Malaysia

Word version of Result and Resources Framework with programme components

26 June 2016

UNICEF Malaysia-CPAP 2016-2020 Final Version

December, 2015

UNIEF Malaysia

Overview Report on Country Programme Action Plan -final draft

13 December 2015

Country Programme Document 2015-PL24-Malaysia-CPD-ODS

August, 2015 UNICEF Malaysia

Overview of country programme

13 December 2015

Country Programme Management FINAL CPMP 11052015 MLY

April 2015 UNICEF Malaysia

Overview of Country Programme Management Plan 2016-2020

10 May 2015

Malaysia- Costed Evaluation Plan

April, 2015 UNICEF Malaysia

Highlights the Costed Evaluation Plan – Malaysia UNICEF country programme of cooperation, 2016–2020

29 April, 2015

Malaysia Country Office April, 2015 UNICEF Malaysia

Organization chart of Malaysia country office

10 May 2015

UNICEF Malaysia Final CPD April, 2015 UNICEF Malaysia

Draft of country programme document

29 April 2015

Rolling Workplan


87

Final Consolidated UNICEF-Govt Malaysia 2016-2017

UNICEF Malaysia

Excel version of final Consolidated UNICEF-Govt Malaysia-Rolling Workplan Milestones

16 October 2018

UNICEF Rolling Workplan 2018 Final

UNICEF Malaysia

Work Plan for CPAP 2016-2020 for period 2018-2019

09 October 2018

UNICEF Rolling Workplan 2016-2017

UNICEF Malaysia

Overview of latest Excel version Updated 11 May 2016

16 October 2018

UNICEF Rolling Workplan 2016-2017 Version June

UNICEF Malaysia

Overview of latest Excel Updated 1 June 2016

16 October 2018

UNICEF Rolling Workplan Template

UNICEF Malaysia

Sample Work Plan for CPAP 2016-2020 for period 2018-2019

16 October 2018

Theory of ChangeAnnex Result Structure – 27 April

UNICEF Malaysia

Overview of Result Framework Visual for period 2016-2020

3 May 2017

Result Structure Visual 11 March

UNICEF Malaysia

Overview of Result Framework Visual for period 2016-2020- Draft

14 March 2017

Results Matrix CPD 2016-2020

UNICEF Malaysia

Excel version of results and resources framework

3 May 2017

TOC Narrative UNICEF Malaysia

May, 2017 UNICEF Malaysia

UNICEF Malaysia country programme report theory of change 2016-2020

3 May 2017

Monitoring DataAnnual ReportsMalaysia COAR 2016 UNICEF

Malaysia2016 Annual Report of UNICEF Malaysia

12 October 2018

Malaysia COAR 2017 UNICEF Malaysia

2017 Annual Report of UNICEF Malaysia

12 October 2018

RAMOutcomes-Outputs and Indicator Status by Regional / Business Area


General overview reported 2018

12 October 2018

Output Performance and Fi-nancial Summary by Results Area


General overview reported 2018

12 October 2018

OrganogramMCO CPAP 2016-2020 (Current)


Latest organization chart of Malaysia Country Office 2016-2020

11 October 2018

Research and Evaluation Documents2013-001 Children with Disability

June, 2013 UNICEF Overview report on mapping the policies, programmes, interventions and stakeholders

11 October 2018


88

2013-002 Profile of Children in Malaysia

November 2013

UNICEF Overview on implementation children’s rights with equality

11 October 2018

Anthrologica Malaysia Disability

UNICEF Overview study of knowledge, attitudes and practise of children with disability in Malaysia

11 October 2018

Child Marriage in Malaysia Final

June 2017 UKM A final study report of child marriage in Malaysia

11 October 2018

Children Without Version-English

February, 2018

DM Analytics & UNICEF

Overview study on urban child poverty and deprivation in low cost flats in Kuala Lumpur

11 October 2018

EA Malaysia Final Report 31 Aug 2016

August, 2016 UNICEF Overview on Evaluability Assessment UNICEF Malaysia Country Programme 2016-2020

11 October 2018

EMR Equity Evaluation Malaysia

June 2015 UNICEF Overview on evaluation management response document highlighting equity focus in country programme

11 October 2018

EMR Document 2016-034 June 2017 UNICEF Overview on evaluation management response document highlighting capacities development initiated by UNICEF Malaysia

11 October 2018

Evaluation of Capacity Development

October, 2016 IDEA & UNICEF

Overview report on evaluation of capacity development initiated by UNICEF Malaysia 2011-2015

13 October 2018

Final Report on CRC Reservation

March, 2017 UNICEF Overview report on the study lifting Malaysia’s remaining reservations to the Convention on the Rights of The Child (CRC)

13 October 2018

Mapping Alternative Learning in Malaysia

November, 2015

UNICEF Overview on mapping alternative learning approaches, programmes, and stakeholders in Malaysia

12 October 2018

UNICEF Malaysia Report 2017

August 2017 UNICEF Overview on Child Deprivation

12 October 2018


89

UNICEF Management Response Plan 2016

2016 UNICEF 6 August 2018

UNICEF Digital Landscape in Malaysia

November 2014

UNICEF Overview report on access and use of digital technologies by children and adolescents

12 October 2018

UNICEF Children in Sabah 2015

2015 UNICEF State of Planning Unit Sabah & UNICEF Situation Analysis handbook on children in Sabah

12 October 2018

UNICEF Profile of Children in Sarawak 2017

2017 UNICEF Overview report on status children in Sarawak

12 October 2018

- UNICEF EAERPRO Documents

o Global Guidance Documents

2013-Synergies Paper V6 WebRevised

2013 UNICEF Indonesia

Overview Analysis Report on Using Human Rights Framework to promote the rights of children with disability

13 October 2018

2013- General Suggestions for Disability

2013 UNICEF UNICEF Guideline for Disability Situation Analysis

13 October 2018

2016 COAR Disability Summary Analysis

May, 2017 UNICEF Overview summary analysis of 128 Country Offices Annual Reports globally on Children with Disability

10 October 2018

2017 COAR Disability Full Summary Final

May 2018 UNICEF Overview summary analysis of 128 Country Offices Annual Reports globally on Children with Disability - 2017

10 October 2018

EARPRO Disability SiTan November, 2014

UNICEF PowerPoint slides on desk review findings on the situation of children and young people with disability in East Asia and the Pacific regions

10 October 2018

EN_Annex A_Intervention by Focus Area

UNICEF Excel List of possible actions / interventions to Children Disability

10 October 2018

EN_ Annex B_List of Resources and Websites

UNICEF A list of resources and websites related to children disability

10 October 2018

EN_X-Sectoral Programme Guidance Disability Final

April, 2007 UNICEF Overview report on ending discrimination and promoting participation, development and inclusion

10 October 2018


90

Malaysia Action Plan for CwD

UNICEF PowerPoint slides on the action plan for Children with Disability in Malaysia

10 October 2018

Proposal UNICEF-DFAT Partnership

May, 2015 UNICEF Partnership proposal to accelerate responses to the rights of children with disabilities, focusing East Asia and Pacific region

10 October 2018

Report of CwD Consultation December, 2017

UNICEF Overview report on East Asia-Pacific regional consultation to the needs of children with disabilities

10 October 2018

2018-EAP Regional Headlines Standard Presentation Latest

2018 UNICEF PowerPoint slides on drafted regional headlines

28 Nov 2018

2018-Growing Up Urban 2018 UNICEF Overview report of conversation with Mayors on making cities safe and sustainable for children

28 Nov 2018

2018- UNICEF Programme Guidance for Second Decade

2018 UNICEF Overview guidance report 28 Nov 2018

2018- UNICEF EAPRO Headline 1

2018 UNICEF Overview report contains description of each of the 18 results areas and key programme strategies on early moments of children

28 Nov 2018

2018-UNICEF EAPRO Headline 2

2018 UNICEF Overview report contains description of each of the 18 results areas and key programme strategies on children grow in a sustainable and safe environment

28 Nov 2018

2018-UNICEF EAPRO Headline3

2018 UNICEF Overview report contains description of each of the 18 results areas and key programme strategies on adolescent potential unleashed

28 Nov 2018

- UNICEF Headquarters Documents

o Strategic Plan 2017-17-Rev1 Strategic Plan 2018-2021

August, 2017 UNICEF Overview report on UNICEF strategic plan 2018-2021

14 January 2018

2017-18-Final Result Framework

July, 2017 UNICEF Overview report on final result framework of the UNICEF Strategic Plan 2018-2021

14 January 2018


91

2017-Annex Common Chapter

July, 2017 UNICEF Annex of the common chapter implementing 2030 Agenda with UNDP, UNFPA, UNICEF, and UN-Women strategic plans for 2018-2021

14 January 2018

2017-EB11-Theory of Change

July, 2017 UNICEF Overview paper of theory of change

14 January 2018

Presentation-SP for 22 Aug Informal Briefing

August, 2017 UNICEF Informal brief on the strategic plan 2018-2021

14 January 2018

SP Brochure 2018-2 English January 2018 UNICEF New York

Overview report of UNICEF Strategic Plan 2018-2021

25 February 2018

UNICEF_Strategic Plan October, 2017 UNICEF Orientation report on UNICEF Strategic Plan

14 January 2018

2013-Take Us Seriously! June, 2013 UNICEF New York

Summary Report of Engaging Children with Disability in Decisions Affecting Their Lives

13 Nov 2018

2018-UNICEF Programme Guidance for The Second Decade

October, 2018 UNICEF UNICEF General Guidance Book on Programming with and for adolescents

13 Nov 2018

2- Government Documents

Eleventh Malaysia Plan 2016-2020

May, 2015 EPU Overview report on the milestones of Malaysia’s history and polices

2 February 2016

Malaysian Vision 2020 EPU Brief on the nation objective by Prime Minister, Tun Dr Mahathir

16 Dec 2015

Tenth Malaysia Plan 2011-2015

2010 EPU Overview report on the milestones of Malaysia’s history and polices

21 January 2016

3- UN DocumentsDraft 2 UNSPFR Agencies Comment

October, 2015 UNICEF Outline document of UN Strategic Partnership Framework on “Government of Malaysia and the UN Country Team”

13 Dec 2015

UN Report on Contributing Po

January, 2015 UN Malaysia

Overview report on “Continuing Inclusive Dialogues On Implementation of The Post 2015 Development Agenda- Malaysia

28 February 2016

UNSPF Outline Document February, 2014

UN Malaysia

Draft on UNSPF 2016-2020 (for discussion purpose only)

6 February 2015


92

ANNEX F: THEORY OF CHANGEEvaluation findings suggest that the strategies used to shape the trajectory of the #ThisAbility Flagship Initiative are not clearly defined; resulting in a Flagship that appears more reactive and opportunistic than strategic. The Terms of Reference for this evaluation called for the reconstruction of the Theory of Change behind the #ThisAbility Flagship Initiative; this was not possible given the apparent nature of Flagship activities. An externally focused forward-looking Theory of Change has been produced instead (in agreement with UNICEF) that is intended to provide strategic direction and an overall goal for UNICEF Malaysia’s future disability work (see below).

This Theory of Change has been informed by a barrier analysis conducted with selected external partners, developed in consultation with UNICEF staff and sharpened using field data collected during the evaluation. The resulting Theory of Change reflects the current status of disability inclusion in Malaysia and the steps considered necessary by informants to effect positive change. It is important to note that the Theory of Change is very ambitious. UNICEF Malaysia’s convening power and strong profile provides a firm basis upon which the Programme can build to deliver the Theory of Change, but as this evaluation report highlights UNICEF Malaysia requires a significant investment in its capacity around disability inclusion. Not least, the Theory of Change requires UNICEF Malaysia to shift the balance of its disability inclusion activities (as suggested by the evaluated #ThisAbility Flagship Initiative activities) from one-off activities to a more strategic and concerted approach to changing public attitudes and policy.


94

Box B: Accessible summary of Theory of Change

The Theory of Change (above) assumes that if UNICEF Malaysia i) mobilizes and amplifies networks and partnerships, ii) generates and disseminates evidence to inform action and iii) strengthens selected systems and/ or builds capacity, the following short-term change will be achieved: • Reduced stigma and discrimination toward children with disabilities at household-

and community-level.• Increased coverage of disability-inclusive models that enable all children to learn and

play together. • Increased use of disability data and disability-inclusive practices. • Enabling environment strengthened regarding inclusion of children with disabilities.

These short-term changes are expected to in turn achieve: • Increased equality between children with and without disabilities in access to services

and the fulfillment of their rights, leading to equality in full.• Improved attitudes and behaviours towards children with disabilities as regards

inclusion. • A renewed, coherent and coordinated national approach to disability inclusion

established that is centered on equality. The final impact of these changes is expected to be:

All children with disabilities in Malaysia access high-quality services and enjoy the fulfillment of their rights on an equal basis with children without disabilities.

The impact sought in the Theory of Change is for all children with disabilities in Malaysia to access high-quality services and enjoy the fulfillment of their rights on an equal basis with children without disabilities. The focus on equality is deliberate; it aligns with language adopted by the Regional Office and UNICEF’s Disability Unit in New York as well as the CRPD and the SDGs. Underlying the chosen wording is also an emphasis on the quality of services for and the delivery of rights to all children; disability inclusion is not at the expense of other marginalized groups and does not preclude investment in better child-friendly services or activities. The intended impact subsequently requires not just a focus on inequalities between children with and without disabilities; it also requires the bar to be lifted for all children through inclusive and accessible strategies and/ or models.Three medium-term outcomes are set in the Theory of Change as stepping-stones towards achieving the intended impact:

• Increased equality between children with and without disabilities in access to services and the fulfillment of their rights, leading to equality in full.

• Improved attitudes and behaviours towards children with disabilities as regards inclusion. • A renewed, coherent and coordinated national approach to disability inclusion established

that is centered on equality.


95

While it is not explicitly included in the Theory of Change, the medium-term outcomes are intended to be delivered by 2025. The three medium-term outcomes are intended to feed directly into the intended impact. Tangibly increasing equality will require proactive steps to adapt existing services or activities to include children with disabilities as well as designing and/ or replicating disability-specific interventions. It is important to remember that implicit in this is the need to improve quality for all children, not to seek equality in services or the fulfillment of rights that fall short of Malaysia’s obligations.

Positive attitudes are a key foundation of equality; children with disabilities will not utilize available services or enjoy their rights if attitudes to disability inclusion remain poor. UNICEF Malaysia’s KAP Survey provides a valuable baseline that can be repeated in 2025 to measure success. Delivering change that goes beyond UNICEF’s interventions to nationwide success and sustaining it will require clear commitment from the Government and an approach supported by all stakeholders. UNICEF’s convening power will be especially critical to achieving this last medium-term outcome as well as strong, ongoing collaboration with external partners to influence Government and to foster ownership across key stakeholders.

Four short-term outcomes are included in the Theory of Change that will be indicators of success toward achieving the medium-term outcomes:

• Reduced stigma and discrimination towards children with disabilities at household and community level.

• Increased coverage of disability-inclusive models that enable all children to learn and play together.

• Increased use of disability data and disability-inclusive practice. • Enabling environment strengthened regarding inclusion of children with disabilities.

Reducing stigma and discrimination will require changes to underlying attitudes and building awareness of disability. Disability awareness training and a countrywide behavior change campaign to transform negative social norms are essential; the behavior change campaign would ideally be a distinct part of a broader advocacy strategy. Examples of campaigns that seek to change social norms regarding marginalized groups are available from the HIV and other sectors. A campaign delivered by M&C Saatchi, for example, successfully reduced stigma and discrimination as a key barrier to HIV service access through the development and dissemination of messages co-created with LGBT and sex worker organizations. The M&C Saatchi campaign worked to humanise key HIV populations, generate positive narratives and role model a changed social norm. Adapting existing methodologies for disability inclusion will require some innovation from UNICEF Malaysia but success will resonate beyond Malaysia. Evidence of proven disability-inclusive models is sparse meaning that UNICEF will need to not only secure a broad commitment to disability inclusion but will also have to innovate and document results to show what works and can be replicated.

Measuring and tracking change will require the routine collection of disaggregated disability data. Critically, as demonstrated in other countries, it is also necessary to ensure decision-makers and key advocates (e.g. Disabled People’s Organizations) can use the data and evidence collected to inform and design effective policies and/ or services. Furthermore, a coordinated national approach will only work with political and financial buy-in (specifically CRPD compliant budgeting) from the Government.

Multiple strategies (or inputs) are identified in the Theory of Change that are required to deliver these outcomes. Strategies are grouped under three sub-headings:

• Mobilizing and amplifying networks and partnerships• Generating and disseminating evidence to inform action• System strengthening and capacity building

It is important to underline that it is not UNICEF Malaysia’s lone responsibility to deliver this Theory of Change; UNICEF Malaysia will require strategic and reliable partners. The first of the three sub-headings focus on galvanizing support for this Theory of Change outside of UNICEF Malaysia and mobilizing action. UNICEF Malaysia will need to be the strong lead for strategies under this first sub-heading, successful delivery of these strategies will have a pivotal impact on the successful delivery of other strategies. UNICEF Malaysia will have a key role to play in the strategies grouped under the other two sub-headings, but it is expected that external partners will also play a more significant role.

UNICEF Malaysia’s Sections have already started to identify how they will contribute to the strategies identified. The Nutrition Section recognizes the need to build the capacity of health service providers and health agencies to include people with disabilities. The Education Section recognizes the need to include persons with disabilities in national education polices and to develop an Inclusive Education roadmap. The Innovation Section identify the importance of designing and testing new inclusive tools and models, the Social Policy Section are capable of supporting capacity building and working to mainstream disability inclusion across Malaysia’s statistical systems, policy processes and legislation, while C4D has underscored its role in changing and tracking attitudes to disability inclusion. It will be important for each Section to more sharply define their specific inputs and to revise workplans annually.

Finally, the Theory of Change identifies a series of assumptions that may affect the delivery. UNICEF Malaysia will need to review and revise these assumptions as work to achieve the outcomes and the final impact gets underway. At this early stage, perhaps the most critical assumption is that UNICEF Malaysia will sufficiently prioritize disability inclusion in the CPD process to adequately resource delivery against this Theory of Change. UNICEF Malaysia is strongly encouraged to review and recalibrate the Theory of Change after the CPD process.

For further information, please contact: UNICEF MalaysiaUnited Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 PutrajayaEmail: [email protected] © UNICEF Malaysia, November 2019