Impact Evaluation Report

Takarangi Competency Framework

September 2010

Matua Raki

National Addiction Workforce Development Centre

Ōtautahi

2

ACKNOWLEDGEMENTS:

This evaluation would not have been possible without the active participation and support of those who have engaged with the Takarangi Competency Framework.

A special thanks to Ana Mules who did much of the ‘leg-work’ to make this evaluation possible and Dianne Hendey who collated the returns and undertook the initial analysis of the qualitative data.

As ever the support of the Roopu Kaitiaki (Titari Eramiha, Nellie Rata, Moe Milne, Pam Armstrong, Te Puea Winiata and Paul Robertson) and the hard work of Tony and Jenny Scott of Klub Ngaru Ltd is appreciated and acknowledged.

Terry Huriwai

Senior Advisor

Matua Raki

3

TAKARANGI COMPETENCY

FRAMEWORK

He mihi

E piki ana taku pikitanga ki ngā maunga kōrero,

Ka tū, ka mihi, ka toro mātakitaki ake atu ki uta, ki tai, ki runga, ki raro,

Ki ngā tihi whakataratara e tū whakatahuri ai i te hau mātao i whea kē!

Ngā homaitanga i ngā kete o te wānanga,

i te tēpu kōrero o te wā iti nei,

Koutou tē kitea o naianei rangi, kā mihi!

E te tākohatanga i te puna ō! Mauri Ora ki a koutou!

Tātou te pātōtōtanga i te tatau ki apōpō

Maranga ake ra!

He kohinga kete, He kohinga wānanga, He kohinga moemoeā

Nei ra e takoto ana… E huri!

This carving is found on the Auckland Māori Mental Health Service whare

Manawanui. It depicts the Ngāti Whatua tupuna Kawharu and the Takarangi spirals on either side represent balance.

4

Impact Evaluation Report

Takarangi Competency Framework

Executive Summary:

The symbolism of the Takarangi Spiral reflects integration of past and future, integration of clinical and cultural components, on-going learning and continuous quality assurance. The ultimate aim of the Takarangi Competency Framework is to contribute to enhanced engagement and outcomes for Māori accessing addiction and mental health services. Participants in the Takarangi Competency Framework introductory workshops conducted during 2009 were followed up to gauge the level of impact on practice. This survey builds on evaluations carried out at the introductory workshops.

We received 94 individual responses and this 84% response rate is pleasingly high.

Individual responses to the Takarangi Competency Framework (TCF) were overwhelmingly positive. Engagement with the Framework appears to have been transformative for many and emancipatory for some. The lives and practice of individuals involved with the TCF have changed.

The TCF was constructed after a number of years of development utilising a range of previous work such as Tikanga Totika (Milne, 2001), Piringatahi (ALAC, 2005) as well as work undertaken by various professions such as social work, psychology and education. It specifies competencies for working with Māori and a method for practitioners to identify those currently held, those yet to be gained and a process to do so. The survey results indicated that participants felt that the TCF validated Māori identity, affirmed Māori processes and made irrefutable the logic of incorporating tikanga Māori into work undertaken with Māori in the addiction and mental health sectors.

Respondents commented on improvements to their:

• practice

• ability to work with clients and whānau

• ability to understand the needs of their clients and their whānau

• understanding of how to use Māori cultural procedures and processes

• confidence in delivering integrated practice

• team/service emphasis on achieving cultural competence and

• team/service ability to work with Māori clients.

This intervention (The TCF) appears to have raised awareness, changed confidence, triggered motivation, improved technical skills and lifted the competence (in measurable behavioural terms) of those involved.

5

While overall very positive, respondents also indicated that impact at an organisation/ service level AOD practitioners also included resistance and little interest. More worryingly they wrote about initial enthusiasm being followed by gradual decline as ‘erosion’ set in.

Even greater benefits might have occurred (and still may) if the roll-out of the TCF is able to take a stronger systems approach such as was utilised in the initial pilots. Matua Raki has a role in promoting the TCF at an organisational level, taking a whole team approach as well as individual skill enhancement and assertive follow-up.

The respondents identified the need for:

• greater organisation/service level buy-in and support

• Matua Raki maintaining its links and supports

• Other competency frameworks and standards to recognise the value of the Takarangi Competency Framework

• Matua Raki to develop training and supervision related to the competencies.

Recommendations:

Given the Takarangi Competency Framework appears to have a positive impact on the practice of those working with Māori clients and their whānau there would seem to be value in ongoing implementation of the framework by Matua Raki.

There is a need for further evaluation and research into the impacts and value of the framework.

There is a need to develop the TCF package beyond just the introductory workshops and look to ongoing competency based training and supervision.

There is a need to explore the use of on-line resources.

Ongoing work with regional workforce coordinators and regional training teams to pursue collaborative funding models is required.

6

Introduction:

The Takarangi Competency Framework has been developed to enhance the practice of those working with Māori in the addiction1 and mental health sector. It is a tool that practitioners, teams and or services can use to set standards of practice (a quality assurance framework) and professional development goals and then map progress. It assumes that incorporating Māori values, beliefs and practices adds value to the work of those in the addiction and mental health sector.

This evaluation is an initial examination of the impact of the Takarangi Competency Framework on those who have engaged with it during 2009. An evaluation of the introductory workshops has been completed and is attached as Appendix 1. During the course of this evaluation a review of the Framework has also been undertaken to ascertain its fit in Matua Raki and the wider competence work currently taking place. At the time of submitting this report that Review is not yet completed.

Implementation

A shortened history of the development and underpinnings of the Takarangi Competency Framework are outlined in previous reports and evaluation. The framework has provided Matua Raki with an opportunity to help shape Māori responsiveness developments in Lets Get Real (Te Pou), Careerforce, Addiction Competencies (DAPAANZ) and the Association of Counsellors2.

Current roll-out has involved two-day introductory workshops followed by support of those practitioners and services that engage with the framework. Workshops have over the past 18 months been delivered in Northland, Auckland, Bay of Plenty, Tai Rawhiti, Lakes, Waikato and Taranaki. The Midlands Mental Health and Addiction Workforce Development group have actively supported the TCF workshops in their region through coordination and resourcing. The aims of the workshops are to:

• Introduce people to the framework (the competencies and assessment system)

• Support integrative practice

• Support the development of a consistent set of Māori values, beliefs and practice in the addiction and mental health sector

Matua Raki identified in it’s 2009 implementation plan that an initial evaluation of impact was to occur and this report represents information gained from surveying participants.

1 Addiction is a generic term to denote alcohol and other drug (AOD) and problem gambling (PG) 2 A number of Māori practitioners are members of NZAC rather than DAPAANZ.

7

Methodology:

The initial plan was to send out a mail survey to all those who had engaged in the framework and supplement the survey results with some qualitative data gained from a series of focus groups.

It was decided that invites to undertake the survey via Survey Monkey i.e. online would be sent. Using attendance lists and registrations from the workshops conducted in 2009, a total of 112 invitations were sent. We became aware that a number of people had left services since undertaking the introductory workshop and we estimate there were about seven (7) in this category. We asked others to pass on the invitation.

The survey included some demographics and work profile information as well as an opportunity for survey participants to make written response to a number of questions. The narratives were analysed for themes.

In terms of the focus groups two were carried out. The first was carried out in December 2009 at Waitangi with a group from Hauora Whanui and the other in Te Tairawhiti in March 2010.

Discussion with a range of providers, a series of interviews recorded on disc (e.g. see You tube clips and Matua Raki website) as well as feedback from the focus groups also inform the qualitative element of this evaluation report.

We analysed all the data and then separated out those who identified as working in the AOD sector. The report is generally structured to describe the findings of the total group first and then the findings of those who specialise in AOD.

8

Results:

The following is a description of the demographics of those who responded to the evaluation survey as well as some of the quantitative information they provided. There were 94 responses and four of those who attended the Te Tairawhiti focus group indicated that they preferred to input via the focus group and so had not filled out a survey.

Gender:

This picture of gender represents a different picture than that presented in the NAC Phone Surveys of the AOD sector and reflects more of a split one would expect to see in the mental health or wider health sector.

Table 1: Gender of those who responded to the Takarangi Competency Framework evaluation survey.

When the responses of those respondents who identified as working in the AOD sector were analysed the gender split for Māori was more closely 50:50.

Age:

Age distribution appears consistent with the general picture of an aging workforce across health.

9

Inspection of the ages of those who identified as AOD practitioners showed at least a third of the sample were aged 36-45 years, half were 46-55 years and the rest over 56 years of age.

Where they worked:

The question relating to where people worked would, in retrospect have been better to split into two separate questions - type of organisation and service provided.

The majority of respondents worked in NGOs. A number of respondents’ ticked health and social services and in the other category described their services as iwi provider or Māori provider. Interestingly some of the health and social services (including iwi or Māori providers) had addiction and or mental health contracts.

Roles:

In terms of role the majority of respondents were either AOD practitioners (21%), nurses (18%) or support workers (18%). The ‘other’ category made up 23% of responses and includes roles such as kai awhina, kai manawa, Taurawhiri, cultural assessor, iwi support worker, ‘dual diagnosis’ clinician, administrator, manager and whānau coordinator.

Although many people were in their current roles for relatively short times (about 60% for three years or less), in terms of those who were in the AOD or mental health sector, over 60% had been in the sector for five years or more.

Taking a closer look at the AOD practitioner respondents 75% had been in their current position for 1-5 years but had worked in the AOD for over 10 years.

Highest Qualification:

In terms of qualifications the respondents who identified as AOD held Post Graduate Diplomas and degrees but the majority held undergraduate Certificates and or Diplomas.

10

Highest qualifications included PGDip Addiction and Mental Health, PGDip Health Science/AOD and degrees in counselling.

Membership of Professional Groups:

52% of respondents were in a professional body, the majority being members of DAPAANZ, NZNO, NZASW and NZCCA. In terms of the AOD respondents, 61% said they are members of a professional group and twelve people specified a professional body. Of concern were the number who indicated that they are not members of any professional group.

Although the question asks about membership it is not clear as to the status of that membership. For example whilst being a member of DAPAANZ means support for working under the DAPAANZ code of ethics etc. it gives not indication under which category of practitioner they are registered e.g. registered practitioner, support worker.

Are you a member of a professional group?

Yes

No

Three things you have learnt? The following table summarises the most consistent responses to the question what three things people felt they had learnt from the TCF. (NB: A number of the respondents only filled in two things).

1 2 3 Importance of a Māori Perspective in my practice

Māori practice connects us to our clients and their whānau better

The dichotomy of being bi-cultural Being cultural is clinical Strong self identity= strong practice

Gaps in my knowledge Pin points exactly what I need to know and how to get there

Shows the way

I know more than I thought Validation of existing competence

Reinforces existing skills

Self awareness Reflective practice Ability to monitor

Ask ‘is your workplace really kaupapa Māori?’

Māori Practice is transferrable across roles and sectors

Wide benefits Safer for tikanga practices Everyone can do it

11

Influence on Practice: Following on from the question about what respondents had learnt, the next question in the survey was in many ways a reconfirmation and asked how the TCF had influenced practice. There was an 85% response rate and no significant difference in replies between those who identified as AOD practitioners or others who filled out the survey. The following six points were identified most:

• Greatly enhanced confidence and competence

• Affirmed identify and practice

• Highlighted the balance between clinical and cultural practice

• Validated my values and practices that are intrinsically Māori

• Being deliberate Changes:

Bearing in mind that the respondents had had different lengths of time of exposure to the TCF, the evaluation was attempting to look at impact beyond the workshop (see earlier workshop evaluation report). There appears to have been a general acceptance that engagement with the Takarangi Competency Framework contributed to an enhanced ability to work with Māori and understanding of the needs of Māori clients.

12

Building on the above questions, respondents were asked more specific questions as to the size of impact engagement with the Takarangi Competency Framework had had.

Over 80% of the AOD respondents indicated a small or big improvement in their understanding of how to utilise Māori values, beliefs and practises and also in their confidence.

Impact on services:

The majority of responses (70%) to this question were positive with some comments talking about teams becoming more focused, other about drawing teams together and general validation of Māori staff.

There were however some less positive responses with comments indicating that in some teams and services there was no interest and in some services open hostility. Kōrero in the focus groups suggested that in some services the AOD (and or mental health) team is one of a number of teams and maintaining traction with other teams not exposed to the TCF was sometimes hard. For others, they indicated resistance from some was based on the perception that they would be judged while interestingly enough some suggested that some colleagues did not believe there was a place for ‘culture’ in a clinical service.

Whānau ora and who would benefit from engagement with the TCF?:

When asked if the Takarangi Competency Framework could contribute to whānau ora about 96% of the responses were positive. Likewise the majority felt that everyone (including non-Māori) would benefit from during the framework. Some were clearer only Māori and most included Managers as a ‘must do’ group.

13

Future Engagement?

Two of the last questions focused on where to and asked what did people need to keep engaged. Generally the responses revolved about three areas – Organisational support and change; External alignments and on-going personal and professional development:

� More wānanga

� More contact and information

� Cultural supervision and support

� More live examples and

� How-to do a port-folio information

� Develop and deliver training related to the competencies

� Greater support from our managers and leaders

� Financial recognition

� More organisational participation

� Link to NZQA and standards

� Links with Matua Raki

Further Comments?

Impressive structure and design Awesome tool Fills the gap in the Weltec and Blueprint courses He Taonga Great premise – needs more buy in and kudos Needs greater promotion Whole team training and more assessors needed Need framework to ‘fit’ administrative and management roles

Common themes to come from the narrative include:

� Greater awareness how Māori practices and values can be and are utilised in practice

� Greater confidence to utilise Māori practices and values

� More self reflection about practice

� Increased awareness of the whai ora and their whānau

� Validation of some current practices and beliefs

� Greater accountability for practice

Again there appears to be overwhelming support for the notion that engagement with the Takarangi Competency Framework has had some kind of impact personally and on teams.

14

Quotes from the narratives

“More precise in the notes I write and the mahi that I do so they correspond with the competencies I wish to achieve”

“More awareness of cultural issues and being able to ask for help, better understanding how to formulate questions regarding cultural issues”

“Aware of what I am doing. Being deliberate”

“Following the TCF will help me to become the best practitioner I can be for the people I work with”

“Participate in team reflections on cultural values”

“Made me focus on increasing my tikanga skill and helps me deliver my service differently cause I am trying to implement change and achieving all the time”

“Helped build team spirit and cohesiveness”

“Greater accountability”

“It has improved how we work with people and each other”

“Being Māori adds value to my practice”

15

45

47

16

Conclusion:

The purpose of the evaluation was to explore whether engagement with the Takarangi Competency Framework had any impact on practice and to ascertain what might be needed to support people engaged on the framework in the future.

Overall there appears to be a positive impact on practitioner confidence and awareness. In terms of the future, the responses reflect the needs of supporting those on the framework, further introductory workshops and training, developing more of a systems approach to promotion and encouragement of the Takarangi Competency Framework.