Assessing the Impact of Policy on Practice:

Using Organisational Case Studies to Assess the Implementation of Mental Health Nursing Policy

Julie Repper

Today ...• To consider the value of organisational case studies as a means of

evaluating the implementation and impact of mental health policy

• To draw on our experience of evaluating the impact of the CNO’s review of mental health nursing: From Values to Action (DoH 2006)1

• To reflect on the involvement of service users and carers in conducting the research and in assessing the impact of policy

1(see full report at http://www.nottingham.ac.uk/nursing/cno-review)

Shift in Mental Health Policy

• From requiring action to develop services with implementation guidance and measurable fidelity criteria (eg NSF for MH, 1999 and associated ‘PIG’s)

• Towards values that underpin practice within those services (New Horizons - equality & justice, reaching potential, being in control of our lives, valuing relationships)

• The implementations of such values is much more difficult to assess, attribution is hard to ascertain and measurement of policy impact is even more challenging!

From Values to Action (DoH, 2006a)

• 15 years since last Review and much has changed in the healthcare environment.

• Review of Mental Health Nursing commissioned by CNO to assess the profession’s fit in the rapidly changing healthcare environment.

• Makes 17 recommendations for current and future practice and education.• Supported by good practice guidance for education and a ‘self-assessment

tool-kit’ for Trusts.

• Evaluation of implementation was an integral requirement of the report and funding for this was announced in 2007 and undertaken 2007-9.

Recommendations in ReviewPutting values into practice

• Recovery to underpin all service structures, individual practice and educational preparation.

• All nurses to base practice on evidence that will improve outcomes

• MHNs will promote equitable care for all groups and individuals.

Improving outcomes for service users

• Meeting greatest need• Developing

therapeutic relationships

• Physical well being• Psychological

therapies• Social Inclusion• Substance misuse• Inpatient care• Holism and risk

assessments• Spiritual need• Impact on carers

A positive, modern profession

•Recruitment and retention•Developing new roles•Supporting CPD•Awareness of review and its impact•Nurse pre-registration education will prepare MHNs to provide effective and values-based care.

Aims of StudyTo evaluate implementation and impact of CNO review

Objectives

• To establish progress and strategies for implementing recommendations

• To identify facilitators and barriers to implementation

• To examine impact of implementation on user/carer experiences

• To explore the relationship between organisational ownership, implementation progress and impact

• To highlight areas of good practice and positive outcomes including effective strategies used to facilitate implementation     

Design

Phase 1: Surveyof Trusts and HEIs

Phase 2: Case studies of6 Trusts

Phase 3: Follow-up survey ofTrusts & HEIs

Case Study“... A strategy (not a method) for doing research which involves an empirical

investigation of a particular contemporary phenomenon within its real life context.” (Robson, 1993)

• Allows complex issues to be examined in detail (Platt, 1992)

• Retains holistic characteristics of real life events (Stake, 1995)

• More variables than data points (Yin, 1994)

• Multiple sources of evidence (Robson, 1993)

• Data collection and analysis guided by theoretical propositions (Yin, 1994)

Organisational Case Studies

In-depth instrumental case studies (Stake 1995) of organisations to:• Understand diverse approaches and processes from different perspectives

and levels • Identify key components of successful (an unsuccessful) processes and

strategies • Suggest wider organisational features that support such processes• Generate theory by identifying key features within individual cases and

building on this through cross case analysis (Eisenhardt, 2002 )

Case studies do not promote generalisability per se (Sandelowski, 1988), but they do:

• generate insights which can be ‘transferred’ or ‘recontextualised’ (Morse 1994) by testing their application in diverse settings

• therefore purposively selected distinctively different Trusts (high and low implementers; urban/rural; north, south & London; PCT/MH & Partnership).

Data Collected• Interviews with most appropriate personnel working at various levels

within the organisation from Executive Directors to team leaders and front line workers (snowballing from Lead Nurse) - topics: response to review, strategy for implementation, monitoring progress, barriers & levers.

• Policy and procedure documents and other relevant sources of documentary evidence identified during interviews (eg records of meetings, audits, and reports)

• Interviews with staff working in related organisations (voluntary and/or private sector, education)

• Interviews and/or focus groups with people using the service and with family members (‘carers’).

Process

• Each Trust visited by a team of four researchers for several days to gain a clear picture of implementation at that site.

• Framework Approach used to guide analysis - starting with a meeting following each Trust visit to identify key themes and areas for follow up on subsequent visits and/or in subsequent case studies (familiarisiation).

• Recorded interviews and commentaries were transcribed and sent to all researchers who met to compile a coding index that was systematically used with each site data to produce a chart of relevant evidence for each Trust and develop site reports.

• Cross case analysis began to identify over-arching themes, categories and patterns in relationships and events. All worked together on this to improve reliability and reduce bias in interpretation.

Service User and Carer Involvement

• In good position to judge impact of review on practice• Contributed to meaningful research design (questions asked, involvement of

external organisations etc)• Elicit more detailed and accurate responses from interviewees with shared

experiences ....

So

• Four people with personal experience on Advisory Group• All SU interviews (n=32) conducted by SU and carer interviews (n=28) by

carer (who had experience and training in research)• All staff interviews done with a service user/carer• All analysis done as a team• Writing up also done as a team.

Findings• No systematic differences between ‘high’ and ‘low implementers’.

• At executive level there was general acceptance of Review’s recommendations but very little evidence of action linked directly to Review (no dedicated strategy group in any Trusts):

“We were doing it all anyway. Maybe it put a bit of wind in the sail but it certainly did not build the boat”

“We decided to check out that we are doing what it says but not to make a big deal about the document”

“The Review box could be ticked because of work we were already doing on other policies”.

• The research objectives were therefore amended to add what are the factors

that influenced the implementation of the Review?

Factors influencing Response to the Review

• Involvement in CNO consultation

• Local ContextStructural changes ( all recent or imminent FT status)Crises demanding a response (eg SUIs, poor HCC review)Leadership (3 Trusts had Acting Lead Nurses, two new executive nurses)

• National ContextMost recommendations reflected other extant policy requirements (eg Recovery, Social Inclusion, equality, EBD, Psychological therapies)Nursing voice on Executive Boards felt to be relatively powerless

• Nature of the ReviewRecommendations are not part of accountability structures, not part of commissioning guidance, bring no linked funding, not worded as specific targets or requirements. Recommendations are not radical or challenging but reflect good practice (“motherhood and apple pie”)

Implementation of Recommendations in Practice (Practitioner interviews)

• Few respondents other than exec team were familiar with the Recommendations so interviews focussed on the action that was being taken in these areas (whether or not this could be attributed to the review)

• Most recognised the principles underpinning recommendations and could comment on progress made in their Trust (this was verified with evidence and compiled into matrices for each site).

• It was more difficult to assess progress with ‘Values’ implementation such as Recovery and Equality: respondents had differing ideas about what these meant and how they should be implemented. No evidence of damaging interpretations, a lot of enthusiasm, much name changing but very little action.

• Many would have liked to know more about the Review as they felt it would be a good guide to service development and improvement.

Impact on Service Users’ & Carers’ Experiences• Despite rhetoric of involvement, all Trusts found it difficult to identify service

users & carers for us to interview; none had an involvement strategy; move to FT status was felt to have sidelined the voices of service users in generic groups/councils etc.

• Most interviewees had not heard of review but most could cite examples of their involvement in small one-off initiatives and in interviews, induction, training, on PPI forum.

• Voluntary sector groups had not heard of Review and had not noticed any additional involvement or funding to support recommendations.

• When asked about changes in nursing services/service provision there was little perception of change other than increasing workloads and increasing levels of disturbance on acute wards.

• Limited understanding of Recovery

• Limited opportunities for social inclusion

Comparison of nurses with SU & Carers on barriers to implementation

Nurses Service Users and Carers

Recruitment Problems Recruitment Problems

Retention problems mean high turnover and use of bank staff

Retention hampered by health problems and carer responsibilities

Resources – limited time, training, policy implementation low priority compared with clients’ needs

Resources – limited time, training, no awareness of policy, current priority is FT status

Bureaucracy – increasing paperwork makes hands on care difficult

Bureaucracy – FT status feels as though it is co-opting users’ voices and reducing critical potential

Morale - low as unable to work as they wish due to low staffing, high acuity

Morale - low as involvement opportunities ever changing and rewards slow

Communication – not informed about policy developments

Communication – not informed about policies in Trust

Sense of powerlessness - no systems for front line workers to get voices heard

Sense of powerlessness – opportunities to get voice heard diminished with FT status. Too often people say yes but nothing changes

Some Conclusions• Organisational case studies do elicit an understanding of the process of

responding to and implementing policy but a three day visit is not sufficient to assess and attribute changes in culture

• Different data sources expose gulf between rhetoric and practice and explain reasons for this

• Different levels of interviewees demonstrate competing pressures in all areas and facilitate understanding of progress (or lack of) with implementation

• But a week long visit is not enough to meaningfully• Similarities and differences between cases test the validity of developing

theories• But the long distance between a policy recommendation and person receiving

care makes it difficult to attribute any aspect of that care to a particular policy• ...made more difficult when the policy is based on values rather than definable

action• ...and complicated if the policy does not say anything distinctive/different from

other policies. • Service users & carers’ voices are an essential part of the research – not only in

determining design but in keeping interviews on track (as interviewers) and in telling their stories (as interviewees).

So ... What about Recovery?

• Conceptual confusion (clinical vs personal vs individual )• No benchmarks for what ‘it’ looks like in practice• No approved Recovery training available• How can you assess the values underpinning a culture?

CMH recently published ‘Making Recovery a Reality: AMethodology for Organisational Change’ which is a project to run in 6 MH Trusts working towards meeting 10 ‘organisational Challenges. These have been developed to guide commissioners – a move away from standardised outcome measures towards transforming a culture:

SCMH Implementing Recovery (2009 )10 key organisational challenges

• Changing day-to-day interactions and the quality of experience • Delivering comprehensive, user-led education and training• Establishing a ‘Recovery Education Unit’ for peers to provide

education and support for staff and for service users• Ensuring organisational commitment, creating the ‘culture’ • Increasing ‘personalisation’ and choice • Changing the way we approach risk assessment – negotiating

agreed safety plans • Redefining user involvement into more equal partnership • Transforming the workforce – peer support workers• Supporting staff in their recovery journey • Increasing opportunities for building a life ‘beyond illness’.

Thank you

Julie.repper@nottingham.ac.uk