2014 health literacy evaluation report...
TRANSCRIPT
1
Evaluation of 2014 Health Literacy
Professional Development Initiatives
Final REPORT
Dr Lucio Naccarella, PhD
Health Systems & Workforce Unit
Centre for Health Policy, Melbourne School of Population & Global Health
The University of Melbourne
April, 2015
2
Acknowledgements
The evaluation was funded via a partnership between the Centre for Culture, Ethnicity & Health (CEH) in
collaboration with HealthWest Partnership (HWP) and cohealth. We particularly thank all the 2013 and 2014 Course
participants, executive sponsors, Community of Practice participants and Community of Practice Leadership Group
who participated in this evaluation as without their cooperation this work would not have been possible.
For research correspondence please contact: Lucio Naccarella, PhD [email protected]
3
Contents 1. Introduction ............................................................................................................................................................... 4
1.1. 2014 Health Literacy Development Course ...................................................................................................... 5
1.2 Health Literacy Community of Practice ............................................................................................................ 5
2. Evaluation .................................................................................................................................................................. 5
2.1 Evaluation foci ......................................................................................................................................................... 5
2.2 Evaluation approach ................................................................................................................................................ 6
2.3 Data Collection Methods ......................................................................................................................................... 7
3. Evaluation Findings .................................................................................................................................................. 10
3.1 2014 Health Literacy Course ........................................................................................................................... 10
3.1.1 Course participant profile ............................................................................................................................... 10
3.1.2 2014 Course Participant Pre-During and Post course survey ......................................................................... 10
3.1.3 Round One Interviews with 2014 Course participants ................................................................................... 11
3.1.4 Round Two Interviews with 2014 Course Participants ................................................................................... 13
3.1.5 2014 Organisation Executive Sponsors Interviews ......................................................................................... 19
3.1.6 2014 Health Literacy Course Facilitators Interviews ....................................................................................... 25
3.2 Health Literacy Community of Practice .......................................................................................................... 27
3.2.1 2014 Community of Practice Leadership Group ......................................................................................... 27
3.2.2 2014 Community of Practice Participants ................................................................................................... 29
3.2.3 2014 Course Participants CoP comments ................................................................................................... 31
3.2.4 Executive Sponsors CoP comments ............................................................................................................ 32
3.2.5 2014 Course Facilitators .............................................................................................................................. 33
3.3 The Ripple Effect in the West .......................................................................................................................... 33
3.4 Evaluation Participant Feedback Workshop ................................................................................................... 35
4. Synthesis and Discussion of Evaluation Findings ..................................................................................................... 36
4.1. The 2014 Health Literacy Training Course: ..................................................................................................... 36
4.2 2014 Health Literacy Community of Practice (CoP) ........................................................................................ 37
4.3 The Ripple effect in the West .......................................................................................................................... 38
5. Evaluation Rigor and Limitations ............................................................................................................................. 40
6. Key Emerging Propositions ...................................................................................................................................... 42
Appendices
Appendix 1: Evaluation data collection tools
Appendix 2: Logic Model
Appendix 3: 2013 Course Participants Stories of Change
Appendix 4: Pre-During –Post Course Participant Survey Findings
Appendix 5: Evaluation Participant Workshop – Draft Summary Report
4
1. Introduction
Health literacy is key to supporting people to better manage their own health and has the potential to improve health-related
outcomes of disadvantaged populations1,2
. According to Australian Bureau of Statistics, 59% of Australians are functionally
health illiterate, and an overwhelming majority (75%) are born in a non-English speaking country3. Low levels of individual
health literacy is associated with more use of health care services, poor knowledge and worse health outcomes4. Melbourne’s
Western region is one of the most culturally diverse communities in Victoria, increasing the risk of low health literacy levels,
which can impact on them accessing the health care system. Accordingly, health literacy is a key priority in the Better Health
Plan for the West5
Health professionals have been found to often lack adequate understanding of health literacy issues, and health literacy
practices are also not routinely used6,7
. In 2013, a collaboration developed between the Centre for Culture, Ethnicity & Health
(CEH), HealthWest Partnership (HWP) and cohealth (previously Western Region Health Centre) to develop the knowledge, skills
and organisational capacity towards health literacy of the health and community service sector in the western metropolitan
region of Melbourne.
In 2013, the CEH, HWP and WRHC developed and delivered a health literacy demonstration course. Ten HWP member agencies
from the western metropolitan region participated in the course, with two people per organisation attending (total of 20 course
participants. The University of Melbourne (Australian Health Workforce Institute, now Health Systems and Workforce Unit) was
commissioned to evaluate the adoption and implementation of the Health Literacy Demonstration Course. Overall the
evaluation revealed that the course had built capacity as demonstrated by course participants:
• Developing leadership in Health Literacy (i.e., inspire Health Literacy thinking and approaches within their organisations and
other networks)
• Building networks / partnerships among course participants and established an interdependent forum / Community of
Practice (CoP) for health literacy knowledge transfer and exchange;
• Developing health literacy workforce knowledge and skills;
• Developing ways to use and apply health literacy resources (tools, frameworks) and
• Serving as a catalyst for building organisational infrastructure (policies and procedures) to authorise and embed health
literacy into routine practice.
The 2013 evaluation also revealed the importance for course participants to have engagement from their senior executive
sponsors and creating a supportive enabling and authorising environment for course participants to implement learnings from
the course and to lead or facilitate changes in health literacy practices.
The 2013 evaluation proposed six propositions to optimise the investment in health literacy initiatives by the CEH, HWP and
WRHC in the western metropolitan region. These included: 1) revising the course to be more ‘HOW TO’ practically oriented; 2)
establishing an on-line Health Literacy Community of Practice Forum to share experiences, problems and learnings; 3) utilising
identified enablers and barriers within the course to build participant capability to adopt, implement and sustain health literacy
approaches; 4) contextualising future health literacy courses by revising the course selection processes to obtain indepth
participants information about roles, readiness and capability to initiate change; 5) continuing the course quarterly workshop
modular structure and exploring compressing course to run over a six month period; 6) investing in developing an evaluation
framework to assess the value of the 2013 health literacy course structure and curriculum at the workforce, organisation and
client level. See Centre for Ethnicity, Culture & Health http://www.ceh.org.au/downloads/2013HLDemoCourseEval.pdf for
further details.
1 Barrett, S. E., Puryear, J. S., & Westpheling, K. (2008). Health literacy practices in primary care settings: Examples from the field. The
Commonwealth Fund. Retrieved from http://tinyurl. com/n3pv9a3 2Sørensen, K; Van den Brouke, S; Fullam, J; Doyle, G; Pelikan, J et al (2012). Health literacy and public health: A systematic review and
integration of definitions and models. BMC Public Health; 12 (80). http://www.biomedcentral.com/content/pdf/1471-2458-12-80.pdf 3 Australian Bureau of Statistics,Health literacy. Canberra, Australia: Australian Bureau of Statistics, 2008.
http://www.ausstats.abs.gov.au/ausstats/subscriber.nsf/LookupAttach/4102.0Publication30.06.093/$File/41020_Healthliteracy.pdf
4 Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Viera A, Crotty K, et al. (2011). Health literacy interventions and outcomes: an updated
systematic review. Evidence Reports/Technology Assessments, No. 199. http://www.ahrq.gov/
5 http://djhs.com.au/fileadmin/filemount/pdf_files/BHPFTW_13_FINAL_JUNE_2012.pdf
6 Coleman, CA; Hudson, S; Maine, LL (2013). Health Literacy Practices and Educational Competencies for Health Professionals: A Consensus
Study. Journal of Health Communication, 18:82–102, 2013 7 Macabadco-O’Connell, A & Fry-Bowers EK (2011). Knowledge and perceptions of health literacy among nursing professionals. Journal of
Health Communication, 16, 295–307.National Patient Safety Foundation. (2010). Ask me 3. Retrieved from http://www.npsf.org/askme3
5
In 2014, in response to the increased commitment to health literacy as a priority area in the West (Better Health Plan for the
West) and the 2013 evaluation findings, the CEH, HWP and cohealth committed and invested in: delivering another health
literacy development course to HealthWest Partnership member agencies within the western metropolitan region of
Melbourne; introducing an Organisation Executive Sponsors workshop, establishing a health literacy community of practice; and
commissioning an evaluation of the 2014 initiatives.
Overall, the 2013 and 2014 health literacy initiatives and evaluations are designed to build upon each other and ultimately to
build health literacy capacity and the evidence base within the culturally diverse western metropolitan region of Melbourne.
1.1. 2014 Health Literacy Development Course
The 2014 Health Literacy Development Course was delivered by the Centre for Culture, Ethnicity & Health (CEH) in collaboration
with HealthWest Partnership (HWP) and cohealth to HWP member agencies. The course was designed to build the capacity of
agencies to respond to health literacy at both a client and an organisational level. Participant recruitment and nomination to
the course occurred through the HWP CEO sending an email invitation to its member agencies CEOs within the western region.
Ten agencies from the western metropolitan region participated in the course, with two people per organisation attending (total
of 20 course participants. Course participants were chosen by the organisation senior management. The course was comprised
of four face to face one day modules over an eight month period. See Centre for Culture, Ethnicity & Health
http://www.ceh.org.au/training/health-literacy-course for further details. Following module one and two, participants were
required to undertake a mini-project of approximately 40 hours within their organisation. The purpose of the projects was to
integrate health literacy practice across the organisation to embed learnings and create sustainable change. See
http://www.ceh.org.au/training/health-literacy-course for further details.
A key finding from the 2013 evaluation was the importance for course participants to have engagement from their senior
executive’s sponsors and creating a supportive enabling and authorising environment for course participants to implement
learnings from the course and to lead or facilitate changes in health literacy practices.
In response to the evaluation finding that engagement from senior executives was important for course participants to lead or
facilitate changes in health literacy practices, in 2014 HWP in collaboration with the CEH and cohealth, conducted an
Organisation Executive Sponsors Workshop. The workshop audience was the 2014 course participants and their senior
executive sponsors. The workshop was not a stand-alone initiative, but designed to supplement the course by: providing a
strategic view of health literacy in the West; workshopping the link between health literacy and accreditation standards; and
discussing the enablers and barriers to supporting organisational health literacy.
1.2 Health Literacy Community of Practice
In response to the 2013 evaluation finding that networks were being established amongst course participants and that
Community of Practice (CoP) would facilitate health literacy knowledge transfer and exchange, in 2014 a Health Literacy
Community of Practice (CoP) was established and led by HWP. The CoP was facilitated by a leadership group comprised of staff
from: Centre for Culture, Ethnicity & Health, HealthWest Partnership (HWP), cohealth, RDNS, ISIS Primary Care and two
consumer representatives from cohealth. The CoP leadership group defined the CoP as: a group of people who share an interest
in improving the capacity of health and community organisations to respond to the health literacy needs of consumers and
communities. The CoP was designed to provide opportunities to: share learnings, experiences, resources; problem solve;
exchange information and to network. The CoP was aimed at anyone championing change, including managers, clinicians,
Health Promotion staff, health literacy course participants, consumers, diversity staff, academics, researchers and peak body
representatives. The CoP met on three occasions (May 5th
, August 25th
and December 1st
, 2014). HealthWest Partnership (HWP)
http://www.healthwest.org.au/images/stories/healthwest/pdf/resources/early_intervention/about_the_hl_cop.pdf for further
details.
2. Evaluation
2.1 Evaluation foci
The University of Melbourne, Health Systems and Workforce Unit was commissioned to evaluate the extent to which the 2014
health literacy professional development initiatives were creating a ripple effect and building health literacy capability at the
individual, organisational, regional and systems level. The evaluation had three foci (2014 health literacy course; health literacy
CoP; the Ripple effect in the West), key and sub-evaluation questions.
6
1. 2014 Health Literacy Course
• To what extent has the 2014 Health Literacy course had an impact upon course participants and their organisation CEOs?
• What are the perceived impacts of the Course?
• What changes in individual participant health literacy competencies (specifically, health literacy practices) pre,
during and post course, have occurred.
• What changes in organisation health literacy attributes pre, during and post course, have occurred
• What contextual factors have influenced these changes?
2. Health Literacy Community of Practice
• What makes the community of practice work (or not), for whom and in what circumstances?
• What assumptions exist about how the CoP is supposed to work?
• To what extent has the CoP been implemented as intended, and what contextual factors may influenced its
implementation?
• What are the perceived impacts of the CoP?
• To what extent does the CoP build on and sustain learnings obtained from the Course?
• To what extent is the CoP sustainable, and what contextual factors may influence its sustainability?
3. The Ripple effect in the West
• What changes have occurred at the individual, organisational and systems level as a result of participating in the 2013
Health Literacy Course?
• What changes at the individual level (behaviour, roles, attitudes, skills, knowledge, relationships, capabilities) have
resulted from participating in the 2013 Course?
• What changes at the organisational level (culture, structures, governance, processes) have resulted from
participating in the 2013 Course
• What changes at the systems level (policies, information, regulations, education/training, key performance
indicators/Human Resources) have resulted from participating in the 2013 Course?
The ripple effect metaphor illustrated several dimensions. Firstly, the health literacy initiatives were conceptualised as a ‘Rock’
that would initially create a Splash (outputs at the individual and organisational level) leading to a ripple effect (immediate
outcomes – or changes at the individual and organisational level) beyond the course participants and their organisations, and
beyond the actual course - to have a longer term and multiplier effect in the West. Secondly, the health literacy initiatives (e.g.,
courses, CoP) were a ‘means not an endpoint’ and part of a ‘transformational change process not an event’ – facilitating the
adoption and implementation of health literacy concepts into practice, at an individual, organisational, and systems level.
2.2 Evaluation approach
It is recognised that the first task of any evaluator is to determine the context for the evaluation, and for funders and
stakeholders to understand that the success of the evaluation rests upon the optimal fit between the evaluation approach and
the initiative to be evaluated.
The choice of evaluation approach is driven by: the stage of development of the initiative, the context within which the initiative
and evaluation are occurring; and the evaluation foci and questions.
Initiative Stage of Development: Many evaluation approaches exist including: randomised controlled trials, developmental
evaluation, formative evaluation, summative evaluation, case study, participatory action learning, realistic evaluation,
appreciative enquiry etc. There is no right or right evaluation approach, but the approach that is the best fit for the stage of the
initiative. As a general rule - if the initiative is innovating and in its development - use a developmental evaluation approach; if
the initiative is forming and under refinement - use a formative evaluation approach, and if the initiative is stabilising and well
established - use a summative evaluation approach. Given that the health literacy initiatives are in the forming (e.g., senior
executive workshop, CoP) and refinement (e.g., course) phases, a formative evaluation approach was considered as most
appropriate.
Initiative and Evaluation Context: Given the dynamic reform and organisational context within which the health literacy
course, senior executives workshop and CoP are occurring, the evaluator needed to be aware of and understand the variable
implementation contexts, as such factors are central to why and how initiatives are successful. To understand the
implementation context, evaluators need to engage with the primary users of the evaluation and with the evaluation
participants. Close engagement enables the evaluation to elicit understandings and judgements about the initiatives, and to
7
define success – as the act of engaging shifts the criteria for assessing initiatives from a few to the many – this is critical as any
evaluation may be seen as value-laden, not neutral or objective. No matter how strong the evaluation claims are – values are
present in all evaluations through the decisions about whose interests, key evaluation questions and agendas need to be
addressed. Kushner (2012)8 calls for engagement to ensure greater inclusivity and equity in the evaluation process.
Several well established evaluation approaches exist to facilitate the engagement of the primary intended users with the
evaluation, including, participatory evaluation (Cousins, 1992) and utlisation-focussed evaluation (Paton, 2008)9. In practice this
means establishing a partnership between the evaluator and the evaluation primary users, to enable both parties to shape the
evaluation and executing the activities with the aim of enhancing evaluation use. A participatory approach requires the
evaluator to get up close and personal – building relationships with the primary intended users and evaluation participants –to
ensure they are immersed and sensitive to the complexities of the initiative.
To understand how and why the health literacy initiatives were creating a ripple effect a realist evaluation approach9
was also
used, because it strives to examine what works, for whom, and in what circumstances. Realist evaluation is a theory–driven
approach to understand how the outcomes (e.g., implementation of health literacy practices by participants) results from the
interplay between intervention mechanisms (e.g., course) and the context (individual, organisational, system) within which the
course learnings were to be implemented. To further assist in clarifying the 2014 health literacy initiatives and refining the
funded evaluation plan, questions and data collection methods, a program logic model10
was developed. Logic models provide a
visual representation about the assumptions about how a program is supposed to work, and the causal linkages between
context, inputs, activities, outputs, and outcomes (See Appendix 2).
The evaluation design also built upon traditional approaches to evaluating professional development initiatives, focusing on:
participant learning outcomes, intentions and confidence to use their newly-acquired knowledge and skills; participant use of
knowledge and skills gained; and participant perception of organisational support required to implement participant learning
outcomes11
, 12
.
Evaluation foci and questions: Given that the evaluation involved exploring, identifying, describing and explaining if and how
the health literacy initiatives were having an impact, a qualitative evaluation approach was the preferred strategy. My
qualitative evaluation approach is underpinned by an inductive interpretavist approach13
– where one sees people and their
interpretations, perceptions, meanings and understandings as the primary data source. Hence qualitative interviewing is my
preferred data collection method which is described in the next section.
2.3 Data Collection Methods
Given the evaluation foci and question, three key data collection methods were used: 1) semi-structured interviewing; 2) Most
significant change technique; and 3) Member validation.
1) Semi-structured interviewing: Qualitative semi-structured interviewing was chosen as the primary data collection method
for multiple:
• Interviewing aligned with the evaluators view that people’s (e.g., course participants) knowledge, views, understandings,
interpretations, and experiences are meaningful for constructing and explaining how and why the initiatives worked
• Interviewing enabled the evaluator to talk interactively with course, workshop and CoP participants, to ask them questions,
to listen to them, and to gain access to their experiences
• Given that the evaluator viewed knowledge and evidence as contextual, situational and interactional, interviewing enabled
me to ask contextualised questions about abut complex and largely social phenomena; and
• Interviewing enabled the evaluator to be active and reflexive in the process of data generation and interpretation.
The evaluator recognised that qualitative interviewing often poses challenges for the evaluator. The interactional aspects of
interviewing are recognised – where the evaluator’s behavior may affect the data generated from the interview (Mason, 2002).
Claims of bias are often made based on the evaluator’s participatory approach and interviewing processes. However, qualitative
8 Kushner, S 2002, ‘The object of one’s passion: Engagement and community in democratic evaluation’, Evaluation Journal of Australasia, vol.
2, no. 2, pp. 16–22. 9 Patton, MQ 2008, Utilization-focused evaluation, 4th edition, Sage Publications Inc., Thousand Oaks, California.
10 Funnell S. (1997). Program logic: An adaptable tool for designing and evaluating programs, Evaluation News and Comment; 6 (1)
11 Mezirow J, and Associates (2000). Learning as Transformation: Critical Perspectives on a Theory in Progress. San Francisco, CA: Jossey–Bass.
12 Guskey, T. R. (2000) Evaluating training programs: Evidence vs. proof. Training and Development Journal, 31(11), 9-12.
12 Kirkpatrick, D. L. (1977). Evaluating training programs: Evidence vs. proof. Training and Development Journal, 31(11), 9-12.
13 Mason, J (2002). Qualitative Researching. London: Sage.
8
methodologists argue that evaluation participants are not objects, but actively interact with the evaluator and that it is the
interpretation data and how they enable a convincing argument to be made with the data that is important (Mason, 2002). The
evaluation rigor and limitations are further discussed in Section 4.
2) Most significant change technique: To capture stories of change from the 2013 health literacy course participants and
their organisation senior executives an adapted Most Significant Change (MSC) technique 14
was used. The MSC technique is
based on a qualitative semi-structured interviewing participatory approach that involves the generation of significant change
stories. To assist in the generation of stories, interviewees were provided with domains of change that emerged from the 2013
evaluation program logic model (see Appendix 2). Traditionally, the most significant of the stories are selected by stakeholders
and in depth discussions of these stories take place. The discussions are designed to facilitate stakeholders to focus on the
impacts of the initiatives that have the most significant effects on the lives (or work practices) of the evaluation participants.
Given that a small number (n=9) of de-identified stories of change were generated from 2013 health literacy course participants
and their organisation senior executives, the stories of change were only distributed to the Health Literacy Steering Group (CEH,
HWP, cohealth) and no selection of the ‘most significant’ stories occurred, as all stories were perceived as worthy and illustrative
of the changes resulting from the participating in the 2012 course.
3) Member validation: In keeping with the recognition that evaluation participants should formally be included in all aspects
of the qualitative process (Straus & Corbin, 1998)15
validation is advocated either via triangulation or member validation.
Member validation is used to verify the adequacy of the analysis, by taking the results back to the field and asking if the
members recognise, understand and accepts one’s interpretation (Bloor, et al 2001)16
. Member validation was chosen as it
provided the opportunity for all evaluation participants to attend an Evaluation Participants Feedback workshop to hear, discuss
and add to the evaluation findings. While member validation is an effective method of corroborating findings, validation is not
just a test, or a way to obtain additional data, but an opportunity for reflexive elaboration – that may lead to altering the
evaluation participants or evaluators perceptions and interpretation of the data.
Overall the 2014 evaluation involved nine data collection activities (see Appendix 1 for copies of data collection tools):
1. one interview with 2013 course participants to explore individual and organizational changes resulting from participating in
the 2013 course
2. one interview with 2013 course participant organisation sponsors to explore individual and organizational changes resulting
from staff participating in the 2013 course
3. pre,-during and post course surveys of 2014 course participants (as part of the CEH quality improvement process) to explore
changes in individual health literacy competency and changes in organizational health literacy
4. two rounds of interviews with 2014 course participants to explore course experiences and individual and organizational
changes resulting from participating in the 2014 course
5. one interview with 2014 course participants organisation executive sponsors to explore health literacy drivers and individual
and organizational changes resulting from staff participating in the course
6. one interview with a sample* 2014 Health Literacy Community of Practice (CoP) Leadership Group to explore CoP purpose,
implementation, impacts and sustainability of the CoP
7. one interview with Health Literacy Community of Practice participants to explore CoP purpose, implementation, impacts
and sustainability of the CoP
8. one interview with 2014 course facilitators to explore experiences of facilitating the course; and
9. evaluation participant feedback workshop
• Please note the evaluation scope only allowed a sample of the CoP Leadership Group to be invited to interview. Furthermore,
to interview the two consumer representative would have required an amendment to the original ethics approval which did
not allow ‘any patients or consumers to be interviewed.
Table 2 provides a summary of the evaluation foci, evaluation questions, participant groups and data collection tools.
14
Davies, R & Dart, J (2005) The Most Significant Change (MSC) Technique: a guide to its use www.mande.co.uk/docs/MSCGuide.pdf;
http://groups.yahoo.com/group/MostSignificantChanges 15
Strauss, A. & Corbin, J. (1998). Basics of Qualitative Research. Techniques and Procedures for Developing Grounded Theory. London: Sage. 16
Boor, MFJ; Thomas, M & Robson, K (2001) Focus groups in social research. London: Sage Publications
9
Table 2: Evaluation Plan Overview
Evaluation Foci Evaluation Questions Participant Group Data collection
2014 Health
Literacy Course
• What are the perceived impacts of
the Course?
• What changes in individual
participant health literacy
competencies (specifically, health
literacy practices) pre, during and
post course, have occurred.
• What changes in organisation
health literacy attributes pre, during
and post course, have occurred
• What contextual factors have
influenced these changes?
• 2014 Health Literacy
Course Participants
• 2014 Health Literacy
Course Participant
Senior Managers
• Two semi-structured
interviews
• Pre-during-Post course
survey
• Evaluation Participant
Feedback Workshop
• One semi-structured
interview
• Evaluation Participant
Feedback Workshop
Community of
Practice (CoP)
• What assumptions exist about how
the CoP is supposed to work?
• To what extent has the CoP been
implemented as intended, and
what contextual factors may
influenced its implementation?
• What are the perceived impacts of
the CoP?
• To what extent does the CoP build
on and sustain learnings obtained
from the Course?
• To what extent is the CoP
sustainable, and what contextual
factors may influence its
sustainability?
• CoP Leadership Group
• CoP Participants
• One semi-structured
interview
• One semi-structured
interview
• Evaluation Participant
Feedback Workshop
Ripple Effect:
Changes
• What changes at the individual
level (behaviour, roles, attitudes,
skills, knowledge, relationships,
capabilities) have resulted from
participating in the 2013 Course?
• What changes at the organisational
level (culture, structures,
governance, processes) have
resulted from participating in the
2013 Course
• What changes at the systems level
(policies, information, regulations,
education/training, key
performance indicators/HR) have
resulted from participating in the
2013 Course?
• 2013 Health Literacy
Course Participants
• 2013 Health Literacy
Course Participant
Senior Managers
•
• One semi-structured
interview to capture -
Ripple effect* (stories of
change based upon Most
Significant Change
Technique)
• Evaluation Participant
Feedback Workshop
The evaluation was submitted for ethical review to the Melbourne School of Population and Global Health Human Ethics
Advisory Group and was granted approval.
10
3. Evaluation Findings
This section provides key findings from all the evaluation activities, organised under the three evaluation foci: 2014 Health
Literacy Course; Health Literacy Community of Practice; and the Ripple effect.
3.1 2014 Health Literacy Course
3.1.1 Course participant profile
Ten HWP member agencies from the West participated in the 2014 course, with two people per organisation attending (n=20).
Table 3 reveals the broad spectrum of participant sectors, organisations and positions participating in the course.
Table 3 Course participants sectors, organisations and positions
Course Participants Sectors Course Participants Organisation Types Course Participants Positions
• Public health
• Not for profit
• Community Health
• Acute care
• Nursing education
• Mental Health
• Aged Community care
• Primary health
• Aboriginal health*
• Hospital
• Medicare Locals
• Community Health Centres
• Primary Care Partnerships
• Peak Health Organisations
• Clinician
• Project officer
• Manager
• Coordinator
• Educator
• Personal Assistant
*Please note – one course participant indicated in the pre-survey that they currently worked in the Aboriginal health sector.
3.1.2 2014 Course Participant Pre-During and Post course survey
A key evaluation goal was to assess the perceived impacts of the 2014 Health Literacy Course on course participants and their
organisation CEOs. More specifically:
• To assess changes in individual course participant health literacy competencies(specifically, health literacy practices) pre,
during and post course
• To assess changes in course participant’s organisation health literacy attributes pre, during and post course
• What contextual factors have influenced these changes?
To assess changes in course participant health literacy competencies and organisation health literacy attributes all participants
were invited to complete a brief survey pre, during and post the course.
Appendix 4 provides survey findings: Pre (n=19), during (n=17) and post (n=14) the course.
Individual Course Participant Health Literacy Competencies
On the basis of the course participant’s pre-during and post course surveys, perceived competencies implementing health
literacy concepts into practice have changed from finding it “difficult” to finding it “easy” to putting health literacy concepts into
practice. Course participants commented that being asked to complete the survey on their individual health literacy
competency was meaningful, as it provided points of self-reflection for them (see Appendix 4).
Organisation Health Literacy
Over the course duration, course participants perceptions about the health literacy of their organisations have also shifted, with
participants being either mainly undecided or disagreeing initially that there organisation was health literate to ‘agreeing’ that
that there organisation was health literate post course. Course participants commented that being asked to complete the
survey on their organisations health literacy was not really meaningful and a challenge, as organisations were complex and
comprised of multiple sections, levels and departments – hence difficult to make generalisations about the health literacy of
their organisations. Please note, while course participants commented that they found it difficult to repond to the
survey questions about the health literacy practices of their organisations, they were able to reflect on these type of
questions when interviewed. (see Appendix 4).
11
3.1.3 Round One Interviews with 2014 Course participants
All 20 course participants received an email inviting them to participate in the first of two semi-structured interviews. A total of
14 course participants agreed to be interviewed. A summary of key findings are presented below with illustrative quotes:
Participant motivation for attending
Participants commented that they had a genuine interest in learning more about health literacy and that they could see how it
could inform and support their roles in their organisation and more broadly.
• “I was nominated- I had heard the term quite a lot about health literacy and had a basic understanding, but it was wanting
to learn more about it and how I can shape, improve the outcome through respect to health literacy in the organisation and
the broader community” HL1- 3
Health literacy understanding:
Course participants were asked to reflect upon their understanding about health literacy prior to participating in the course.
Most participants commented that they had little or a broad understanding of health literacy.
• “I actually didn’t have much understanding of health literacy at all myself. I think I've got a much greater understanding
now and it's actually opened my eyes quite a bit…” HL1- 13
Having participated in the Two Course module, participants commented that their awareness, knowledge and skills had
increased.
• “I think it was a broad understanding. It's provided me with some resources and tools to assist in that systematic approach
to health literacy”. HL1- 14
• “I thought I had an understanding, but I now realise I didn’t. It broadened and made my - it's making me think a lot more
about how” HL1- 7
Course participants also commented that the course was assisting them view health literacy from an individual point of view to
more an organisational lens.
• “I looked at it from a very individual point of view, about patient understanding. Whereas the course gave me a bit more of
what can organisations do to improve health literacy” HL1- 12
• “No, I don’t think I had heard of it. I think if I did start hearing. But now I can see that health literacy is such a broad term
with so many definitions, that I think it just means for us as an organisation, I mean we're working on this now, but I think
health literacy for us is just making sure our community, our local area understand where to go for further information” HL1-
7
• “Yes. I didn’t realise it was so big. You tend to think health literacy is more for, I don’t know, non-English speaking people, et
cetera, but to learn that nearly 50 per cent are health illiterate” HL1- 9
Project contextual factors
All course participants were asked about contextual factors (enablers and barriers) to implementing their health literacy
projects. Table 4 provides a summary of key enablers, barriers at an individual, organisational and systems level.
12
Table 4: Summary of Project Implementation Enablers and Barriers
Project Implementation Enablers Project Implementation Barriers
Individual
• Compliance from clinicians
• Senior management support, networks and know-how
about organisation
• Upper management on board
• Support from frontline /on the ground staff to health
literacy
• Cooperation from other team leaders
Individual
• Staff changes
• Project too big
• Time constraints
• Getting upper management on board and there approval
• Variable understanding about HL at upper management
level
• Competing priories at upper management
• Terminology – health literacy
Organisational
• Alignment between project and organisation tasks e.g.,
accreditation
Organisational
• Difficulty accessing project specific data (eg., patient
complaints)
• Ongoing organisation changes and re-structures
• Organisation has multiple sites which differ greatly –
organisation, priorities, and competing interests
Systems level
• Health literacy project aligns / fits with accreditation
system / standards requirements
Systems level
• Funding cuts
• Redundancies due to funding cuts
Course impact
All course participants were asked about any changes they had made as a result of participating in the course. Overall, for many
it was too early to comment on actual changes.
• “No, but it's going to be coming because I know that there is a component of training staff - for feedback of that process, the
results of the focus testing - but no, not yet, but it will be coming” HL1- 3
For one participant who had been made redundant, they commented that the course would have a lasting impact, as they
intended to use their new knowledge in their new jobs as seen below:
• With the redundancies, none of that work will continue. Look, I'm definitely converted to the importance of health literacy,
so wherever I go, that'll come with me anyway. I'm keen to look at and contextualise - maybe with my new job - is the
checklist for organisations where they can assess how they go and what health literacy. HL1- 14
Others commented on having a greater awareness had impacted upon their work:
• “I think probably just greater awareness. Therefore when I am delivering that health information I'm stopping and thinking
about the way I'm delivering that and making sure that I'm incorporating those practices such as the teach back myself to
ensure that it is understood”. HL1-13
For other participants the course had made them more reflective on where organisation was at, in particular with regard to
accreditation:
• “I think the fact that we’re preparing for accreditation, we’re ready in principle, but in terms of rolling it out to the whole
organisation, I think there’s work to be done”. HL1-9
Course reflections
Overall course participants commented positively on the course content, structure and presenters.
• “I found it very well run. I found the content very useful” HL1- 13
13
• “I've found them very interesting, very informative” HL1- 7
• “I've found it all to be excellent. I'm eager for information because it's something that I've not been exposed to before, but I
can see huge potential implementing all the learnings from this course with myself and anyone else who goes through it”
HL1-5
Participants commented on how the course was providing participants with new resources:
• “I think it's been good. It's been providing with a lot of tools and techniques moving forward” HL1- 12
Participants also commented on how the course was facilitating / building networks among course participants.
• “It was great also that networking opportunity with people from different areas” HL1- 14
• “There has been encouragement to network...”HL1- 2
3.1.4 Round Two Interviews with 2014 Course Participants
All 20 course participants received an email inviting them to participate in the second of two semi-structured interviews. A total
of 14 course participants agreed to be interviewed. A summary of key findings are presented below with illustrative quotes.
Understanding of health literacy concepts and practices
Course participants were asked to reflect upon to what extent their understanding of health literacy had changed or developed
since they began the course. Overall course participant reported that the course had increased their depth and breadth of
knowledge about health literacy concepts and how to apply them into practice. However, several participants did report that
explaining health literacy concisely was still a challenge.
• “My understanding of the concept has increased but to concisely explain what health literacy is can be a bit tricky” HL2-1
Most participants also emphasised how the course had reinforced that health literacy was not solely an individual trait but that
organisations had a role.
• “…my initial thought was it was more of an individual capacity. So someone's capacity to - probably the literal
understanding of literacy almost. Now understanding the interrelations between an organisation, the policy environment
and also really just that it's not just for CALD populations or for people with non-English speaking backgrounds” HL2-5
• “Yeah, it's definitely changed. I think before probably starting the course and also the role in the project officer I probably
did think about it a bit more from the individual's perspective as well, but now I use it much more about what the
organisation can do and what the employees can be doing to actually be helping the patients” HL2-3
• “I've got a better understanding about how it's not just the individual that's coming to our service and what they bring. But
it's also what our clinician level of understanding of it is and ensuring that our clients are leaving with an understanding of
what's been explained to them during their sessions and pitching interventions, discussions and education at the level of
each individual client. Also a better understanding about our organisation's responsibilities in relation to health literacy as
well” HL2-2
Implementing health literacy practices
Course participants were asked to reflect upon to their experiences taking course learnings back into their workplaces.
Underpinning this question was the fact that the course was designed to support participants facilitate change at the individual
and organisation level – with participants being ‘agents of change’. Overall course participants commented on the readiness
14
and willingness of staff and colleagues to listen and take on board health literacy facts and practices. When asked about
whether participants considered themselves as agents of change, mixed opinions emerged.
Most participants confirmed that they did see themselves as an agent of change and that was the understood goals/ purpose of
the course. This finding confirms that the selection process that was undertaken by the HWP member agencies to identify
relevant individuals to be agents of change worked.
• “yeah. I think it has to be that. I think that's the purpose of the course - is to come back with a bit of fire in your belly and
say, look, this is really relevant to all of us. But I think it's also a bit of a skill or a skillset to try and then articulate that and
be a champion for it, I guess” HL2-5
• So as an agent of change, I guess, I can educate from the ground up as well because I'm tapping in to the admin team. So
when we're talking about changing our phone system over, we'll go to put it on to a press one, press two, press three, and it
was no, that's not a health literacy angle. That's not going to support the consumer on the end of the phone who may have
dementia and forgotten why they called. By the time they get to the end of that message they don't know what they're
doing. So that's where I think my manager was seeing I can feed it in to so many doors, if you like, of the business. HL2-7
• Well I'd like to, yes. I mean I would like to be able to implement that. That would be fabulous. I mean it's one of the goals of
actually participating in the course isn't it, the [unclear] to be able to make these changes. So it may take us a little bit of
time and first steps is a policy change and getting that understood and then we can hopefully go from there. HL2-13
• “Yes agent of change as sits well with me in my role as an educator – need to implement change..but a challenge though is
getting buy in from colleagues” HL2-14
While other course participants, did not see themselves as ‘agents of change’ due to their role as Project Officers, with no
decision-making status nor roles.
• “Not really agent of change – ok for managers role but not for individual project officers” HL2-9
Course Impact
Course participants were asked what changes they had made as a result of participating in the course. Overall participants
reported: increased knowledge about health literacy; increased communication, writing and document reviewing skills;
increased critical assessment skills; and overall being more mindful.
• “I think at my level of individually noticed - I get involved in a lot of document development for clients or reviewing document
development and so I think I've picked up lots of skills that are enabling me to develop a better end product for the clients
like brochures and client education and information material and things like that. I also produce a quarterly community
health newsletter and I really have gone at it with the health literacy sort of lens to bring those principles into that
development”. HL2-2
• It's helped me be a lot more critical in assessing where we are as an organisation, I think. HL2-5
• “Well I think that I have, in my own writing, have just tried to really think about what I'm writing more so. Everybody gets
into the same pattern of writing by themselves and the learning. So really thinking about, now hang on a moment, how is
that going to sound to somebody else. So being more mindful about that I think that is something that I have been trying to
do. By doing that I have actually focused on how I say things when I'm writing. So that has been - I've found that really
useful”. HL2-13
Course participants were also asked what health literacy practice changes had occurred within their organisations. Overall
participants commented that their organisation was ready and willing to adopt health literacy practice, as demonstrated from
them supporting them to do the course and to implement the health literacy projects.
15
• “Our manager and program director..championed not just the topic of health literacy but actually the practice” HL2-1
Several participants also recognised that despite their organisation being ready to adopt health literacy practices their
organisation was only at the beginning, and that change across organisations with multiple sections would be a challenge.
• “Yes organisation ready but only at beginning…” HL2-14
• “So we have - well, we have made, I think the ball is rolling, but I still think that because the organisation is so large, the ball
is rolling in certain areas and maybe not others as much. HL2-3”
Most participants also reported that visible changes had occurred within their organisations, such as increased staff awareness
about health literacy.
• “I would say it's probably changed a bit. I'd say that initially there was not much awareness throughout the organisation but
I think from [participant] initially rolling out some short half-hour sessions that sort of captured - I wouldn't say it's a lot but
it's captured two, three or four more staff and their awareness”. HL2-2
Contextual influences upon Individual & Organisation Level Changes
All course participants were asked about what contextual factors (enablers and barriers) may have influenced any individual and
organisation level health literacy practice changes. A core set of enabler and barriers at an individual, organisation; and system
level exist (see Table 5)
Table 5: Summary of Contextual Influencers - Enablers & Barriers
Enablers Barriers
Individual
• Clinician engagement
• Senior management support
• Support from frontline staff
• Cooperation from team manager
Individual
• Engaging colleagues
• Staff changes
• Staff redundancies
Organisational
• support from our manager and program director
• funding dedicated to implementing health literacy practices
• Support from board
Organisational
• Multiple organisational authorisation processes
• Ongoing organisation changes, re-prioritisation, re-
structures and funding cuts
• Ongoing organisation changes and re-prioritisation
• Ongoing organisation re-structures and funding cuts
Systems level
• Accreditation requirements
Systems level
• Ongoing reforms and restructures across sectors
• Redundancies due to funding cuts
Course content structure and participant composition
All course participants were asked to comment on the course content, structure and presenters.
The four module and 2 projects structure was positively received as it enabled reflection, putting concepts into practice and
seeing immediate benefits, particularly the projects. The mixed organisation and participant composition was positive as it
provided opportunities to hear about the differing challenges and opportunities that existed.
Course Content
Overall the course content was perceived as very informative, comprehensive and interactive.
16
• “The presenters were very professional. The information provided was more than adequate. They certainly were committed
to what they were doing. If they said they'd do something, they would. It was a good mix of interactions, lots of opportunity
to forward ideas and feedback”. HL2-7
Several participants commented positively that it was noticeable that it was a development course – illustrating the responsive
and adaptiveness of the course and the course facilitators
• “…it is quite obvious it's a development course and that does come through…I think it's a positive thing though... our
opinions are taken very much on board in what we have to say rather than we're the teachers, you're the students. So I think
it's a positive thing that we're all kind of contributing to this course if that makes sense”. HL2-1
• “Course trainers picked up energy of people and where at and they changed – very responsive” HL2-8
Only one course participant commented that the course was too repetitive, leading to a decreased value of the course overall.
• “Value of course and content dropped off across module as too repetitive” HL2-12
One course participant commented that the course could have had a broader scope than just cultural diversity, to also address
disability or diverse abilities.
• “We were the only mental health team doing the course. Cultural diversity is generally a consideration in health literacy but
when you're dealing with people with varying degrees of cognitive ability and people who have difficulties with
comprehension because of mental illness. I just thought that course could have looked more broadly around the health
obstacles experienced by people in understanding health information. I thought a stronger focus was around the cultural
diversity but I'm just thinking there's also scope there to think about disability and different ability, diverse abilities…I'm
thinking that if we developed a mental health specific four day session - I don't know where you would get the money for
that. Then you'd actually have commonalities of experience and a bit more diverse discussions”. HL2-4
Course Participants Composition
Overall there were mixed views about the composition of the course participants. It is important to contextualise responses to
course composition, as recruitment for the course was from HWP member agencies.
Several participants commented that having a mix of staff from across health and non-health agencies reinforce the fact that
health literacy was a key issue for all.
• “It just shows that health literacy isn't just in acute settings, it is across all the healthcare”. HL2-1
The composition mix also provided the opportunity to network and learning other viewpoints and challenges.
• “I think there were lots of opportunities to liaise with lots of other staff from other services that was good as well and get
their views” HL2-1
• “Good mix of health & non-health – challenges when we hear it” HL2-11
• “Good to have mix – not just health” HL2-8
• “I think that was a positive because I think we can all relate to lots of different projects that we were doing anyway, so at
[organisation] we've got an acute hospital and we've also got a community health centre as well, so I think it was really
relevant to have people from different areas. It was interesting to hear their stories and what they're doing, I thought that
was quite useful” HL2-3
17
• “I think it was really useful. I think that it actually helped us - helped me I guess to know more about these organisations
that are out and get a better understanding of what their point is and how they stand with health literacy. So I think that
was a good thing” HL2-13
Other course participants commented that having both front-line and management staff was a positive as it gave them broader
views of the issues facing organisations.
• “…I thought there was a really good blend of people and some really interesting organisations. I liked the fact that there
was a lot of real direct frontline. There was people from the management side or more policy side. That gave me a broader
view of the issues facing organisations. HL2-5
However, several participants also commented that despite being positive about having a mix of staff and agencies, not dealing
with patients, and not being in the service industry meant that often other course participants were not relevant nor aligned
with their thinking.
• “Good to have mix- but I do not deal with patients directly – so some were not really relevant”. HL2-9
• “I guess the only thing that I struggled with was that a lot of the other people at the course - and this just might I'm the odd
one out - because we were in the service industry… the rest of them, I think, were all from medical or mental health
backgrounds. Sometimes it was difficult for me to align myself to their thinking - although it's all the same across the board -
but it would have been good to see some other people coming from this type of industry as HL2-7
Another course participant, while recognising that the course participants were drawn from a membership base, commented
that given the recent health care reforms regarding Primary health Networks, the course could have been open to a broader
participant base such as to private or commercial entities or even insurers
• “Yeah. I really thought the - there was a really good mix of the actual organisations involved. I know it's probably - I know
it's drawn from the membership, but I think it'd be interesting, particularly in the new environment that we're all going to be
launched into where they’re trying to pull in private organisations into the health networks and things, that there's probably
some really good activities even around resources and things. So there's - I’ve seen some excellent Gamblers Vic materials
for CALD communities and things like that. I think there's probably some private or commercial entities that are doing really
excellent work. It might be interesting - even the insurers or even like that that often have a lot more resources than the not-
for-profits or the government funded, where they can actually invest in it and even - yeah. So...HL2-5
Course Structure
Overall there were mixed views about the structure of the course.
Most participants were positive about the course being spread over the year with the mini-projects as it allowed reflection,
building knowledge and making changes by putting learnings into practice.
• “I liked the fact that it did go over the year and that it was just one day every two months because it did allow you then go
back and reflect on the information that you had learned…I also liked the mini projects like I've already mentioned because I
think that that actually got you to put what you'd learned into practice as well and it made it more practical”HL2-6
• “A lot of the time when you do attempt professional development, you go for a day or two, you go and then you come back
and then you might make a couple of changes and then a couple of months later maybe you’ve gone back to your old ways.
Whereas with this program I felt like we were continuing to build on our knowledge over the year, and the facilitators, they
were great”. HL2-3
18
• “Modules and projects flowed well…Project solidified things for me” HL2-14
Several participants commented that the course could have been compressed into fewer days or even having half days – with
differing reasons, such as: not all course material was specific to health literacy; eight hour block were too much and too
repetitive; and to maintain momentum. Please note, the accuracy of comments by course participants about the course
structure need to be contextualized, with the knowledge that the course was factually of eight months duration with six hours of
training every two months.
• “Could have been compressed into fewer days – as some of material is not about health literacy, but about ‘how to do power
points,..maybe not the whole day –maybe half a day HL2-9
• “8 hours too much – to repetitive” HL2-12
• “I thought that maybe it might have been more regular sessions. I think there were was one every three months and
personally I would have thought to have - monthly would have been good thing”HL2-13
• “I'd have maybe have liked to have seen maybe half days more regularly than the four full days. It was quite difficult to
maintain some of the momentum through the year, which - is it quarterly. So almost like a half a day every two months
might have been a bit more useful for me” HL2-5
• “Yes. Too long over 12 months. That's just a case that you've done a course and you're in a roll kind of thing. Come on, let's
do it kind of thing and you've got to make sure that the momentum keeps going over that period” HL2-13
One participant commented that providing ‘place-based visits’ could enhance the structure of the course to further demonstrate
the implementation of health literacy practices
• “Do more place-based visit e.g., Western Hospital” HL2-10
Several participants commented that the timing and provision of information in relation to the Modules, projects and partner
roles - could have occurred sooner to assist with planning. Please note, the accuracy of comments about information on the
modules, projects or partner roles need to be contextualised, with the fact that information was provided in all materials, and
course participants were required to sign off on them as part of the application process.
• “…40 hr good but only heard about project in Module 1 – did not have enough time to plan ahead – need to work in with
other priorities…need to spell out at the start what expect re project and partner involvement HL2-11
One participant commented that a ‘health literate product could have been worked upon and generated by the whole
participant group.
• “I would have liked to have seen something tangible generated by the group, maybe…So maybe just if it's safe as it was a -
even a five full day or something that one kind of resource is worked through, so all of those readability tools - pictures,
infographics, that kind of thing - where - so because as I say, there was so much input from the management or the different
aspects of different organisations, it would have been nice to see a product out of it. So that everyone goes through the
actual - yeah. I guess the process of creating a health literate item, I guess” HL2-5
In relation to the course structure, several participants commented that despite the course facilitators encouraging all to
network with participants within the modules, beyond their organisation colleagues and outside of the course, networking
occurred predominantly with their organisation colleagues.
19
• “No, because I did have a colleague attend as well: So no, we didn't. Look, sorry, there was an opportunity to, but I didn't”.
HL2-7
• “Probably not outside the course - the days that we were there. That was something that didn't really - I don't know
whether it was applicable. There was no - I didn't really have any need to other than to find out what they were doing and
then we did have information sent through about what each group was doing anyway. But on the day, no, we'd have
conversations” HL2-13
3.1.5 2014 Organisation Executive Sponsors Interviews
All 10 course participant organisation executive sponsors received an email inviting them to participate in a semi-structured
interview. All 10 agreed to be interviewed. A summary of key findings are presented below with illustrative quotes.
Health Literacy Drivers
All executive sponsors were asked about what motivated their organisation to focus upon health literacy. Several commented
that health literacy was a core individual value and organisation value – thus a strategic and operational priority:
• “…for us it's, I suppose it's congruence with our values. We're a - every organisation's values driven I know and has their sets
of values, but our organisation is quite a walk the walk kind of organisation. Our values in relation to compassion,
hospitality, respect, etc cetera extend to how accessible our services and our information is for our consumers. So I think
that's really at the heart of it. HL ES 8
Others commented that compliance with national standards overall and national standard two was a key driver:
• “I will say that at the moment there's a strong compliance focus also under the national standards and I would lie if I didn't
say that compliance wasn't a strong driver for this effort as well with the national standards and standard two” HL ES 8
Several executives mentioned that health literacy emerged as a core priority from there organisation’s population health needs
assessment processes:
• “So we've had fairly sustained investment in it. As I say, it's not just one needs assessment, but a series of needs
assessments, and they've all highlighted similar issues. Really, we've got to develop the capacity internally to understand and
- if we're going to develop programs and strategies that are going to take health literacy into consideration, we've got to
have staff here who have got those skills. The most appropriate way to do that is by sending them off to professional
development” HLES 5
One executive commented that having an organisational structure (e.g., Consumer Advisory Groups) which strongly advocated
for a focus on health literacy was a key driver:
• “We have a consumer advisory committee that we - I suppose advocating and being a source for us to check that if that's
occurring. So it's sort of built momentum from that particular process” HLES 7
Changes in health literacy practices
All executives were asked what changes in health literacy practices at an individual and organisational level they had noticed
since their staff had attended the course. A broad array of changes were reported including:
Several executives reporting developing and adopting health literacy policy and procedures.
• “ main changes for us was that we've embraced an organisation wide commitment to health literacy through the
development and adoption of a policy and some procedures falling out of that. The impact of those procedures and
guidelines I think are yet to be declared” HL ES8
20
Several executives whose organisations had sponsored staff to attend the 2013 and 2014 course reflected upon the time it takes
and the ongoing transformational change process (awareness, development, adoption, implementation, sustainability) required
to put into practice new concepts.
• “…it's amazing it's actually taken two years to actually get some runs on the board, if you like, so that the first year I don’t
think that there was a lot, other than general awareness raising, and having the people that were attending the course talk
at brief intervals, if you like, at meetings and that sort of stuff; whereas, this year, what we've actually found is that we've
been able to embed it in some of our language. The other thing that we've actually done is we've actually used it to develop
up some of our communication material with clients, which can be terrific, and been really well received. HL ES 6
• “…we didn't get a lot out of it last time because it was - just insofar as I know how long it actually takes to implement a new
- I know this isn't new, but it's new to us - to implement a new concept across an organisation, and particularly when it's
such a crowded space anyway, insofar as what's happening more broadly in the industry” HE ES6
Other executives commented on specific changes in staff practices, such as: improvements in written communication; and
championing and advocating for health literacy.
• “I think the other thing that we were expecting and we are seeing as a result of it is staff being more conscious of
information flyers and brochures and how they write things. The two staff that attended the course have been great
advocates for again educating other staff HL ES 7
• “What I've noticed with [participant] is that because of the specific work that she's been required to do she's had to go
through the, almost a discipline of adopting health literacy principles to a particular body of work and I think that she's
developed some skills around writing as a result, and perhaps critical review of particularly written information for
consumers” HL ES 8
Contextual influencers upon routinely using health literacy practices
All senior executives were asked about what enablers and barriers existed to routinely using health literacy practices. From
Table 6 it can be seen that senior executives mainly reported individual level enablers and barriers.
21
Table 6: Summary of Contextual Influencers - Enablers & Barriers
Enablers Barriers
Individual
• Having dedicated time and resources for staff training
• Embedded health literacy concepts into tools that staff use
• Having staff participating who are connected and have
decision making status
• Having staff with skills and commitment to health literacy
practices
Individual
• Having capacity to influence
• No clear benefits to routinely using health literacy
practices?
• Complexity of health literacy and work required to
facilitate change
• Keeping momentum up
• Paternalism within health professions
• Staff not-complying with organisation expectations
Organisational
• Congruence with organisations values
• Compliance with national standards
• Embedding health literacy approaches into organisation
tools
• Organisation Executive support
• Having annual business planning to drive organisational
action on health literacy
• Health literacy interweaved as a key focal area
Organisational
• Competing organisation priorities
• Ongoing organisational changes re priorities and
structures
Systems level
• Embedding health literacy approaches into organisation
systems
• Incorporating health literacy into human resources
management systems
• Viewing action on health literacy as a quality improvement
approach
Systems level
• Requirement to show value of taking action on health
literacy
• Takes time to see seeing changes in policy and
procedures
Key enablers: included:
• having dedicated time and resources for staff training
• embedded health literacy concepts into tools that staff use
• “Well, the enablers are insofar as were embedding it into the tools they're actually giving staff to use. So insofar as that will
be actually breaking down the barriers, I suppose it would just become stuff that they do”. HL ES 6
• Having staff participating who are connected and have decision making status
• “Having [course participant] do it has been absolutely fantastic because we work really closely together. So she's obviously -
and the other thing that she does is she drives a lot of the quality stuff. So that's where we've been able to embed it quite
readily” HL ES 6
• Having staff with skills and commitment to health literacy practices
• “…look enablers have been that I think people are - we've got a lot of staff that have a lot of skill in the health literacy area
regardless, or at the very least understand the logic and are committed to the concept and, as I said, the kind of values base
22
beneath that…I think we've got a lot of enablers also in this environment of which some of the specific knowledge and
outcomes of the work of this program have made a significant contribution to. HL ES 8
• Congruence with organisations values
• “Look, for us it's, I suppose it's congruence with our values. We're a - every organisation's values driven I know and has
their sets of values, but [organisation] is quite a walk the walk kind of organisation. Our values in relation to compassion,
hospitality, respect, et cetera extend to how accessible our services and our information is for our consumers. So I think
that's really at the heart of it”. HL ES 8
• Compliance with national standards
• “I will say that at the moment there's a strong compliance focus also under the national standards and I would lie if I didn't
say that compliance wasn't a strong driver for this effort as well with the national standards and standard two. HL ES 8
• Embedding health literacy approaches into organisation tools and systems
• “Absolutely. The thing about [health literacy] is embedding it into some of the communication tools that need to actually run
through that quality management system. HL ES 6
• Organisation Executive support
• “The executive was supportive in funding staff to attend, with the expectation that they would come back with a plan and that
that would be presented back to the executive” HL ES 4
• Having annual business planning to drive organisational action on health literacy
• “…for me this is - our annual business planning is where I put the patient information centre, once it's sitting in my annual
business plan I have to deliver on it. HL ES 3
• Health literacy interweaved as a key focal area
• “…health literacy was one of our key areas of focus anyway. So I think what it's done is just given them a bit of grounding in
their - and I guess it's just interweaved through nearly everything we do”. HL ES 2
• Incorporating health literacy into human resources management systems
• “Look, our HR team have been very supportive of us incorporating [health literacy] asking the question because they're on
our diversity working group, but I'm sure I would have equal support if I wanted to incorporate it into staff induction, health
literacy”HL ES 1
• Viewing action on health literacy as a quality improvement approach
• “Yeah, yeah. Also it's a quality improvement approach. It's not just like oh well, we've got to have these resources translated.
It's a much more - let's try and translate some resources and see how well they're received, and then being - is it making a
difference. HL ES 5
Key barriers including:
• capacity to influence
• “ how much ripple effect can you expect to get from that? While the people who went the first year - and so this is not,
definitely not any - they're good people, but their ability to influence in 5000 staff is pretty limited. I think most of the
organisation would never know this existed or that people had gone to it. HL ES3
23
• no clear benefits to routinely using health literacy practices?
• “…we haven't converted people I don't believe, rightly or wrongly, what it can practically bring to bear, can bring fruit to. So
what is it actually going to achieve? Is it going to achieve less presentations to health services? How can you convert it into
something that converts back into a business case [or/a] resource? It's got to be able to deliver something to free up
resource, to re-invest…”HLES 3
• complexity of health literacy and work required to facilitate change
• “I think it's a very valuable program. I think the problem with a lot of these things is that individuals who participate in them
can become somewhat isolated. Because of the, I guess, complexity of health literacy and also the volume of potential work,
running ongoing focus groups that would allow them to share what they are doing and focus more on what is potentially
achievable - and/or they can partner with another organisation - would actually probably help refocus their attention”.
HL ES 4
• keeping momentum up
• “I think the biggest barriers - look it's got momentum at the moment, but as participants finish their projects the risk is that
we start to lose the momentum again. So I suspect that it's the sort of course that I think we would be keen on sending
another couple of staff to almost on an annual basis to keep that momentum and rhythm going” HL ES 7
• paternalism within health professions
• “the sector that we find ourselves in that this persisting kind of paternalistic top down cultural health professional. For
certain professional groups the inability, or the - some might see improving, becoming a more health literate organisation is
the dumbing down of medicine, as an example and it's a very extreme view. But it's difficult for some physicians I think to
connect because of the way they've been raised, I guess as health professionals which has been in that we do it to you not
with you kind of approach. So I would see those as persisting barriers perhaps that are long term cultural shifts. But I think
we've got a lot of enablers also in this environment of which some of the specific knowledge and outcomes of the work of
this program have made a significant contribution to. HL ES 8
• Competing organisation priorities
• “…when I'm on the Community Advisory Cultural Diversity Committee. Their issues read something like, interpreters,
interpreters, interpreters. Followed by food, food, food. Followed by parking, parking, parking. Followed by ED access, ED
access…. - it's difficult then when you're in the exec position to say, if I had resource where would I put it? Would I put in the
health literacy...if it's picked up as a strategic theme for [Organisation] it could well get itself into that league. But it's going
to be in competition obviously with some other themes. HL ES 3
• Requirement to show value of taking action on health literacy
• So I had this opportunity come up, the Better Health Plan for the West – we put a lot of focus on this. Again, it's that
difficulty about sure there's focus but what's some practical things we can actually do to get them wins really? To see if we
can get some wins around it. HL ES 3
• Takes time to see seeing changes in policy and procedures
• “No. I think it's a bit too early for [seeing changes in policy and procedures] because the timing has been that we've just gone
through accreditation” HL ES 5
24
• Staff not-complying with organisation expectations
• “The executive was supportive in funding staff to attend, with the expectation that they would come back with a plan and
that that would be presented back to the executive. That hasn't happened”. HL ES 4
Strategies to sustaining changes in health literacy practices, including
All senior executives were asked about what strategies could support changes in health literacy practices. A broad array of
comments and strategies were suggested.
• Keeping momentum up by touching base with 2013 and 2014 course participants in 2015
• “What could we do insofar as to keep the momentum up? Probably just touch base with the guys that are running through
the course maybe. These guys will finish next year, so it would be good to actually touch base with them, maybe twice next
year. It doesn't have to be face to face. It might be via a teleconference or something like that, how's it going? How are you
going with - have you had any more - how are you going with the initiative that you've just pulled together? Have you
actually implemented any other initiatives that you'd like to share? Then that - then people can be thinking okay, so I know
that I'm actually going to be called to task, for want of a better word, around thinking about this mixture. I'm not just going
to finish it at the end of this year; go tick, did it, wasn't that great Yeah, exactly. Then I think if you did that for another year,
I think that that would then maintain the momentum in the organisations of those participating” HL ES 6
• provide annual investment in staff professional development
• “Yeah, look I'm a fan of the program. I think that if there's the opportunity to run it again would absolutely support
nominating. I think it's a very valuable program. I know that the two that have participated have, value very highly. We've
got some fantastic things from it this year, and really important things for us. I think that any kind of long term view of how
you build capacity in health for doing this kind of work probably needs to, might need to be multilevel. So both the most
specific send the people away, do the deep work and work out how you can continue to build capacity back in organisations
by having that, people come in. Because I know what the objective is of it, but I'm a fan, so I'll be watching the space with
great interest and as I said supporting and sponsoring any further participants if we're fortunate enough to have that
opportunity down the track. HL ES 8
• maintain project requirements to get immediate benefits
• “I like the course content where they actually come back and do a project in the organisation. I think that that also makes it
quite sustainable and we get immediate benefits from it. So you feel like you're - by sending staff there and getting them to
invest their time or our time into that course, you feel like you get a direct benefit back from it” HL ES 7
• continual focus on supporting a cultural shift – Health literacy part of good person centred care and NOT additional work
and not a Fad
• “…look I think we've got to keep on ensuring that we have the resources to support our advocates and our change
champions, and that's not just throwing money at people to do programs or giving people extra time to do educational units
to do that. It's about being able to have enough capacity to identify within our work groups where it sits and place it central.
For me I don't think it's about talking about health literacy as a particular defined end to end thing, I think it's talking about
health literacy for my organisation in the context of person centred care as a key component of it rather than something
separate to it. Again, going back to barriers, I think that's one of the things that where people put the window down fairly
quickly is when they hear oh, this is the latest thing, health literacy is the latest thing. When do we get the time to do that.
A lot of it's already been done, but we've got to make it relevant and central to the patient centred care movement. So that
whole shift in partnering with consumers rather than doing things to them” HL ES 8
• building interdependent self-sustaining organisational focus on health literacy
25
• “… I'd like to be able to see our own organisation build enough skills and knowledge to be able to kind of do it ourselves in a
sense. I think we've still very much at the stage of needing support of an external program to, like the one that we've had
this year, to participate in and bring that information back. But ultimately it needs to be central to our organisational
development” HL ES 8
An interview with a senior executive also revealed a recognition that the current course had been trialed and targeted at an
organisational level with HWP member agencies, and that it was worth exploring running a regional health literacy course”.
• “…maybe looking at running a regional health literacy course. Because we’ve trialled it in the west and just looking at
other executive officers, to see whether they would have any interest in doing something at a regional level”
2013 Senior Executive
Executive Sponsors Workshop, 16th
September, 2014
The workshop was designed for 2014 course participants and their organisation senior executive sponsors to: provide them with
a strategic view of health literacy in the West; to workshop the link between health literacy and accreditation standards; and to
discuss enablers and barriers to supporting organisational health literacy.
Overall senior executives were very supportive of the workshop as it provided them with the opportunity to: get senior
executives attention and involvement in understanding, adopting and implementing health literacy practices; becoming
knowledgeable about their staff commitments to health literacy; demonstrate their support for their staff attending the course;
and to demonstrate organisational accountability to investing and supporting staff to attend the health literacy professional
development course.
• “ Yeah, absolutely. I mean it was - it gave me a couple of hours where I was basically able to sit down and have a look at
what's being done, some of the projects, some of the people involved. So it was a really good way of, number one,
getting my attention, and number two, for me to actually take away a really good idea of what the program's actually
delivering, not only for my guys, because I know that, but more broadly …So if there was a choice of whether to do it like
that again I would say most definitely. I think it's great. I also think we need to be a little bit accountable and I think
that…”. HL ES 6
• “ I always find those forums really interesting from a sharing point of view, particularly if there are multiple agencies
there and I always find those discussions really engaging on a personal level. I also think that it's incredibly important
for executive sponsors to know exactly what their staff are involved in. HL ES 8
3.1.6 2014 Health Literacy Course Facilitators Interviews
One interview was conducted with the 2014 course facilitators to explore experiences of facilitating the course. The interview
revealed an array of themes that have been clustered into four areas: 1) Health Literacy Course facilitation; 2) Health Literacy
Course Governance; and 3) Health Literacy Course Participants. A summary of key findings are presented below with illustrative
quotes.
1) Health Literacy Course Facilitation
Given that the 2014 Health Literacy Course was designed to build the capacity of agencies to respond to health literacy at both a
client and an organisational level, the course facilitators emphasized that there role was as ‘facilitators and educators’ and not
trainers’, providing tools and frameworks to apply in their workplaces, as can be seen below:
• “I don't particularly mind what we were called here. But I tend to think more - I have a bias or a preference to be more
of a facilitator and educator than a trainer… The idea is to, I guess, give some people some tools and frameworks and a
system around where they could use it but they do that application, rather than saying this is the way you do it and you
get assessed in that…” HL CT-2
2) Health Literacy Course Governance
26
Course facilitators also commented about the linkage and interactions between the health literacy leadership group and
themselves- requesting that more interaction would have been beneficial all round.
• “… there felt like a disconnect between the leadership group and the trainers… Well, it would have minimised
duplication and going down a path and then going oh, oh, it would have been nice to know that a while ago. I would
have provided opportunities to discuss pivotal points in relation to content and some of the issues that could have been
easily resolved. …” HL CT-1
• “..we're operationalising [the course]. We're not linked.... Isn't it interesting …oh yeah, the leadership group. Haven't
thought about them. There's been no contact. They've not been on my horizon…”HL CT-2.
3) Health Literacy Course Participants
Course facilitators reflected upon the differences between the 2013 and 2014 course participants, with 2014 participants
being more respectful and enthusiast, and how the context has dramatically changed, with health literacy now on most
organisations' radars, as seen below:
• Last year they were just - it was like they were sitting there last year with their arms folded like all right, come on. Show
us what you've got. That was really hard to work with. I think the - so I've often said that the difference between last
year and this year is the context is so dramatically changed. Health literacy is on most organisations' radars now.
There's been enough movement that the cynicism has settled down a little bit. So I think they're the big things. But I also
do think there's something to be said for personalities. I just think last year the critical mass was the other way. Whereas
this year the critical mass was enthusiasm and very deeply respectful group. Respectful towards each other and towards
us.
HL CT-1
Course facilitators also commented upon the need to more explicitly articulate expectations about course participants
qualities, skills, positions or networks that are required to lead or facilitate change within their workplaces.
• “So what are organisation’s expectations around sending two people there? …I need to ask that question here
internally a bit more around what we think are the expectations because I think - I don't think I've articulated that for
myself, that discussion. HL CT-2
• this is the purpose of the course and we believe that the best outcome would be if you recruit or send along a person
that has these attributes and has this role and function or influence in the organisation. I think - and we just
articulate… we need to just say right up front, that this is an expectation and this is what a change agent with the
qualities or kind of networks or position would happen to facilitate that” HL CT-2
• “… it is about having that authority to facilitate or lead change within an organisation - you know, within reason,
whether they - that was by virtue of their positions, it was their personality or whatever it was. HL CT-1
27
3.2 Health Literacy Community of Practice
This section provides evaluation findings from the interviews with the CoP Leadership group, CoP participants, 2014 Course
participants, 2014 senior executive sponsors, and the 2014 course facilitators
3.2.1 2014 Community of Practice Leadership Group
Four members of the CoP Leadership Group were invited to participate in a semi-structured interview about the health literacy
Community of Practice (CoP). A summary of key findings are presented below with illustrative quotes:
Perceived purpose
The CoP leadership group were asked about the intended purpose of the CoP. Responses ranged from:
• the CoP was for like-minded practitioners who know each other, have a strong identity with the group and participate to
share ideas, experiences and to learn together;
• “it's about, again, like-minded individuals that have an interest in health literacy who can hopefully put it into practice into
the organisations” HL CoPL-4
• “Well I see it as a group of people that come together to discuss issues around a particular topic, so there's like common
interest amongst the group members to discuss particular issues, to engage in dialogue on particular issues and in this case,
health literacy. So yeah, community of practice is that group of people or practitioners coming together to engage in
discussion around health literacy. That's my understanding of community of practice” HL CoPL-1
When asked for their views about whether it mattered if it was a CoP or a Network, interviewees had mixed opinions. Several
commented that a CoP was for problem solving and practice development as compared to a Network meeting which was for
mainly knowledge transfer and exchange.
Implementation of Community of Practice
The CoP leadership group were asked about the extent to which the CoP had been implemented as intended. Interviewees
commented both upon the actual implementation and key enablers and barriers to implementation.
• the CoP is in its iterative phase
“I think we're learning along the way. There's a lot of learnings along the way, I think we're finding our way, I don't believe we've
got it right yet” HL CoPL-4
• a dynamic tension exists between the Leadership Group delivering on behalf of members versus facilitating members i.e.,
Community / organically driven Vs Leadership Group driven
• “…even though we're the Leadership Group are we the right ones to be running it?” HL CoPL1
• need to balance being inclusive with clarity of purpose
• “So when I was looking at who's invited, it started to make me laugh, because it's beautiful. It's so inclusive, it's like I don’t
know what you'd have to do to not be able to come. So managers, clinicians, health promotion staff, participants, graduates,
consumers, all diverse staff. Academics, researchers, representatives, everyone can come. That's a really lovely principle, but
I think in the inclusive, I think they've traded off a bit of clarity and I think that could be part of the problem. Because all of
those people actually don’t share in us, just because they have an interest in health literacy” HL CoPL-2
• need to create and maintain momentum between members between CoP events
• “Yeah, look I think we didn't get the momentum happening from that first one and I was quite keen for that to happen…I
think we should have had that date out a lot earlier” HL CoPL-1
28
• opportunity for a consumer voice now exists
• “The other thing too is that we have two consumer representatives, because we thought when we first started that was one
thing we identified that we needed to hear the consumer voice, to just mix - so we have two consumer representatives,
which I think is a great thing to have” HL CoPL-2
Key enablers to implementing the CoP included: riding the crest of a wave; commitment from leadership organisations and
leadership group; investment in a committed Coordinator for CoP; dedicated resources to health literacy; and increasing
dialogue amongst practitioners;
• “It was like riding the crest of a wave or something, that health literacy had somehow apparently been discovered by a
whole lot of people in various agencies. All of a sudden, somehow, there was a great momentum that they were talking
about” Hl CoPL-3
Key barriers to implementing the CoP included: membership of CoP is all/ too inclusive – no defined boundaries; CoP Region Vs
state focus; CoP seen as Led not facilitated; No sense of shared identity / ownership among members yet! Limited interactions/
communications between CoP event; limited time to participate in CoP meetings (esp for clinicians); level of org commitment &
health literacy being a priority
• “I just think it's an attitudinal issue as well. I think it depends on how much of a priority it is for the organisation, how
much value is placed on health literacy within an organisation. I guess if it's a priority, then it should reflect - there has
to be an organisation wide commitment to ensuring that staff are supported, if they want to attend, if they want to
increase their knowledge about these issues, then they have to be supported. If they are at that higher level, there has
to be buy-in from management” HL CoPL-2
• “Lack of time to actually participate in the meetings, so especially those from the clinical world. I think it's very difficult
to create time to participate in these events. It becomes very difficult also for senior management, for example, to allow
staff to be part of the - for the same reasons, I guess, time issues” H CoPL-2
Perceived Impacts
The CoP leadership group were asked about the perceived impacts of the CoP so far. Responses ranged from:
• Sharing and learning from experiences, knowledge, and stories
• Consumer participation in CoP potentially leading to improved understanding of client needs, improve quality of service
provision and ‘client centred care’
• creation of health literate settings – “making it possible for clients to access and use health information”
Sustainability of CoP
The CoP leadership group were asked about whether the current health literacy CoP was sustainable and what enablers and
barriers may influence its sustainability. Responses ranged from:
• need to revise and re-align leadership group roles to one that facilitates members
• “It does feel a bit over-governed, in a sense, for what it is. But I think that that leadership group could see its role differently
and not see its role as to deliver on behalf of the group or - but really facilitate the group doing the doing. I just think a great
session would be us not planning anything at all, except maybe booking a venue. Maybe there's a theme, I don’t know”
HL CoP L 2
• sustaining the CoP will require broader perspectives from service providers, service planners from health (community
health, primary care and acute care) and non-health (local councils, non-government organisations)
• “I sometimes wonder if, even though it's a great group, do we need to extend that group to get perhaps different
perspectives or the right drivers and leaders in the room” HL CoPL-4
29
• greater buy-in and commitment is required from senior management of organisations to support staff participating and
contributing in and to the CoP
• “I think it's quite sustainable, I just need to - it just needs to be an organisation wide priority and that need from buy-in from
senior management of organisations, I think that's how it can be sustained, so using that commitment from higher levels or
organisations. Also, perhaps making it policy, health literacy policies in organisations, something that is binding for…”
HL CoPL-2
• practitioners need to have access to more modal options (eg., face to face meetings, on-line forums) to participate and
contribute
• “So the more options people have of participating and contributing to discussions, the more sustainable it's going to be. So I
think all options should be left open. If people have other options as well, I think people should be allowed to nominate which
options work for them and also provide their suggestions on how best they can be operationalised, I think” HLCoPL-2
3.2.2 2014 Community of Practice Participants
A purposive sample of 14 Community of Practice participants were identified from the list of those who participated in the first
two CoP events. All 14 were sent an invitation to participate in a semi-structured interview. Of these 13 agreed to be
interviewed. A summary of key findings are presented below with illustrative quotes
Perceived purpose
The CoP participants were asked about the intended purpose of the CoP. Responses ranged from
• bringing together community professionals, to creating a benchmark on current understanding in the professional and
community context and to reflect upon future challenges to health literacy
• “My impression of the community practice as a group of clinicians working towards a theme. We've got a theme and we're
working towards an outcome. So I imagined that this community practice is going to work towards an outcome but I'm not
quite sure that I've identified what the outcome is, whether it's for the individual agencies that were represented there to be
able to bring something back to the community at the end of the day, or whether it's more around information sharing and
best practice and ideas” HL CoPP-2
• “I wanted to find practical ways on the ground where in a - in communities extended from Melton to the border, rural and
remote, how I could have some impact in the engaging consumers and in so doing, carry out activities X, Y and Z so as to
improve their health literacy” HL CoPP-8
• “So my understanding of Community of Practice was people that were doing like work - so some thematic I suppose overlap
in their work; coming together to share practice experiences and yeah I suppose discuss any difficulties they might be having
or blocks they may be having and so problem solving together about moving forward - so sharing knowledge and practice”
HL CoPP-6
• “So to me probably it was still - like I had joined this Community of Practice with the view that I hoped to participate in an
ongoing way. So for me I saw the first one as a bit of a coming together; getting a sense of who else is participating in the
group. So I think for me it's just very early days. I'm looking forward to the next ones just to see how - but I imagine it will
just - like there will be more connections between the participants. It will be a little bit more - what's the word? I'm trying to
avoid using the word coherent again - but yeah, that…”HL CoPP-6
When asked for their views about whether it mattered if it was a CoP or a Network, interviewees had mixed opinions- the
predominant view was – what matters is that is enables sharing and reflective learning of ideas, strategies - professional
development.
30
• “I don't think it does, to be honest, because to me, I'm always focused on, what do we get out of it? Not what we call
things It's really, again, to share ideas, to see what other sides there were in this space and to try and pick up if there are
any strategies or tips or just anything that may be useful in terms of where we are at in our journey” HL CoPP=1
Implementation of Community of Practice
The CoP participants were asked about the extent to which the CoP had been implemented as intended. Interviewees
commented both upon the actual implementation and key enablers and barriers to implementation. Comments ranged from:
• no connection to other participant s yet!, “we do not know them” ie CoP runs like a Network, with a strong sense of
professional development;
• “I think our network would be easier to understand. Community of Practice I would never - I didn't have any idea of what
that meant, but if it had been called a network - health literacy network or something like that well then I would have had a
better understanding of what it was about. … Well I guess it'd be like learning from others about their experiences and being
able to openly share your own issues maybe that you had or were having, or give them ideas. Yeah, whereas Community of
Practice I wasn't really sure what it was I was getting into” HL CoPP-7
• mixed opinions about meeting expectation (e.g., expected to get information about Health Literacy policy but just got more
confused);
• “…perhaps a little bit more focused as well, because I think the first one it was a lot of scoping about what people were
hoping to get from the Community of Practice. So I suppose the focus will change a little bit more to - the focus of the
discussions and activities will change a little bit more to developing whatever I suppose it is; the skill, the practice, yeah
moving on to problem solving aspects”. HL CoPP-6
• “For me even the word community of practice - because it's so new to literature. It actually makes people think twice about
what does this really mean, what is a community of practice. For me, I think there needs to be something much more
productive. The whole name of the group has to be a lot more productive, like perhaps a health literacy working group” HL
CoPP-5
• No follow-up occurred after the CoP – or not yet! (eg., summary of discussions or actions);
• mixed opinions about workshop formats (eg., did not engage nor get into depth with regard to topics);
• lack of clarity about link or continuity between the 1st
and 2nd CoP.
Key Barriers to implementation included:
• Health Literacy is broader than the Western region - how to work across boundaries;
• limited time;
• how to retain continuity from meeting to meeting?;
• no clear member accountability – no clear Plans/ KPIs;
• need to demonstrate value; and
• “It's just competing priorities and look to be honest, because I'm time poor, as many health professionals are - time poor
- as long as it stays relevant to what I'm doing it will, it'll be a higher priority. As soon as the relevance of the group is
less or the focus changes to something that's less relevant to me - it might be relevant to other members, but less
relevant to me - then [unclear] drop down the priority list; like with my time. HL CoPP-6
Key enablers to implementation
• positive supportive organisations
• interest from wide range of professionals and organisations in health literacy
31
• “So the positive is that health literacy is of interest to a wide range of professionals in a wide range of contexts. The
challenge is retaining continuity from one meeting to the next around who's in the room” HL CoPP-2
• “I think you've got your passionate few who are in there waving the flag but if the organisation doesn’t see a benefit or an
outcome they're not going to support those people to remain in the community of practice potentially”. HL CoPP-1)
Perceived Impacts of CoP
The CoP participants were asked about the perceived impacts of the CoP so far. Responses ranged from
• great initiative but matter of resources;
• not sure, as our organisations are really supportive but need to see benefits - outcomes;
• it benefits the champions, as it facilitates ground swell support to share stories, success and tools
• “My memory serves me is it challenged some of my thinking”. HL CoPP-5
• “Knowledge, not only for myself but for my team. Because I manage a team of case management staff and administration
staff. So gaining more information about what is out in the community and how we can use it HL CoPP- 4
• “ …I saw the first [CoP] as a bit of a coming together; getting a sense of who else is participating in the group. So I think for
me it's just very early days” HL COPP-6
Sustainability of the CoP
The CoP participants were asked about whether the current health literacy CoP was sustainable and what enablers and barriers
may influence its sustainability. Responses ranged from:
• need to clearly articulate purpose of CoP to demonstrate benefits
• “I think when framing the purposes of the community of practice, that needs to be articulated - consideration for how that
can be articulated in such a way that practitioners can use that to go back to their managers or to their organisations and
saying, this is why I go and this is what I believe I'm getting out of it”. HL COPP -2
• requires organization buy-in
• requires online forums could enable more information sharing;
• may depend on topics and time available – due to competing interest; requires greater emphasis on skill development to be
sustainable
• “Yeah, but I think, again, it would depend on specific topics. It's simply the time. If we all had all the time, I love going into
those things, but I think it would probably depend a little bit on being selfish and thinking, does this really relate to where I'm
at? Unfortunately, it's competing interests”. HL CoPP-1
• need to make members accountable with clear plans / Key Performance Indicators.
• “There needs to be a plan on how long it's going to take before we start looking - you know, like everything else that we do,
unless you have a plan and stuff like that it just becomes ongoing and never ending which is makes it unproductive. This
means people are meeting or the sake of meeting and nothing is coming out of it. I'd much rather go in there with a set plan
and know exactly what is due when and how that's going to be fed back and when that's going to be measured” HL CoPP- 5
3.2.3 2014 Course Participants CoP comments
All 2014 Course participants that were interviewed (n-=14) were asked about their knowledge, participation and perceived
impact of the health literacy CoP.
32
All course participants were aware that the health literacy CoP existed. However, most had not participated due to various
reasons including: increased workload demands (e.g., accreditation); timing of the CoP did not align with working days;
competing priorities and commitments; and having limited time availability.
• “Yes aware of it ...but too busy” HL2-7
• “Yes aware but can’t come on Mondays” HL2-8
• “Yes knew about CoP but can’t participate as do not work on Mondays” HL2-9
• Yes heard about it but capacity limited” HL2-11
• “Yes aware of CoP but too many other commitments” HL2-14
• “Look I wouldn't say not interested just as I guess competing - I don't even know if I'd say interest just...busy, crazy time -
yeah priorities - like accreditation is a priority. We're doing crazy hours at work”. HL2-1
• Look, I think [CoP] is a great idea, but the difficulty for me is that from now it's not part of my core workload. I think that for
me I would love to continue to go to like a meeting every two to three months, whatever it is, but it's the time and I think
that my manager is already probably getting a little bit touchy with me doing so much health literacy when I don’t have
allocated time, so I think as of next year, I think that will be really difficult for me to continue to do a lot of work in the area”.
HL2-3
• “I look at it and give it a thought and then I move on and do something else. Look I don't know. I think so. I think the only
thing is, is that sometimes I read that and I'm thinking about people talking more around health promotion stuff. Really we
don't - well in my role here I don't have capacity to do more broadly health promotion”. HL2-4
• “Well, I do know. I've seen the emails coming through, but no, I haven’t.. It's probably time for me, being able to take the
time to do that is a challenge… Look, if the community of practice runs next year, then I would do it, but I can't fit the two in
at the same time, if you know what I mean?”. HL2-7
The course participants who had participated in the CoP commented positively.
• “…I’ve been attending those and found them quite useful.” HL2-5
One course participant also commented that it was not clear how the CoP related to the course.
• “Not sure how CoP relates to course” HL2-10
3.2.4 Executive Sponsors CoP comments
All the executive sponsors (n=10) were also asked about their knowledge, participation and perceived impact of the CoP. Overall
executives had variable and limited knowledge about the occurrence of the CoP. Overall they were supportive.
• “In the community practice, I don’t think so. I get bombarded with a lot of stuff, but not that I'm aware of. But maybe I
am missing something” HL ES2
33
3.2.5 2014 Course Facilitators
Overall the course facilitators were positive about the opportunities that the Health Literacy CoP presented for the past and
current course participants as seen below:
• “…in theory I think it's a great idea because I think there needs to be a place where people who have done this work
who might not be able to have these conversations within the workplace or need some nourishment or exchange of
ideas with others who have done the course who are interested parties where they can come together”. HL CT-2
3.3 The Ripple Effect in the West
A total of 10 interviews were conducted with 2013 course participants and five interviews with their organisation executive
sponsors. On the basis of these interviews, stories of change’ using the adapted Most Significant Change technique were
developed (see Appendix 3 for nine stories).
Overall the stories reveal how participation in the 2013 health literacy course, as a capacity building initiative has acted as
“catalyst for action” for individual participants, their organisations and beyond, resulting in incremental transformation in
knowledge, attitudes, intentions and behaviours of course participants. The stories also provide a mechanism to contextualise –
individual and organisational changes. The stories also reveal that ‘transfer of training’ defined as the generalisation and
maintenance of knowledge and skills gained via training17
has occurred, particularly with course participants who have changed
employment during the course. Furthermore, the stories also reveal that Training – defined as the systematic acquisition of
knowledge, skills, and attitudes has resulted in Training outputs, namely learning and retention. The stories also confirm that
the courses are continuing to build capacity as demonstrated by: developing leadership, workforce knowledge and skills,
organisational infrastructure, resources and networks amongst the course participants.
The interviews were specifically designed to assist interviewees to think back before, during and after the 2013 health literacy
course
In responses to being questioned about what awareness or knowledge course participants or their organisation had about
health literacy before the 2013 Health Literacy Course, a spectrum of views emerged. Course participants mentioned having
limited to good knowledge, as can be seen below, as well as the importance of having support and engagement with health
literacy from their organisation.
• “No, before I started I had no idea, absolutely no idea, and now I'm leading the health literacy work at [organisation].
So that in itself has been a massive change But some of the things that have facilitated that, I guess, is that the
[organisation] at the time took a lead role in the development of the course with [CEH] and so we have, and still have,
lots of that support from high level management. It now will become part of our work”. 2013 Participant HL MSC 1
• “I think it was pretty good. I think it was pretty good because I come from a community health background and had
done a lot of work with culturally diverse populations so it was very much on our agenda as an organisation. Although I
don't know that we did it very well with signage and literature we provided for patients so we can always do better. But
yeah I had a fairly acute awareness of those things”. 2013 HL Participant MSC 5
In responses to being questioned about what was the most significant thing that happened to them and in your organisation
during the 2013 health literacy course, a spectrum of views also emerged. Course participants mentioned: building their
networks; increasing their knowledge, recognising that the course was ‘an agent of change’;’
1
• “…to me personally I think building those networks with the others in the course - I think that was really useful for me. I
did meet people through that course who I hadn't previously met here at [organisation]” 2013 HL Participant MSC 2
• Yes, definitely the knowledge. I think that was the most important part of the course… understanding just how broad
this category was and how fundamental it was to health in general through a variety of mediums. But I think the other
important part of the course was it was really an agent for change, so those organisations that were involved, obviously,
17
34
I had dealings with from a [organisation] perspective. It really banded us together to, for example, do a response to the
Australian council for Workforce Health and Safety health literacy submission. 2013 Participant MSC 7
When senior executives were questioned about what was the most significant thing that happened during the 2013 health
literacy course, a spectrum of views also emerged. Senior executives reported: the course provided the opportunity for
participants to “champion” health literacy by applying and sharing learnings with colleagues and service users;
• “Certainly, there was more visibility through the course. Also, though, I think that, as the course aspired to those
participants - and it was certainly true of the participant that we had within the course - became internal champions. So
the knowledge that was gained, I guess, through the course, was coming back into the organisation and being shared,
predominantly through the projects initially. It gave a really practical opportunity to apply learning and in that
application, you're automatically sharing it with colleagues and service users. So for me, that championing type concept
sort of occurred immediately. I think that helped to increase knowledge and understanding”. 2013 Senior Executive 2
In response to being questioned about what was the most significant change that occurred in them or their organisation in
relation to health literacy practices since the 2013 Health Literacy Course, course participants mentioned: being authorised to
lead or facilitate change and broadening their networks as two examples.
• “I think having done the course it's given me permission to say well actually I do know something about this as well so
it's actually reinforced for me that I have got a knowledge base to clarify..” 2013 HL Participant MSC 5
• “it's just broadened my networks, and I feel pretty happy that I've got a good network of people who understand health
literacy that I can call upon now. So that was probably the most significant for me”. 2013 Participant MSC 2
The interview with a course participant who had changed employment during the course provides evidence of a where the
‘ripple effect’ was happening – the course: changed their and their colleagues approach to work”; influenced their choice of new
job; and influenced their role in their new job.
• “Yes, for sure. So being involved in the course, the way it changed my approach to work and several others that I was
working with was a real consideration of the information that we were presenting to health practitioners to clarify, for
example, processes. How e-health was presented to patients? What benefits were they getting out of it? …But yes, it
really is an interest of mine now. I think in a way it [course] influenced the choice of job that I took after leaving
[organisation1], because [organisation 2] and the service that [organisation 2] has developed through internet is really a
frontline tool. So it does impact practitioners, it does directly impact consumers, or clients of health care services, and
also Allied Health, so it’s got that broad spectrum usage and I think I want to have more of a role in our company in just
making sure that it is very easily usable and it’s a real tool for health literacy in those particular groups”
2013 Participant MSC 7
Interviews with senior executives about what was the most significant change that occurred since the 2013 Health Literacy
Course, revealed multiple examples of how health literacy had become embedded into other organisation activities and
structures.
• “We’ve taken health literacy into the WING network, which is the intake network for the west in Melbourne, so all the people
who do intake in the west come together. We've trained or done a professional development session with that group…We’re
just about to put a health literacy position statement up to the board. 2013 Senior Executive 1
• “ main changes for us was that we've embraced an organisation wide commitment to health literacy through the
development and adoption of a policy and some procedures falling out of that. The impact of those procedures and
guidelines I think are yet to be declared”2013 Senior Executive 8
35
Another senior executive commented on how their organisation recognised how the course had developed “health literacy
champions” and how they were providing secondary consultations (from a health literacy context) within their organisation.
• Yeah. I think that, certainly, with embedding it in integrated health promotion and also having a champion, or now three
champions, within our team, we are recognised for that internally. We are used as almost like a secondary consult across
the organisation. So, certainly, people like [A, B, C] are called upon to review a document that dental health were
developing. A pictorial sort of document for their clients, around tooth care after an extraction. So in very practical
ways, they're called on to say, we've developed - we've seen a need to develop a more plain English, visual document for
our clients. Could you review this in a health literacy context? So, yeah that...2013 Senior Executive 2
3.4 Evaluation Participant Feedback Workshop
A workshop was conducted on the 11th
December, 2014 to: present emerging themes from the evaluation; to create an
opportunity to discuss, confirm, or redefine the themes; and to gain further insights into the experiences of the Health Literacy
Course participants and their organisations. All evaluation participants from across 2013 and the 2014 courses, the Health
Literacy Community of Practice and Organisation Executives workshop were invited. A total of eight evaluation participants
attended the workshop. A Draft Summary Report (Appendix 6) was also sent to all evaluation participants who were not able to
attend for their feedback. No further comments were received. The following section provides an overview of key themes that
emerged from the workshop.
Overall workshop participants commented that they valued the opportunity to hear about and discuss the key evaluation
findings and emerging propositions. Overall the key emerging findings and propositions resonated with participants experiences
and resulted in lively discussions far beyond the health literacy development course and the health literacy CoP.
In response to the overall evaluation findings, key emerging themes included:
• Conceptual underpinning design of the course: Participants emphasised that it was important to recognise that the
health literacy course was designed to facilitate health literacy approaches and not just implementing health literacy
manuals, frameworks or tools.
• Iterative investments in health literacy: Participants commented that it was important to recognise that the health
literacy initiatives (e.g., CoP) were in there infancy and in ongoing development and refinement phases in response to
participant feedback. In response to the evaluation findings that the CoP may not have been ‘fit for purpose’ based
upon CoP literature, workshop participants commented that as professionals were attending in substantial numbers
(from 30 – 60 participants), this indicated a local need for health literacy knowledge transfer and exchange and
networking.
• Health literacy agents of changes: The concept of ‘agents of change’ resonated with participants. One participant
commented that they recognised the difference between participants who had and did not have authority to lead or
facilitate change. Several participants also raised the question- Who gives authority to staff to lead or facilitate change
within organisations? One participant further commented that position-based versus person based leadership had
implications for the level of confidence participants had in sharing issues, problems and ideas in group based events,
such as the CoP – thus there was need to create or facilitate an authorising environment for participants to share
issues, problems and ideas. Workshop participants commented that the 2014 participants were different from the
2013 participants – they were the right people for the course. Another participant suggested having the course
participant’s line managers engage in being agents of change as well – so that course participants did not act in
isolation- Executives also needed to be part of the transformative change process. The WIN (Workforce Innovation
Network) model was mentioned. Another participant suggested having in writing that course participants were given
authority to not just implement projects but to lead or facilitate change in organisations – to endorse / get sign off.
Another participant suggested that the sponsoring organisations have key performance indicators for course
participants and not the Course – as this would further facilitate course participants being accountable. Discussion
also occurred that health literacy was not to be seen only as a professional development event but as a core role and
responsibility – this led onto discussions about the need to recognise multiple forms of literacy – intersectoral literacy;
and re-emphasised health literacy as an approach not a tool or simple act (e.g., translate a brochure). Others
commented on the need to reflect upon the next level of course – building organisational level capacity – not just of
individual – as there was a tipping / saturation point – just like in quality improvement – where everyone talks about
health literacy. Several participants also commented on the need to ‘not forget about consumers involvement’ in the
health literacy initiatives.
36
• Demonstrating value of investing in Health Literacy: One participant commented that it was important to recognise
the need to have ‘leap of faith’ as a precursor to demonstrating the monetary value of the health literacy initiatives –
and cited other areas e.g., Place-based projects. Another participant commented about the need to view health
literacy within a health and well-being frame – and not just ‘health’ – as there organisations approach was to embed
health literacy as a capability approach into work roles and responsibilities. Several participants cautioned that the
term ‘health literacy’ may be an issue - a barrier. Several participants also commented on the need to reflect on what
is the motivation to focus on health literacy and the need to explore social return on investment (RoI) and not just
monetary RoI. One participant also mentioned that the Australian Commission on Quality and Safety – had released
quality indicators that were useful to explore. Another participant pointed out that the transfer of training was a key
indicator of the ROI.
• Building the evidence: In response to the need to build evidence demonstrating the value of the health literacy
initiatives discussion occurred regarding what types of evaluative approaches was required. Overall participants
endorsed the need to focus on: developing and piloting an outcomes framework; the need to research drivers and
longitudinally self-sustaining health literacy practice. One participant also commented on the need to “dig deeper” –
into the organisational changes resulting from staff participating in the health literacy initiatives – expand the
individual most significant stories of change. Another participant mentioned the need to map the types and qualities
of relationships (maybe via Social Network Analysis- SNA) that was creating a ripple effect. One participant also
commented that we needed to remind ourselves that the health literacy course, CoP and evaluation were all ‘ground-
breaking‘ and the need to build a bigger profile for it all.
4. Synthesis and Discussion of Evaluation Findings
On the basis of 62 interviews overall the evaluation has revealed that the health literacy professional development initiatives are
creating a ripple effect and building health literacy capability at the individual, organisational, regional and systems level.
Synthesis of key evaluation findings are presented under the three evaluation foci: 2014 Health Literacy Course; Health Literacy
Community of Practice; and the Ripple effect in the West.
4.1. The 2014 Health Literacy Training Course:
Course content: The 2014 course content was perceived by course participants as more comprehensive, applied, contextualised,
responsive, adaptive and ‘how-to’ practically oriented. Overall participants commented that the content had increased the
breadth and depth of knowledge about health literacy as not solely an individual trait, but that organisations have a key role to
play. While participants recognised the need to focus upon organisational health literacy, there were some concerns, whether
there was too much content and emphasis on health literate organisations - at the expense of content on the health literacy of
patients. Participants who had limited authority to facilitate or lead change within their organisations (e.g., Project Officers,
Front-line staff) also commented that the transfer of training course content was not relevant, and at the expense of health
literacy specific content. Participants also commented that information related to the mini-projects requirements (e.g., to be
conducted with partners and scope (e.g., 40 hours) needed to happen earlier than Module 1 to enable better planning.
Participants valued presentations from course alumni.
Course structure: The four modules and two mini-project course structure was positively received by course participants, as it:
enabled reflection; putting concepts into practice; provided immediate benefits (particularly the mini-projects), and facilitated
mainly intra-organisational networking opportunities. Several participants commented that the course could have been
compressed into fewer days or even having half-days, to keep the momentum up. Several participants commented that
activities within the course modules could have been structured to further facilitate inter-organisational networking.
Participants also suggested building in place-based visits for course participants to organisations already implementing health
literacy practices – further demonstrating leadership.
Course participant composition: Course participants were positive about having a mix of course participants from diverse
sectors, organisations and roles, as it provided opportunities to hear and discuss other viewpoints, differing challenges and
opportunities that exist. The pairing of participants from the sponsoring organisation was viewed positively by both
participants and course facilitators – as it created a platform for sharing learnings, problems and generating solutions. Several
participants also commented that pairing of organisations could further facilitate and optimise shared learning. Several
participants commented that the course composition mix could be further broadened to include non-health, private, businesses
and insurance based professionals as this reflected the “real” state of care provision.
37
Overall comments by the 2014 course participants’ revealed that the course developers and facilitators have learnt from,
utilised and embraced the 2013 evaluation findings, particularly for the course to become more how to / practically oriented,
contextualized, responsive and adaptive with examples of implementing health literacy concepts into practice. This finding also
provides evidence that the participatory evaluation approach was appropriate and useful, and that the working partnership
between the course developers and the evaluator led to increased evaluation learnings utilisation, and fostered a culture of
learning between all.
The health literacy course was designed to support participants facilitate change at the individual and organisation level – with
course participants being ‘agents of change’. However, the evaluation findings suggest that there were mixed outcomes in
participants leading or facilitating change upon their return to their organisations. These findings require some discussion.
Firstly, the evaluation revealed that course participants who were in Project Officer and Front-line roles (as compared with
managerial, educational or decision-making roles) perceived themselves as having limited authority to facilitate or lead change
within their organisations. While this was not a universal finding, it raises several issues. Given that health literacy is everyone’s
responsibility, does this finding suggest the need to further explore the characteristics or qualities required to be a health
literacy change agent? Alternatively, given that course participants were recruited / nominated / chosen by their own
organisation (and not the course or Health Literacy Steering group), does this finding suggest the need to review the selection
process / criteria organisations are using to nominate staff to undertake the course and become health literacy change agents?
Secondly overall the evaluation has revealed once again that a core set of enablers and barriers exist at the individual,
organisation; and system level to implementing health literacy practices. This finding further highlights that a myriad of factors
can influence the success of professional development courses as a transformative change process18
. Future course may
consider how to further involve course participant alumni and their organisation to mentor or reveal how such factors can be
addressed to optimise the implementation of health literacy practices
Organisation Executive Sponsors workshop
The evaluation revealed that the Organisation Executive Sponsors workshop was overall well received by course participants and
senior executives. It provided the opportunity to: get senior executives attention and involvement in health literacy; to become
knowledgeable about their staff commitments to health literacy; to demonstrate to their staff their support to implement health
literacy practices; to become knowledgeable about the challenges facing other organisations in the Western region committed
to health literacy action; and to demonstrate accountability of their organisations investment in health literacy. This finding also
provides evidence of the utilization of the 2013 evaluation findings – as a key finding was the importance for course participants
to have engagement from their senior executives sponsors and creating a supportive enabling and authorising environment for
course participants to implement learnings from the course and to lead or facilitate changes in health literacy practices. This
finding also confirms the appropriateness of involving senior executives in the evaluation process, as it provided rich insights
into the multiple drivers for engaging in health literacy action and importance of maintaining momentum.
4.2 2014 Health Literacy Community of Practice (CoP)
The evaluation has revealed that the CoP is contributing to the health literacy ripple effect in the Western region – as
demonstrated by the evidence that the three CoP events were attended by a large number of professionals who had not
participated in either the 2013 or 2014 courses. The CoP events have also revealed the increased visibility of commitment to
investing in health literacy in the West. The CoP has created a novel and local space for the sharing, reflecting and learning of
experiences, particularly from consumers. The CoP also provided a structure for 2013 course alumni to take up health literacy
leadership roles (e.g., Health Literacy CoP Leadership Group).
While the health literacy CoP does appear to be contributing to building health literacy capacity and a ripple effect, comments
by CoP participants and CoP Leadership group about whether the current CoP is ‘fit for purpose’ require discussion.
Communities of Practice have been defined as “Groups of people who share a concern, a set of problems, or a passion about a
topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis19
(Wenger, McDermott &
Snyder, 2002, p4). Furthermore, CoP’s exemplify three key characteristics:
1. Domain – an identity that is defined by a shared domain of interest
2. Community – members engage in joint activities and build relationships - imply shared interest
18
Mezirow J, and Associates (2000). Learning as Transformation: Critical Perspectives on a Theory in Progress. San Francisco, CA: Jossey–Bass.
19
Wenger, E; McDermott, R & Snyder, W.M (2002) Cultivating Communities of Practice, Boston, MASS, Harvard Business School Press
38
3. The practice- members are practitioners – who develop a shared repertoire of resources, experiences, stories, tools ways of
addressing recurring problems – develop a shared practice-that takes time & sustained interaction
However, the evaluation has revealed that the CoP has: multi-purposes, an all inclusive membership base, and is designed for all
circumstances, and lacks a sense of shared identity, ownership or “us” amongst its members.
Interviews with 2014 Course participants also revealed that the CoP appeared to be competing for participants time - indicating
a possible ‘ceiling effect’ on how much time professionals can spend on professional development activities, especially when
staff do not have dedicated time/resources allocated to health literacy – even when organizational management are generally
supportive.
Interviews with the CoP Leadership Group revealed that the CoP was being run as an organisation driven knowledge transfer
and exchange event-based network rather than CoP with a group of people who are learning, innovating & interacting together
on an ongoing basis.
Thus the current health literacy CoP may not be ‘fit for purpose’. In other words, the current novelty and local appeal of the
health literacy CoP may not sustain engagement – thus consideration needs to be given if it is to remain an ‘organisation driven
knowledge transfer and exchange event-based network’ versus a ‘group of people who learn, innovate and interact together on
an ongoing basis’
At a time of ongoing sector and organisational reforms and restructures, increasing workloads and fatigue, there is a need to
reflect upon several key questions including:
• What is the Health Literacy CoP intended to achieve?
• Does the Health Literacy CoP have the essential CoP characteristics?
• Is the CoP facilitating learning and innovation together in trust-based relationships that make things happen between
the cracks in the systems?
• Can CoP members be more accountable via explicit plans and Key Performance Indicators?
• For whom is the CoP designed for?
• Is the CoP designed for Course Participants, Service Providers, Managers, Senior Executives – all with differing know-
how and decision-making & agent of change status
• Is the HL CoP creating a safe space, trust, openness & honesty, creating a sense of obligation, giving new perspectives;
providing peer support and creating a space to be inspired ?
• For what circumstances is the CoP designed for?
• Is there room for a topic (e.g., mental health) or setting (e.g., hospital) specific CoP?
• Is there a need to focus equally on health literacy at a patient and organisation?
• What structure fits the purpose?
• Does the CoP need someone/organisation to lead it, manage it or facilitate?
• Does the CoP need to be multi-modal (F2F, online)?
• Is the current CoP structure promoting a self-sustaining independent health literacy structure?
• How many CoPs is enough and should the CoP be ongoing or have fixed time span?
4.3 The Ripple effect in the West
Overall the evaluation findings suggest that the health literacy professional development initiatives are creating a ripple effect
and building health literacy capability at the individual, organisational, regional and systems level.
Interviews with 2013 course participants and their organization senior executives revealed that the initiatives were creating an
initial splash (i.e., outputs) – leading to changes at the individual level (e.g., increased participants knowledge, skills, information
sharing, networking) and at the organisational level (e.g., audit of policies and practices, development and adoption of health
literacy policies and procedures). Interviews also revealed a ripple effect (immediate outcomes) at the individual level (e.g.,
enhanced health literacy confidence, advocacy, leadership, partnerships, networks) and at the organisational level (e.g.,
dedicated resources to implement health literacy principles & practices; embedding health literacy organisationally into
standard practice). These findings confirm that the health literacy capacity building initiatives (e.g., courses) are a ‘means not an
endpoint’, part of a ‘transformational change process not an event’, and are acting a catalyst for action – resulting in
incremental transformation in knowledge, attitudes, intentions and behaviours beyond the course participants and their
organisations - to have a longer term and multiplier effect.
39
The interviews with 2013 course participants who had changed employment organisation during the 2013 course, provided
further evidence of a ripple effect and of transfer of training – namely the generalisation and maintenance of health literacy
knowledge, skills and practices in organisation workplaces not participating in the courses.
Overall the evaluation confirms that health literacy course are continuing to build capacity to implement health literacy
concepts as demonstrated by: course participants developing leadership skills, building health literacy workforce knowledge and
skills, developing organisational infrastructure, developing resources and building networks mainly within and outside
participant organisations
To date five of the 10 HWP member agencies in the West have supported staff to attend the 2013 and 2014 health literacy
courses. Interviews with these course participants and senior executive sponsors revealed that:
• It takes time to engage staff and organisations to see any incremental changes in health literacy practice;
• Staff in a managerial and educational roles and who are connected to decision makers were more likely to have the
capability to lead or facilitate change (i.e., be ‘agents of change’);
• Multiple drivers (individual/person based drivers (e.g., supportive CEO, Board); organisation wide drivers(e.g., Accreditation
/ standards and/or Key Performance Indicators); and system-based drivers (e.g., quality improvement practices) are
required to start to focus on health literacy and to maintain the momentum amongst and staff across the organisation;
• Need to build internal organisational self-sustaining capacity to focus on health literacy, and not be dependent upon and
external organisation (e.g.,CEH);
• There is value in having access to a “Health Literacy Advisor/ Mentor ” – a person who has the knowledge, skills and
experience at engaging staff and organisations in health literacy; building evidence; developing and adopting health literacy
policy and procedures;
• A health literacy organisation maturity was developing – increased awareness amongst staff, management and boards that
health literacy is a priority; increasing use (or intentions to use) of health literacy concepts, frameworks and tools into
practice; increased reflection and critical appraisal about the most appropriate staff to do the courses and be ‘agents of
change. The concept of health literacy ‘magnet organisations could be considered – where organisations demonstrate:
leadership in health literacy (i.e., inspire health literacy practices); health literacy workforce knowledge and skills;
application of health literacy resources (frameworks and tools in practice; development of health literacy policy and
procedures; and partnership approaches to addressing health literacy issues;
• A continual focus on supporting a cultural shift – health literacy is part of good person centred care and NOT additional
work and not a fad is required; and
• A demonstration of how investing in health literacy practices will have practical impacts and benefits and converts into a
business case is required.
Utilising the Health Literacy Logic Model developed in March 2014 (see Appendix 2) and the Splash & Ripple Effect metaphor,
the evaluation also reveals and provides evidence that:
o the hypothesised program logic underpinning the 2013 health literacy course is confirmed. The hypothesised contextual
factors were confirmed and remain key issues needing to be recognised including: the variable knowledge amongst
professionals and organisations about health literacy; the ongoing organisational and systems reforms; the ongoing
competing priorities; the limited time & resources; the existence of strategic support for health literacy; and that quality
and safety are essential drivers for health literacy due to existing accreditation & governance systems
• the hypothesised project inputs (or the Rock) necessary for the health literacy practice to be implemented were also
confirmed including: collaborative leadership, the Course, and support from Course participant’s organisations.
• the hypothesised activities were also reinforced. The stories revealed how the activities are interdependent (i.e., not
mutually exclusive) in other words – it is difficult to identify any causal or direct / linear links between activities and actual
outputs. It is the cumulative impact of the activities that initially creates the SPLASH (outputs) and then the RIPPLE effects
(outcomes).
• the 2013 health literacy course has created a Splash (i.e., Outputs - changes at the individual level (increased participants
knowledge, skills, information sharing, networking, relationships) and organisational level (audit of policies & practices,
development of health literacy policies) level
• The short evaluation time frame limits our ability to comment on Outcomes achieved so far. However, the stories reveal
that the 2013 health literacy course is having a Ripple effect (i.e., Outcomes - immediate level) -at the individual level (e.g.,
40
enhanced health literacy confidence, advocacy, leadership, partnerships, networks) and organisational level (e.g., dedicated
resources to implement health literacy principles & practices; embedding health literacy organisationally into standard
practice, a culture where health literacy is the norm).
5. Evaluation Rigor and Limitations
The evaluator recognised that the health literacy initiatives developers (CEH, HWP, cohealth) required useful, specific and
contextualised evidence to inform future health literacy professional development initiatives in the Western region of
Melbourne. Thus, a participatory realist qualitative evaluation approach guided the evaluation methods, analyses and
interpretations.
Overall the 2014 evaluation (particularly related to the course and CoP) has provided evidence that the qualitative participatory
realist evaluation approach was appropriate and useful because it contributed to the initiatives development. The evaluation
findings also reveal that the working partnership between the health literacy initiative developers and the evaluator led to
increased evaluation learnings utilisation, and fostered a culture of learning between all.
As mentioned in Section 2 qualitative participatory evaluation approaches are often criticised as been value-laden, not neutral
or objective – even biased (Kushner 2012) due the evaluator needing to get up close and personal and build working
relationships with the primary intended evaluation users and evaluation participants (Paton 2008). Given this, some reflection
and discussion is required.
The evaluator used several processes to optimise the integrity of the evaluator, the transparency of the evaluation approaches,
and validity of data generation methods, analysis and interpretation.
Firstly, ethics approval from the Melbourne School of Population and Global Health Human Ethics Advisory Group was obtained.
Ethics approval demonstrates that the proposed methodology and methods has undergone an independent peer review process
by a panel of academic researchers.
Secondly, all interviews were transcribed verbatim by an independent transcribing organisation, to accurately capture the
experiences of evaluation participants.
Thirdly, the data analysis and interpretation occurred through a three step iterative coding process - open, axial and selective
coding – a recognised transparent qualitative process.
Fourthly, to test the validity of the data generation methods, analysis and interpretation I used two processes. Firstly, in Section
2, the evaluator showed how the evaluation approach matched the initiatives stage of development, context and evaluation foci
and questions - revealing the validity of the data generation method. Secondly, to demonstrate the validity of the data
interpretation, a member validation (Evaluation Participant Feedback) workshop was conducted for all evaluation participants
and the primary users of the evaluation. The workshop provided the opportunity to discuss, confirm, or redefine the emerging
evaluation themes; and to gain further insights into the experiences of the Health Literacy initiatives participants and their
organisations.
It is also important to recognise that qualitative data analysis and interpretation is about identifying and explaining the patterns
that are emerging – and not quantifying (as that is quantitative mind set). The evaluator has presented the findings in terms of
“Several participants” and not an exact number to illustrate the emerging findings. To provide an exact number (e.g., 4 out of
10) without providing context (e.g., participants roles, organisation, sector, knowledge, skills, experiences etc) would reduce the
value of the findings. Furthermore, given the small number of evaluation participants, providing such contextual information
would have led to disclosure and identification of the participants. Ethics approval from The University of Melbourne required
the protection of anonymity and the confidentiality of responses.
Lastly, the evaluator acknowledges that he is part of the Health Literacy initiatives Steering Group and was invited to the Health
Literacy CoP Leadership Group to provide input based on his observations, experience conducting the 2013 evaluation and his
broader evaluation experiences. To provide transparency and to be objective - avoid evaluator bias – I provide regular Progress
Reports to the funders (CEHH, HWP, cohealth) where I document my progress and observations based upon my involvement in
all the initiatives.
The evaluator does acknowledge that despite our evaluation approaches, the evaluation had limitations. While the sample size
(n=62 interviews) was reasonable, it was not representative of the entire 2013/2014 course participants nor CoP participants
41
cohort, but illustrative of all those involved. Despite inviting via email all 2013/2014 course participants to participate in a semi-
structured interview and / or workshops, the evaluator was unable to involve all in the evaluation process, notwithstanding
many invitations and email reminder attempts.
42
6. Key Emerging Propositions
To build upon and consolidate the investment in health literacy initiatives, the following propositions are presented structured
according to the three evaluation foci: Future Health Literacy Courses; Future Health Literacy Community of Practice; and the
Ripple effect in the West, as they have future policy, practice and research implications.
Future Health Literacy Courses
1. The health literacy courses are designed to create agents of change to lead or facilitate changes in health literacy practices.
Given that the evaluation revealed differing perceptions amongst course participants about their capability to be health
literacy agents of change, and that organisations choose their staff to do the course, the selection process and criteria that
organisations are using to choose staff to undertake the course may need to be reviewed.
2. Given that the current health literacy courses are targeted and been trialed with HWP member agencies in the West,
exploration of the development of a regional health literacy course, utilising the Better Health Plan for the West as a
positive policy platform’ could be considered.
3. The health literacy courses to date have been targeted at health professionals. However, consumers are recognised as part
of the workforce and can contribute to the co-design of the health care system. Given this, research investigating what
education, training and support consumers require to contribute to the health literacy of organisations could be considered.
4. The current course content and structure (four modules, mini-projects, Organisation Senior Executive Sponsors workshop) is
building individual capacity and a supportive organization authorising environment to implement course leaning into
practice - thus needs to be maintained.
Future Health Literacy Community of Practice
5. The increasing interest across the Western region agencies in health literacy as demonstrated by participation in the three
CoP events - warrants further investment in a Western region-wide health literacy knowledge transfer and exchange
network events (face to face and on-line)
6. Given the cultural, organisational, social and diverse health outcomes in the western metropolitan region of Melbourne,
piloting of topic (e.g., mental health) or setting (e.g., hospital) specific Health Literacy Community of Practices could be
explored.
7. The evaluation revealed mixed opinions about the role of the Health Literacy CoP Leadership Group and composition,
suggesting further exploration is required about what leadership and governance structure is fit for the CoP purpose?
8. The evaluation revealed mixed opinions about the identity and purpose of the current CoP, suggesting the need to further
explore: What is the CoP intended to achieve? For whom is the CoP designed for? For what circumstances is the CoP
designed for?
The Ripple effect in the West
9. The evaluation has revealed that the health literacy initiatives are creating a ripple effect amongst HWP member agencies
and beyond at an individual and organisational level in the West. However, health literacy action momentum needs to be
maintained. The role of health literacy mentors and health literacy magnet organisations, to further support the work of
CEH, HWP and cohealth, could be explored as a way to support individuals and organisations to implement and self –sustain
health literacy practices.
10. The evaluation (particularly senior executive interviews) revealed that it takes times to engage staff and organisations to see
incremental changes in health literacy practices. Given this, further investment is required in evaluating longitudinally
individual and organisational self-sustaining health literacy practice changes resulting from the health literacy initiatives.
43
11. Given the persisting set of enablers and barriers at the individual, organisation; and system level to implementing health
literacy practices, researching how course participant alumni and their organisation are responding to these factors will
contribute to understanding how to sustain the ripple effect.
12. The evaluation revealed evidence of a ‘ceiling effect’ for professionals attending multiple professional development
activities (e.g., course and the CoP). Exploration of the benefits of promoting future health literacy courses as a longer-term
commitment to a ‘community of practice’ could be considered.
13. To date, the evaluation approaches have been formative to inform decision-making, and not summative to assess the
outcomes resulting from the health literacy initiatives. Investment in developing and piloting an outcomes framework
(outcomes, outcomes indicators, data sets) to assess the ripple effects of the health literacy initiatives at the individual
(consumers, workforce), organisational, regional and systems level in the West is required
44
Appendix 1: Evaluation Data Collection Tools
2013 Health Literacy Course Participant & Senior Managers
Interview Topic Guide – Stories of Change
Background: This interview is designed to collect stories of change as a result of your involvement in the 2013 Health
Literacy Course. To assist in the collection of stories of change, we provide below a framework or domains of change – you may
choose to refer to.
Domains of Change
The evaluation of the 2013 health Literacy Course by the University of Melbourne revealed that the Course had built capacity as
demonstrated by course participants:
1. Developing leadership in Health Literacy (.i.e., inspiring Health Literacy thinking and approaches within their
organisations and other networks)
2. Building networks / partnerships among course participants and established an informal network for health literacy
knowledge transfer and exchange;
3. Developing health literacy workforce knowledge and skills;
4. Developing ways to use and apply health literacy resources (tools, frameworks) and
5. Serving as a catalyst for building organisational infrastructure (policies and procedures) to authorise and to use health
literacy practices routinely.
The Interview: As all good stories - the interview has three main parts: a beginning; a middle and an end.
Part 1: Beginning
1. Thinking back to before the 2013 Health Literacy Course, what was it like - what awareness or knowledge did you or
your organisation have about health literacy?
Prompts:
• When providing services to your client, was health literacy a consideration?
• When you interacted with colleagues and/or key stakeholders, was health literacy a consideration
Part 2: Middle
2. What was the most significant thing that happened to you and in your organisation during the 2013 Health Literacy
Course?
Prompts:
• What happened? What was that significant?
Part 3: End
3. Since completing the 2013 Health Literacy Course what was the most significant change that occurred in you and your
organisation in relation to health literacy practices?
Prompts:
• What motivated you to want to change?
• What factors have contributing to sustaining the changes
• Why do you think this is a significant change?
• What difference has it made/or will make in the future?
Thank you for your time and commitment.
For further information please contact: Lucio Naccarella [email protected]
45
Health Literacy Community of Practice
Face to face / Telephone interviews -Topic Guide
Introduction
Ethical considerations: How the interview will operate
Thank you for agreeing to be interviewed. This interview predominantly focuses upon the Health Literacy Community of
Practice.
• Could you tell me a bit about the Health Literacy Community of Practice that you feel involved with, or have some
connection with?
• Are you aware of the intended purpose of the Community of Practice and do you agree that its purpose is valuable?
• What assumptions do you feel existed about how the Community of Practice was supposed to work?
• Were you consulted in any way about that Community of Practice during its development or now while it is being carried
out?
• Do you feel you have any role in the conduct of that Community of Practice ? If so, what is that role and how enabled do
you feel to contribute in that way?
• Can you tell me a little bit about the context within which the Community of Practice has been established, from your
perspective as a key stakeholder?
• To what extent has the Community of Practice been implemented as intended?
• What contextual factors (barriers and / or enablers) may have influenced the implementation of the Community of
Practice?
• What are the perceived impacts of the Community of Practice?
• To what extent is the Community of Practice sustainable?
• What contextual factors (barriers and / or enablers) may influence the sustainability of the Community of Practice?
Thank you for your time and commitment.
For further information please contact:
Lucio Naccarella [email protected]
46
2014 Health Literacy Course Participants Interview
Face to face or Telephone Interview Two Guide
Introduction
Ethical considerations: How the interview will operate
Thanks for agreeing to be interviewed again
1. Can you briefly explain your current role? Has your roles changed since you began the course?
2. To what extent, and in what ways, has your understanding of health literacy changed or developed since you began the
course?
3. Could you tell me about the health Literacy projects you conducted? Would you like to add anything about the factors that
have enabled and constrained you in implementing the projects?
4. Can you share your experiences of taking your learnings back into your workplace? More broadly, what steps
have you taken to adopt and implement health literacy practices?
5. What changes have you made individually as a result of participating in the course so far? What contextual
factors have influenced these changes?
6. What health literacy practice changes have occurred within your organisation?
a. To what extent was your organisation ‘ready’ for adopting health literacy practices?
7. What contextual factors have influenced these changes
8. Can you briefly comment on the health literacy course content, participant composition and structure please.
9. Further discussion points raised
Thank you for your time and commitment.
For further information please contact:
Lucio Naccarella [email protected]
47
Appendix 6a
Post Module 3: 2014 Health Literacy Course Participants Senior Managers
Interview Guide
Your Name: …………………………………..Your Organisation: ………………………………………
1. What motivated / drives you and your organisation to focus on health literacy?
2. What changes in health literacy practices at an individual and organisation level have you have noticed since the start of the
Course
3. What enablers and / or barriers exist for your organisation in routinely using health literacy practices?
4. What strategies could support sustaining these changes in health literacy practices at an individual and
organisation level?
5. Is there anything else that you want to comment on concerning health literacy practices?
Thank you for your time and commitment.
For further information please contact:
Lucio Naccarella [email protected]
48
Appendix 2: 2014 Health Literacy Professional Development Logic Model
49 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
Appendix 3: 2013 Course Participants Stories of Change
Permission to spread and advocate health literacy practices- [C’s] story
[C] became involved in the 2013 Health Literacy course due to her role as the Senior Clinical Liaison for Nursing and
Immunisation within a primary health care organization. Being a nurse and a midwife who had previously worked in community
health, with diverse community groups, [C] had a good awareness and understanding of the importance of health literacy. Due
to work commitments, [C] only undertook Module 2 and Module 3 and was involved in two health literacy projects, focused
upon resource development and policy development. For [C], the most significant change resulting from participating in the
2013 Health Literacy Course was ‘being given permission’ to spread and advocate health literacy practices to colleagues and
organisations – as health literacy was not just about written materials for patients. The Course enhanced [C’s]’s understanding
about health literacy by making her think/reflect, and the Course provided her with insights into how one presents to and how
ones expresses oneself to patients and colleagues. Since the 2013 Course, [C] has changed employment – moving to a nursing
specific organisation, where she is spreading and advocating health literacy practices - as demonstrated by her desire to –
develop an on-line module on health literacy for general practice nurses, and eventually an organization-wide health literacy
policy. [C] is cognisant that within her new organisation, there is variable knowledge and understanding about health literacy –
thus, she recognizes the need to engage with known academics (e.g., Deakin University), practice experts (e.g., Centre for
Culture, Ethnicity & Health) and networks (e.g., Community of Practice to inform and support her health literacy advocacy. All of
this is significant, as the 2013 Health Literacy Course has enabled [C] to not just talk about health literacy, but has provided her
with additional and up-to-date knowledge, skills and tools to take action on health literacy at a nursing profession and
organisation-wide level.
Being seen as the expert leading health literacy practices – [K’s] story
[K] became involved in the 2013 Health Literacy course due to her role as a Health Promotion officer within a community health
centre. Prior to the Course [K] had limited knowledge about health literacy – and did not really think about health literacy in her
role, as she did not see clients. During the Course, [K] was tasked with writing an Integrated Health Promotion Plan, where
health literacy was a priority area – this provided her with both the opportunity to focus on health literacy, but also the
challenges involved in how to facilitate and embed health literacy work across the organisation. For [K], the most significant
change resulting from participating in the 2013 Health Literacy Course was ‘being seen as the expert – and tasked with leading
the health literacy work at her organization. [K] was cognizant that despite being seen as the expert, she had limited confidence
and knowledge to lead the work- due to having had only 12 months training in health literacy. Since the 2013 Course, [K’s]
organization has merged with two other organisations, and she has been tasked with leading the development of a health
literacy strategic plan. [K] has also become involved in developing the Health Literacy Community of Practice, which has
required her to take a leadership role – which has again challenged her notion of being an expert – she prefers to be seen as a
facilitator – leading health literacy work. All this is significant, as the 2013 Health Literacy Course and the Community of Practice
has provided [K] with the platform to take up the challenge of leading health literacy work.
Building networks – builds health literacy leadership capability- [T’s] story
[T] became involved in the 2013 Health Literacy course due to her role as Project Manager for Home and Community Care
within a health care organization - which had already planned to build health literacy into their new strategic plan. Prior to the
Course [T] had little knowledge about health literacy. For [T], the most significant change resulting from participating in the
2013 Health Literacy Course was ‘building networks’. [T] broadened her networks and reconnected with colleagues from Public
Health who had a broader view of health and understanding of health literacy (than her current clinical colleagues), that she
could call upon. Since the 2013 Course, [T] has also been tasked with the Health Literacy Portfolio in her organisation and has
become the Coordinator of the newly formed ‘Health Literacy Community of Practice’. [T] is seen as “the go-to person for health
literacy’. All this is significant, as the 2013 Health Literacy Course and now the Community of Practice has not just broadened
[T’s] networks, but built her health literacy leadership capability.
Increased knowledge and confidence to - always have a health literacy lens – [L’s] story
[L] became involved in the 2013 Health Literacy Course due to the role as Coordinator of Early Intervention and Chronic Care
Integration within a health care organization that was a core driver for the development of the 2013 Health Literacy course.
Prior to the Course [L] had some knowledge and awareness about health literacy. For [L], the most significant change resulting
from participating in the 2013 Health Literacy Course was increased knowledge, awareness and understanding about health
literacy into the new strategic plan. Since the 2013 Course, [L] observed changes at the individual level (e.g., increased
awareness, knowledge and confidence to apply health literacy) organization level (e.g., increased conversations about health
literacy within project teams) and systems level (e.g., health literacy emerging as a strategic priority area; investment in the
Health Literacy Community of Practice). Recently, [L] has moved to changed employment – to a non-health organization, where
50 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
she has a new role as Executive Officer. [L] is intent to always have a health literacy lens – as her new role enables her to have
authority and she has proposed the development and delivery of a health literacy professional development course for staff,
using information, frameworks and tools directly from the Course. [L] also intends to engage the Health Literacy Community of
Practice Coordinator to run a session at the organisation, in particular to emphasise the importance of the Ten Attributes of a
Health Literate Organisations – in order to raise health literacy as an organization-wide policy issue. [L] is cognisant that in a non-
health organisation, the phrase ‘health literacy’ is an issue and can act as a barrier to engaging staff. All of this is significant, as
the 2013 Health Literacy Course has led to health literacy becoming a way of thinking for [L] and has built [L’s] capability to
advocate the implementation of health literacy practices at an individual and organisation level.
New knowledge led to a change in work approach, career choice and becoming an advocate for health literacy practices –
[R’s] story
[R] became involved in the 2013 Health Literacy course due to his role as Program Manager within the e-health arena within a
primary health care organization. [R] had little to no knowledge about the term ‘health literacy’ prior to the 2013 Course. Due
to a change in employment, [R] only undertook three Course Modules (1,2,3) and was involved in two health literacy projects,
focused upon resource development and policy development. For [R], the most significant change resulting from participating
in the 2013 Health Literacy Course was obtaining new knowledge that led to a change in his work approach, his career choice
and becoming an advocate for health literacy practices. [R’s] overall approach to his e-health work changed – reflecting more
on patient perspectives. Since the 2013 Course, [R] has changed employment – moving to organization that specializes in e-
health solutions for people suffering chronic illness. [R] reported that participation in the Course had influenced his choice of
new employment – as his new employer’s were knowledge about health literacy, and that cdmNet (the new organisations
frontline chronic disease management network) had been developed considering health literacy issues. [R] reported that he
now advocates health literacy practices – to the point that that his colleagues are tired of him “harping on about health
literacy”. [R] has utilized his new knowledge to facilitate improvements to cdmNet – for example, improving client accessibility
to the computer measurement screen interface. [R] also commented that he would do the Course again, as “he is not an expert
yet” and that at an organisation level, competing project/work demands was real challenge to investing and developing an
organisation wide health literacy policy. All of this is significant, as the 2013 Health Literacy Course was an “agent for change’
and a “catalyst for action” for [R] to support him put health literacy concepts into practice at an individual work level.
Dedicated resources to implement health literacy principles & practices –[ K’s] Story
[K] became involved in the 2013 HL Course due to her role as a Communication Officer for a non-government organisation. For
[K] participating in the Course enhanced and broadened her view of Health Literacy as impacting at the individual, provider and
system level. For [K] the most significant change resulting from participating in the Course was her organisation’s dedicating
resources to the establishment of a Consumer Advisory Group and appointing a staff member to support maintain a health
literacy lens across the organization. upon [K] reported how across the organisation all staff were more aware of health
literacy, to the point where the CEO used then term health literacy when commenting about the need to re-do posters – we
have a “health literacy issue”. All of this is significant, as participating in the 2013 Health Literacy Course has led to [K]
advocating health literacy and eventually leading to changes at the organizational level.
2013 Course Participants Senior Managers Stories of Change
Emdedding Health Literacy Organisationally “down the Chain” – [J’s] story
[J] works for a community health centre that was a key driver and developer of the 2013 Health Literacy Course. [J] has been
instrumental in supporting staff attend the Course and facilitate the transfer and implementation of course learnings into
practice. Prior to the Course, [J] reported that health literacy was not explicit at the organization level. The Course had led to
increased visibility and internal health literacy champions to emerge. For [J], the most significant change resulting from the
2013 Health Literacy Course was the ‘embedding of health literacy organisationally’. The 2013 Course initially provided the
opportunity for a Course participant to embed health literacy into the newly developed Integrated Health Promotion Plan. The
creation of ‘cohealth’ (a three organisation merger) then provided the platform to focus and embed health literacy across the
whole new organization – with support from the senior executive level. There is a now a commitment to develop a Health
Literacy Strategic plan on its own right. All this is significant, as it demonstrates that the 2013 Health Literacy Course acted as a
catalyst and platform for emdedding health literacy organizationally – providing the drivers for organization-wide plans,
department plans, team work plans and individual staff work plans (e.g., position descriptions, performance appraisals) with
regard to health literacy.
A systems approach to embedding health literacy into standard practice [G’s] Story
51 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
[G] is the CEO of a health care organization whose strong interest in consumer participation and a background in quality
improvement, led her to having conversations and developing partnerships with like-minded organisations about the
importance of supporting health literacy practices. [G] advocated for health literacy to be a key result area (KRAs) in her
organisation’s strategic plan and to get the Board to support it and to dedicate resources. The Strategic plan was
operationalised by: the delivery of the 2013 Course, the creation of a health literacy position – all building blocks to embed
health literacy practices. [G’s] faith and trust in the partner organisations and individuals to develop, deliver and evaluate the
2013 Pilot Course was testimony that the time was right to be innovative. Since the 2013 Course, organisational resources have
been dedicated to a health literacy portfolio, which includes the Health Literacy Community of Practice (CoP) Coordinator
position. Furthermore, health literacy has been embedded at a regional level. For example, a Health Literacy professional
development session was conducted with the intake WING network for the west in Melbourne. Overall, the partnerships have
enabled the submission of a joint response to the Australian Commission Health Literacy Consultation Paper. At an
organisational level, [G’s] is trying to model good health literacy practices. For example: a health literacy lens is being used to
review the website, a Health Literacy position statement has been drafted for the Board, health literacy principles are being
incorporated into all Projects, and Project Managers are being asked to deliver on their KRAs (of which health literacy is one). [G]
has ensured that health literacy becomes embedded into standard practice by including health literacy into the strategic plans,
operational plan, work plans and project plans. All of this is significant, as a platform now exists for a more region-wide plan for
health literacy. However, [G] is cognizant that an authorising environment is required with the right players and organisations to
build an appetite, interest and traction to embedding health literacy into practice
52 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
Appendix 4: Pre-During –Post Course Participant Survey Findings
Table 1: Pre-During & Post Course Participant Health Literacy Competencies
Health Literacy Competencies Pre-Course
(n=19)
During Course
(n=17)
Post-Course
(n=11)
Explain differences in the various ways that health
literacy is defined and conceptualised
1= Easy
18=Difficult
15= Easy
2=Difficult
9= Easy
1=Difficult
Outline patient indicators and outcomes of low
health literacy
2= Easy
17=Difficult
13= Easy
4=Difficult
10= Easy
1=Difficult
Explain that it is the responsibility of the health care
sector to address the mismatch between patients’
and health care providers’ communication skills and
tactics
3= Easy
17=Difficult
14= Easy
1=Difficult
9= Easy
1=Difficult
Explain the relationship between health literacy and
health equity / Identify population groups that are
at increased risk of low health literacy
6= Easy
13=Difficult
12= Easy
15=Difficult
10= Easy
0=Difficult
Outline the risk management and quality
improvement imperatives of improved health
literacy
2= Easy
17=Difficult
12= Easy
5=Difficult
9= Easy
2=Difficult
Describe the rationale for applying a universal
precautions approach to health literacy
1= Easy
18=Difficult
10= Easy
5=Difficult
10= Easy
0=Difficult
Recognise, avoid and/or constructively correct the
use of medical jargon
9= Easy
10=Difficult
14= Easy
0=Difficult
10= Easy
0=Difficult
Effectively use the Teach-Back technique for
assessing patients’ understanding
5= Easy
14=Difficult
17= Easy
0=Difficult
10= Easy
0=Difficult
Apply plain language principles in written
communication and write in English at the grade 5
level
9= Easy
10=Difficult
16= Easy
0=Difficult
10= Easy
0=Difficult
Please note: responses have been clustered into ‘easy or difficult for ease of interpretation and that there is missing data
Table 2: Pre-During & Post Course Participant views on Health Literacy of Organisations
My organisation is … Pre-Course
(n=19)
During Course
(n=17)
Post-Course
(n=11)
Integrating health literacy into planning, evaluation
measures, patient safety, and quality improvement.
7- agree
7- undecided
11- agree
2- undecided
8- agree
1- undecided
Preparing the workforce to be health literate and
monitors progress
8 – disagree
5- agree
9- agree
6- undecided
6- agree
2- undecided
Including populations served in the design,
implementation, and evaluation of health
information and services
7- agree
5– undecided
10- undecided
6- agree
6- agree
3- undecided
Meeting the needs of populations with a range of
health literacy skills while avoiding stigmatization
8 – undecided
5 – disagree
6- agree
10- undecided
5- agree
2- undecided
Using health literacy strategies in interpersonal
communications and confirms understanding at all
points of contact
8 – undecided
6– disagree
5- agree
9 - undecided
5- agree
4- undecided
Please note: responses have been clustered into ‘agree, undecided or disagree for ease of interpretation and that there is missing
data
53 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
Appendix 5: Evaluation Participant Workshop – Draft Summary Report
Evaluation of Health Literacy Professional Development Initiatives
DRAFT SUMMARY REPORT – 4th
December, 2014
Dr Lucio Naccarella, PhD
Health Systems & Workforce Unit
Centre for Health Policy, Melbourne School of Population & Global Health
The University of Melbourne
This summary report has been prepared for the ‘Evaluation Participant Feedback Workshop on the 11th December, 2014
_________________________________________________________________________________________________________
Context To build the capacity of health professionals to use health
literacy concepts the Centre for Ethnicity, Culture & Health
(CEH) in collaboration with HealthWest Partnership (HWP)
and cohealth has delivered two Health Literacy training
courses, in 2013 and in 2014. The 2014 course comprised
of four face to face one day modules over an eight
month period. The Modules included: 1) Health literacy
and communication; 2): Organisational health literacy; 3):
Health literacy train-the-trainer; and 4): Building on the
health literacy knowledge. In 2014, two additional
initiatives occurred: A Health Literacy Community of
Practice (CoP) was established to provide opportunities to:
share learnings, experiences, resources; problem solving;
and to network. The CoP met on three occasions (May 5th,
August 25th
December 1st
, 2014). An Organisation
Executive Sponsors Workshop was also conducted on the
16th
September, 2014 to provide a strategic view of health
literacy in the West; to workshop the link between health
literacy and accreditation standards; and to discuss
enablers and barriers to supporting organisational health
literacy
Evaluation Focus The evaluation was designed to assess the extent to which
the health literacy professional development initiatives are
creating a ripple effect and building health literacy
capability at the individual, organisational, regional and
systems level. Key evaluation questions included:
• To what extent has the 2014 Health Literacy course
had an impact upon course participants and their
organisation CEOs?
• What makes the community of practice work (or not),
for whom and in what circumstances?
• What changes have occurred at the individual,
organisational and systems level as a result of
participating in the 2013 Health Literacy Course.
A developmental participatory realist evaluation approach
was used due to the iterative nature of the health literacy
initiatives. An adapted ‘most significant change’ technique
was also used to generate stories of change. Eight data
collection activities occurred:
1. one interview with 2013 course participants to explore
individual and organizational changes resulting from
participating in the 2013 course
2. one interview with 2013 course participant
organisation sponsors to explore individual and
organizational changes resulting from staff
participating in the 2013 course
3. pre-during and post course surveys of 2014 course
participants (as part of the CEH quality improvement
process) to explore changes in individual health literacy
competency and changes in organizational health
literacy
4. two rounds of interviews with 2014 course participants
to explore course experiences and individual and
organizational changes resulting from participating in
the 2014 course
5. one interview with 2014 course participants
organisation executive sponsors to explore health
literacy drivers and individual and organizational
changes resulting from staff participating in the course
6. one interview with 2014 Health Literacy Community of
Practice (CoP) Leadership Group to explore CoP
purpose, implementation, impacts and sustainability of
the CoP
7. one interview with Health Literacy Community of
Practice participants to explore CoP purpose,
implementation, impacts and sustainability of the CoP
8. one interview with 2014 course facilitators to explore
experiences of facilitating the course.
The evaluation had ethics approval from The University of
Melbourne Human Ethics Advisory Group.
54 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
Key Emerging Themes On the basis of 62 interviews overall the evaluation has
revealed that the health literacy professional development
initiatives are creating a ripple effect and building health
literacy capability at the individual, organisational, regional
regional and systems level.
The 2013 and 2014 Health Literacy Training Courses:
4 are creating a splash (outputs) – leading to changes at
the individual level (e.g., increased participants
knowledge, skills, information sharing, networking) and
at the organisational level (e.g., audit of policies &
practices, development and adoption of health literacy
policies and procedures)
5 are having a ripple effect (immediate outcomes) at the
individual level (e.g., enhanced health literacy
confidence, advocacy, leadership, partnerships,
networks) and at the organisational level (e.g.,
dedicated resources to implement health literacy
principles & practices; embedding health literacy
organisationally into standard practice)
6 are leading to transfer of training – as demonstrated
by the generalisation and maintenance of health
literacy knowledge, skills and practices, particularly in
participants who had changed employment
organisation during the course
7 are acting a catalyst for action – resulting in
incremental transformation in knowledge, attitudes,
intentions and behaviours of course participants
8 are continuing to build capacity to implement health
literacy concepts as demonstrated by: course
participants developing leadership skills, building
health literacy workforce knowledge and skills,
developing organisational infrastructure, developing
resources and building networks mainly within and
outside participant organisations
The 2014 Health Literacy Community of Practice (CoP)
has contributed to the health literacy Ripple effect in the
Western region as demonstrated by the:
• increased visibility of commitment to investing in
health literacy in the West
• creation of a novel and local space for the sharing,
reflecting and learning of experiences, particularly from
consumers
• provision of a structure for course alumni to take up
health literacy leadership roles; and
• provision of knowledge transfer and exchange avenue
for professionals not participating in the course
However, the current CoP does not have the essential
characteristics of a CoP, thus may not be ‘fit for purpose’,
as it:
• has multi-purposes, has an all inclusive membership
base, for all circumstances
• is governed by a consultative distributed leadership
model
• lacks a sense of shared identity, ownership or “us”
amongst members
• is competing for professional ‘time’
• is run as an organisation driven knowledge transfer &
exchange event-based network rather than CoP with a
group of people who are learning, innovating &
interacting together on an ongoing basis.
Sustained Health Literacy Action requires:
• multiple drivers:
o individual/person based drivers (e.g.,
supportive CEO, Board)
o organisation wide drivers(e.g., Accreditation /
standards and/or Key Performance Indicators)
o system-based drivers (e.g., quality
improvement practices)
• staff attending health literacy training who have
authority to either lead or facilitate change and who
are connected to the organizations decision makers
(e.g., CEO) and organisation management processes
(e.g., accreditation, quality management processes)
• leadership through the demonstration of health
literacy practices
• continual focus on supporting a cultural shift – health
literacy is part of good person centred care and NOT
additional work and not a fad
• demonstration of how investing in health literacy
practices will have practical impacts and benefits and
converts into a business case
Key Evaluation Findings
Profile of Health Literacy Course Participants
Ten agencies participated in the 2014 course, with two
people per organisation attending from a broad spectrum
of sectors (Public health; Not for profit; Community Health;
Acute care; Nursing education; Mental Health; Aged
Community care; Primary health; Aboriginal health),
organisations (Hospital; Medicare Locals; Community
Health Centres; Primary Care Partnerships; Peak Health
Organisations) and positions (Clinicians; Project officers;
Managers; Coordinators; Educators; and Personal
Assistant). Five of the ten agencies participating in the
2014 course had supported staff to attend the 2013 and
2014 health literacy courses – creating a receptive platform
for further adopting, implementing and maturing health
literacy practices across staff and organisations- this
observation was consistent with the views of the course
facilitators.
2014 Health Literacy Course: based upon the two rounds of
interviews (Round 1 n=14; Round 2 n=15) with the 2014
course participants
55 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
• Course content: the course content was perceived as
comprehensive, applied, contextualised, responsive,
adaptive and ‘how-to’ practically oriented. Overall
participants commented that the content had
increased the breadth and depth of knowledge about
health literacy as not solely and individual trait, but
that organisations have a key role to play. While
participants recognised the need to focus upon
organisational health literacy, there were some
concerns, whether there was too much emphasis on
health literate organisations - at the expense of the
health literacy of patients. Participants who had
limited authority to facilitate or lead change within
their organisations (e.g., Project Officers, Front-line
staff) also commented that the transfer of training
course content was not relevant, and at the expense of
health literacy specific content. Participants also
commented that information related to the mini-
projects requirements (e.g., to be conducted with
partners and scope (e.g., 40 hours) needed to happen
earlier than Module 1 to enable better planning.
Participants valued presentations from course alumni.
• Course structure: the four modules and two mini-
project structure were positively received, as it enabled
reflection, putting concepts into practice, seeing
immediate benefits, particularly the mini-projects, and
mainly intra-organisational networking opportunities.
Several participants commented that the course could
have been compressed into fewer days or even having
half-days, to keep the momentum up. Several
participants commented that activities within the
course modules could have been structured to further
facilitate inter-organisational networking. Participants
also suggested building in place-based visits for course
participants to organisations already implementing
health literacy practices – further demonstrating
leadership.
• Course participant composition: overall participants
were positive about having a mix of course participants
from diverse sectors, organisations and roles, as it
provided opportunities to hear and discuss other
viewpoints, differing challenges and opportunities that
exist. The pairing of participants from the sponsoring
organisation was viewed positively by both participants
and course facilitators – as it created a platform for
sharing learnings, problems and generating solutions.
Several participants also commented that pairing of
organisations could further facilitate and optimise
shared learning. Several participants commented that
the course composition mix could be further
broadened to include non-health, private, businesses
and insurance based professionals as this reflected the
“real” state of care provision.
2014 Health Literacy Community of Practice (CoP): based
upon interviews with CoP participants (n=12) and CoP
Leadership Group (n=4):
• Purpose of the CoP: The CoP was perceived as having
multiple purposes from: bringing together community
professionals; to creating a benchmark on current
understanding in the professional and community
context and; to reflect upon future challenges to health
literacy. Other CoP participants commented that the
CoP was for like-minded practitioners who know each
other, have a strong identity with the group and
participate to share ideas, experiences and to learn
together. Mixed opinions existed if it mattered if it was
named a CoP or a network. Most commented that
what mattered was that is enabled sharing and
reflective learning of ideas, strategies - professional
development. While others viewed the CoP for
problem solving and practice development as
compared to a Network meeting which was for
knowledge transfer and exchange. Interviewees also
commented that the CoP membership may not be
balanced - needed to ensure having a balance of
practitioners, decision-makers, consumers, and the
right drivers and leaders in the room”. Organisational
executives had variable and limited knowledge about
the occurrence of the CoP, however, they were overall
supportive. Interviews with the 2014 course
participants revealed that while all were aware of the
CoP, most had not been able to attend due to current
workloads, lack of capacity or competing priorities.
Indicating a ‘ceiling effect’ on the capacity of
professionals attending professional development
activities.
• CoP Implementation: Interviewees had mixed opinions
about the CoP implementation. Several participants
commented that the CoP ran like a network, and that
they had no connection to the other participants.
There were also mixed opinions about the CoP
structure. For example the workshop format was
perceived by several participants as not engaging nor
get into depth with regard to topics); and their was a
lack of clarity about the link or continuity between the
1st and 2nd CoP. Key CoP implementation enablers
included: high interest on health literacy; having
organisational buy-in, commitment and dedicated
resources; increasing dialogue; and consumer
participation. Key CoP implementation barriers
included: no sense of us existed in CoP; no defined
member boundaries; health literacy is broader than the
Western region; no shared identity/ownership existed;
competing priorities; and the CoP had no clear member
accountability plan/Key Performance Indicators and no
demonstrable benefits or value. A dynamic tension
existed between the Leadership Group being viewed as
delivering on behalf of CoP members versus facilitating
CoP members.
56 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
• Perceived impacts CoP participants had mixed opinions
about whether the CoP was meeting expectations. For
example, several participants expected to get
information about health literacy policies, but just got
more confused. Interviewees valued the sharing
learning stories, successes and tools; and hearing the
consumer voices and experiences. Others commented
that the CoP facilitated ground swell among those
already advocating action on health literacy, as it
consolidated knowledge gained. Overall the CoP was
viewed as a worthwhile initiative, but interviews
commented that even though organisations were
supportive, they needed to demonstrate benefits to
their managers.
• Sustainability: most participants considered it too early
to say if it was sustainable, as there were mixed
opinions as far as it was useful. Several interviewees
commented that broader perspectives were required
as well as greater executive buy-in. Having multi-modal
options available such as face to face and online
forums could enable more information sharing.
However, this would depend upon the topics and time
available – due to competing interest. A greater
emphasis on skill development was also expressed to
be sustainable. Several participants commented on the
need to make members accountable with clear plans
and key performance indicators. Most participants
commented on the need to create and maintain
momentum between members between CoP events,
and to more clearly articulate the link between the CoP
and the training courses.
Executive Sponsors Workshop (Sept., 2014): based upon
interviews with the 2014 Organisation Executive Sponsors
(n=10) and the second Course participant interview (n=14).
The workshop was perceived as very worthwhile by both
executives and course participants. Senior executives
commented that the workshop provided them with the
opportunity to: get their attention and involvement in
understanding, adopting and implementing health literacy
practices; becoming knowledgeable about their staff
commitments to health literacy; demonstrate their support
for their staff attending the course; and to demonstrate
organisational accountability to investing and supporting
staff attending the health literacy course. Course
participants commented that the workshop was key to
securing organizational buy in and further creating a
supportive authorising environment for the
implementation of health literacy policies and practices.
Several course participants commented that it may have
been better to have the workshop earlier to get senior
executive and organisational buy in early on – however, this
was not a consistent finding.
Changes at Individual Level based upon the 2014 Course
participant pre (n=19), during (n=17) and post (n=11)
course surveys, interviews with 2013 course participants
(n= 10 and their participant organisation sponsors (n= 5),
and two rounds of Course participant interviews (n=14 and
n=14):
• Individual health literacy competency –self-reported
individual health literacy competency via the
pre/during /post surveys has shifted from finding it
“difficult” to finding it “easy” to putting health literacy
concepts into practice. Course participants
commented that being asked to complete the survey
on their individual health literacy competency was
meaningful, as it provided points self-reflection for
them.
• Readiness for change: Interviewees revealed that
individuals and organisations were willing and ready
for change (i.e., adopting health literacy practices).
Course participant work roles (e.g., lead clinicians,
managerial, educational) and who were connected to
decision-makers had a greater capacity to be agents of
change – as these roles enabled them to have authority
to lead or facilitate change, as compared to course
participants who were Project Officers or front-line
staff, with limited perceived capacity to lead or
facilitate change. This finding was confirmed by the
course facilitators.
• Implementing health literacy practice– a core set of
enabler and barriers at an individual, organisation; and
system level exist (see Table 1)
9 Perceived impact (stories of change) Based upon the
stories generated using an adapted ‘most significant
change’ technique, participants reported that the
courses have led to changes at the individual level (e.g.,
increased knowledge about health literacy; increased
communication, writing and document reviewing skills;
increased critical assessment skills; overall being more
mindful, information sharing and networking) and at
the organisational level (e.g., development and
adoption of health literacy policies and practices).
The courses are also generating immediate outcomes
at the individual level (e.g., enhanced health literacy
confidence, advocacy, leadership, partnerships,
networks) and at the organisational level (e.g.,
dedicated resources to implement health literacy
principles & practices; embedding health literacy
organisationally into standard practice). Interviews
with participants who had changed employment during
the course, revealed their continued use and
application of health literacy knowledge, skills and
practices.
Interviews with participants from organisations who
had sponsored staff in 2013, further revealed that the
courses are acting as a catalyst for action – resulting in
incremental transformation in individual health literacy
knowledge, attitudes, intentions and behaviours
57 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
Overall the stories of change have revealed that the
courses have continued to build capacity as
demonstrated by: course participants developing
leadership skills, building health literacy workforce
knowledge and skills, developing organisational
infrastructure, developing resources and building
networks.
Changes at Organizational Level based upon the 2014
Course participant pre (n=19), during (n=17) and post
(n=11) course surveys and the interviews with 2014
Organisation Executive Sponsors (n=10):
• Organisational health literacy: over the course
duration, course participants perceptions about the
health literacy of their organisations have shifted, with
participants being either mainly undecided or
disagreeing initially that there organisation was health
literate to ‘agreeing’ that that there organisation was
health literate post course. Course participants
commented that being asked to complete the survey
on their organisations health literacy was not really
meaningful and a challenge, as organisations were
complex and comprised of multiple sections, levels and
departments etc – hence difficult to make
generalisations.
• Health literacy drivers: Executives reported several key
drivers including: health literacy being a core individual
value and organisation values – thus a strategic and
operational priority; health literacy emerged as a core
priority from organisation population health needs
assessment processes; organisation has structures
(e.g., Consumer Advisory Groups) which strongly
advocate for a focus on health literacy
• Implementing health literacy practices: Executives
reported a core set of enabler and barriers at an
individual, organisation; and system level (see Table 1)
• Perceived impact: Executives reported improved staff
knowledge, communication skills; the development of
organisational policies and procedures; the conduct of
organisational audits; increased intra-organisation
networking; and improved demonstration of health
literacy leadership.
• Sustaining health literacy practices: Executives
reported the need to keep the health literacy
momentum up by touching base with 2013 and 2014
course participants in 2015; providing annual
investment in staff professional development;
demonstration of health literacy practices by auspices
of Course, continually focusing on supporting a cultural
shift – Health literacy part of good person centred care
and NOT additional work and not a fad.
Key Emerging Propositions To further build upon and consolidate the investment in
health literacy initiatives, the following FOUR domains and
FOURTEEN propositions could be considered as they have
future policy, practice and research implications.
Domain 1: Future Health Literacy Training Courses
1. Course content could be further balanced to ensure
equitable focus upon health literacy at the individual,
organisation and system level. Explore using the health
literate organization survey as an activity-based
discussion in Module 1 and Module 4 to enable
participants to reflect upon the readiness of their
workplace context to adopt and implement health
literacy practices.
2. Course content information (e.g., project
requirements) could be provided prior to Module 1 to
assist participants to better plan an embed projects
into their routine work.
3. Course structure – the four modules and mini-projects
need to be maintained and supplemented by
increasing activities within modules to facilitate inter-
organisational networking. Building in place-based
visits to further contextualise learnings and
demonstration of health literacy practices could be
explored.
4. Course participant composition could be broadened to
include non-health, private, businesses and insurance
based professionals to reflect the “real” state of care
provision.
5. Course participant composition - to optimise sharing
and learnings future courses could recruit pairs of
organisations and staff from sectors.
Domain 2: Maintaining Health Literacy Momentum
6. Further invest in Western region-wide health literacy
knowledge transfer & exchange network events(face to
face and on-line)
7. Invest in facilitating topic (e.g., mental health) or
setting (e.g., hospital) specific Health Literacy
Community of Practices.
8. Conduct and expand the Organisation Executive
Sponsors Workshop to gain buy-in from executives and
organisations – involving course alumni and there
organisations.
Domain 3: Future Health Literacy Transformative
Changes Processes
9. Recruit professional staff in managerial, educational or
decision –making roles to do the course and to be
health literacy agents of transformative change.
10. Recognise and support health literacy course alumni
and other health literacy individual advocates to
become Health Literacy Mentors – to support
58 The evaluation was funded via a partnership between the Centre for Ethnicity, Culture & Health, HealthWest Partnership and cohealth
individual & organisations to implement health literacy
practices
11. Recognise and support organisations who have
developed health literacy maturity (i.e., demonstrate
adoption & implementation of health literacy policies
& practices) to be ‘Health Literacy Magnet or Beacon
Organisations’ to further support the leadership work
by the Centre for Ethnicity, Culture & Health,
HealthWest Partnership and cohealth to facilitate
individuals & organisations to implement health
literacy practices
Domain 4: Demonstrate the Value of Health
Literacy Practices
12. Invest in developing and piloting an outcomes
framework (outcomes, outcomes indicators, data sets)
to assess the ripple effects of the health literacy
initiatives at the individual (clients, workforce),
organizational, regional and systems level in the West.
13. Invest in research to explore the role, contribution and
impact of health literacy drivers (individual,
organizational, system-based) on self-sustaining health
literacy practices
14. The participatory realist evaluation approach has
generated rich evidence about the ripple effect of the
health literacy initiatives. Further investment in such
evaluation approaches could be considered to continue
to evaluate longitudinally individual and organisational
self-sustaining health literacy practices.
Acknowledgements
The evaluation was funded via a partnership between the
Centre for Ethnicity, Culture & Health (CEH) in collaboration
with HealthWest Partnership (HWP) and cohealth. We
particularly thank all the 2013 and 2014 Course
participants, executive sponsors, CoP participants AND
Leadership Group who participated in this evaluation as
without their cooperation this work would not have been
possible. For research correspondence please contact:
Lucio Naccarella, PhD [email protected]
Table 1: Summary of Enablers & Barriers to Implementing Health Literacy Practices
Implementation Enablers Implementation Barriers
Individual
• Compliance from clinicians
• Senior management support, networks and know-how
about organisation
• Upper management buy-in
• Support from frontline staff for health literacy
• Cooperation from other team leaders
• Dedicated staff training
• Staff connected to decision makers
• Staff in positions of authority to lead or facilitate change
Individual
• Staff changes
• Time constraints
• Getting upper management buy-in and endorsement
• Variable understanding about HL at upper management
level
• Competing priorities at upper management
• Terminology – health literacy
• Professional paternalism
• Reforms & fatigue
• No clear benefits to routinely using health literacy
practices?
Organisational
• Alignment between projects and organisation tasks e.g.,
accreditation
• Supportive authorising environment
Organisational
• Difficulty accessing project specific data (eg., patient
complaints)
• Ongoing organisation changes and re-structures
• Organisation has multiple sites which differ greatly –
organisation, priorities, and competing interests
Systems level
• Health literacy project aligns with accreditation system /
standards
Systems level
• Funding cuts resulting in staff redundancies