final - maramahdcrtf.co.nz/.../cbg-final-pilot-report_rtf_hdc-septe… · web viewcbg health...

Real Time Feedback System Pilot for People Using Mental Health and

Addiction Services

Final Pilot Report

September 2014

Provided to the Health and Disability Commission and Advisory Group

TA B L E O F CO N T E N T S

TABLE OF CONTENTS...............................................................................................2

1. INTRODUCTION AND BACKGROUND............................................................3

2. SETUP........................................................................................................................6

3. SITE VISITS..............................................................................................................8

3. DATA COLLECTION.............................................................................................10

4. ADDITIONAL QUESTIONS................................................................................13

5. LANGUAGES..........................................................................................................16

6. SURVEY SOFTWARE...........................................................................................17

6. ANALYTICS.............................................................................................................21

7. SUMMARY OF QUESTION MODIFICATIONS.............................................23

8. CONCLUDING COMMENTS..............................................................................25

2CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL

1 . INTRODUCTION AND BACKGROUND

This is the final pilot report provided by CBG Health Research (CBG) on the Real Time Feedback Pilot

for People Using Mental Health and Addiction Services 2014. This report summarises and references

key findings from the first and second qualitative reports and literature review. It is limited to the set

up and running of the pilot, and review of pilot findings to inform recommendations and further

discussion for a pilot extension and/or future roll out.

The Health and Disability Commission (HDC) contracted CBG to develop an electronic survey system

that will allow the public to provide feedback directly to service providers in real-time. Development

included collaboration with service users, their families / whānau, and service providers. We have

also worked closely with the HDC, an Advisory Group and other key stakeholders.

Preliminary work began in December 2013 and implementation began in January 2014, with data

collection using tablet devices commencing in April 2014. A six month data collection period will be

complete by the 28th September 2014. An independent evaluation commissioned by the HDC is

currently in process implemented by Malatest International and is scheduled for completion in

October 2014.

The HDC recruited seven pilot sites to represent a range of provider types, consumers, families /

whānau, and demographic groups. Twenty devices (Samsung Galaxy Tab 3) were purchased and

provided to the sites, some sites purchased additional devices of their own. The table below lists the

participating sites, the provider types utilising the survey and where the devices are located.


Pilot Site Name Devices Provider Types Device Locations

Counties Manukau DHB 2 Adult Mainstream, Pacific, Community Outpatient

Front desk, outpatient clinics

Waikato DHB 4 Adult Mainstream Inpatient and Outpatient


Front desk, inpatient ward

Northland DHB 4 Adult Mainstream Inpatient, Outpatient and Community



Waitemata DHB 7 Adult Mainstream Inpatient, Outpatient, Community and Residential



Turuki Healthcare 2 Mental Health Support Services within Adult Primary Healthcare


Offered at home visits

Odyssey House 6 Child, Youth and Family Admission’s, Assessment, Residential and Schools

Offered during individual or group sessions / assessments

Youth Horizons 10 Child, Youth and Family Community, Functional Family Therapy (FFT)

Offered during family therapy sessions


An online project plan was formally developed using a Gantt chart format, showing progress and

milestones. The plan has been accessible at any time and has been used to share relevant

documentation and communications.


2. SETUP

ORIENTATION VISITS

Post recruitment, an initial round of orientation visits were arranged by CBG with a HDC

representative attending where possible. The primary purpose was to commence engagement with

the key contacts, outline the project, and collect some preliminary information on systems and

context. The meetings identified and stimulated discussion on pilot site considerations, such as

where to place devices and how to manage their distribution.

INFORMATION PACK

An information pack was developed and distributed at the start of the implementation in January

2014. It provided a system and project overview, hardware specifications and some considerations

to inform placement of devices in service areas.

The HDC developed and released a consumer, family /whānau information sheet which was released

at the same time. This was later replaced with a public leaflet.

QUESTION DEVELOPMENT AND DESIGN

A programme of research and literature review was completed In February 2014. Considerable effort

was made to harmonise the proposed questions with existing work already being undertaken in New

Zealand, in particular by the Health Quality and Safety Commission on collecting data about in-

patient patient experience, based on question sets from the Picker Institute. A range of other

national and international research was also accessed and summarised in a literature review.

Possible domains of patient experience were developed and draft questions proposed.

To engage providers in project development a discussion guide was developed, including a bank of

possible questions. This was sent to pilot sites for internal distribution, using their own processes for

engaging consumers and their families / whānau. The discussion guide gave a summary of the

reasons for including the referenced domains and questions, which aimed to provide a starting point

to stimulate ideas, but not to limit the discussion about possible questions or the scope of the pilot

questions.


Consultation with consumer and family / whānau groups at each pilot site took place mostly

independently of CBG and elicited a significant amount of feedback, most of which was presented

and discussed with CBG at the first site visits. The feedback is reported in detail using thematic

analysis in the First Qualitative Report and had a significant influence on final question design. In

particular it confirmed the domains of patient experience considered important and the

development of alternative terminology for words identified as ‘labelling’, ‘negative’ and ‘illness

orientated’.

SET UP -RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE

The orientation visits were instrumental in generating enthusiasm and starting the planning process

for the sites. The information pack and leaflets would need to be reviewed and updated for any new

sites coming on board in the future, citing updated questions and titles more relevant to

preventative support services e.g. youth services. We would recommend that this information be

integrated into a larger resource pack which collates all information and training materials

developed throughout the pilot.

Including the consumer and family / whānau in the question design was key in setting up the survey.

Not only did it contribute significantly to the initial core questions, it provided guidance when

reviewing the questions, and also when deciding on individual site questions at a later stage in the

pilot. This process also resulted in an engagement process with consumer groups, many of which

then went on to support and promote the survey throughout the pilot. A longer period of

consultation would have allowed for a more varied approach to giving feedback, targeting those who

would not necessarily feel comfortable in a group situation or in a face to face interaction. For this

reason, we cannot presume that feedback given at this stage of the pilot represents the majority,

but clear themes emerged from those who contributed and all feedback was considered in the

question design. The planned independent evaluation methodology using online surveys will

hopefully allow a greater number of consumers, family / whānau to contribute to the developed

questions.

The demographics of the people and groups providing feedback was not always possible to

determine as this information was not always collected. For any future independent feedback run by


the sites, it would be advisable that more detailed guidance be provided to the sites requesting they

collect this additional information where feasible to ensure as many ethnicities and ages are

represented as possible.

It could be beneficial to have a means of on-going feedback on the core questions and additional

questions. How this would be implemented and the results considered would need further

discussion and a commitment from the sites, but would be a useful resource to continue to build a

provider, consumer and family / whānau sense of ownership and enthusiasm longer term.

3. S ITE VISITS

FIRST SITE VISIT

All sites were visited in March 2014 and as detailed above, the primary aim of the site visits was to

gather in feedback and present to the group, allowing further opportunity to contribute and

validation of existing feedback.

Other agenda items covered at the site visits are listed below. The initial plan to provide

demonstration and training on installation of the survey was in most cases not appropriate. The

attendees for the first site visit had a greater consumer focus and the appropriate person/s to be

taking responsibility for this were not always present, as well as practical constraints i.e. lack of Wi-Fi

at the meeting venues. There was also limited time and some items were given less attention than

ideal. Despite earlier communications detailed above, a lot of unplanned time was spent orientating

people to the survey and explaining the purpose of the pilot. The short time frame did not allow

internal communications to occur as effectively as they should.

Feedback from consumer, family / whānau representative’s and pilot site representative’s

on sample questions/patient domains as per discussion guide

Discussion on information pack

Distribution of allocated devices

Demonstration of sample analytics for feedback and discussion

SECOND SITE VISIT


This round of site visits allowed for at least six weeks of data collection and primarily aimed to gather

in feedback on the first set of core questions, response options and initial analytics as well as the

opportunity to identify additional questions specific to their services. Please see Second Qualitative

Report for feedback analysis and p.20 ‘Summary of Question Modifications’ for the revised

questions. All sites were then contacted to ensure they understood the purpose of the meeting,

discuss who should attend and to encourage internal discussion on any additional questions prior to

the meeting. The main agenda items covered are listed below.

Nature of core questions including, other questions or question types that should be

included

Possible additional questions to be added

The way that response options were presented

Managing the device/updating the survey and its use within the service

Usefulness of analytics

PILOT SITE WORKSHOP

Additional to the original project plan, a workshop was scheduled for mid-way through the pilot in

July. Two to three representatives from each site and all advisory group members were invited to

attend. This was primarily arranged and facilitated by the HDC with CBG collaboration into the

agenda and content. The workshop received good feedback and allowed attendees to talk directly

with the HDC and advisory group members, and to confirm their feedback was being heard. Giving

opportunity for further feedback into the revised questions, was a valuable exercise which modified

the questions further and promoted a sense of ownership.

SITE VISITS- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE

For the pilot process, the site visits were invaluable, not only on a practical level covering the agenda

items listed, but to fully engage the sites. The need for future site visits would depend on the

purpose, now that the questions have been modified. If further detailed qualitative research and

evaluation was required then future site visits would be recommended, but on a wider scale would

be costly and time consuming. The site visits also limit discussions to those attending. Methodology


that captures feedback on a wider scale e.g. online, would be a more cost effective means to

continue this process longer term for a larger number of sites.

Maintaining the face to face interaction via continued site workshops maybe a way forward, to allow

the sharing of experiences, ideas and ensure a continued connection with the HDC, developers and

advisory group. This would also allow for key decisions to be made that impact all sites and where

useful to hear the perspectives of other organisations i.e. data ownership or strategies that have

worked to increase survey uptake longer term.

3. DATA COLLECTION

FIRST SET OF CORE QUESTIONS

The First Qualitative Report analysing the provider, consumer, family / whānau feedback was

presented to attending advisory group members and the HDC on teleconference. A core question set

was subsequently proposed with further opportunity for feedback given to all advisory group

members. Some minor changes were made to reflect the feedback and a final core question set was

approved by the HDC. See table p.23 ‘Summary of Question Modifications’.

COMMENCING DATA COLLECTION

The first version of the survey was released on the 28th April. Instruction on downloading the survey

initially was provided and telephone support. Once the survey was loaded this would not need to

reoccur throughout the pilot, future versions required only an update, which is a much simpler

process. It took up to 4 weeks before all sites were actively returning results and up to 7 weeks for all

devices to be in field. The majority of the sites were returning some data within a few weeks.

MANAGING THE DEVICE AND TRAINING

1 0CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL

At the early stages of the pilot, wider internal staff training using the ‘train the trainer approach’ was

limited due to time constraints, but in reality there was very little ‘training’ involved. The need was

more around orientation and awareness. As the pilot progressed the term ‘training’ was avoided as it

painted a picture of time consuming formal training that was not required. Other issues arising

detailed below:

Poor access to Wi-Fi

Internal staff orientation and training

Management decisions on where to place the devices

Ownership of the devices

A significant arising issue was the need to upload results completed offline manually, currently

automatic upload only occurs when results are complete online. CBG were not aware that this was a

process that would need to be done manually and therefore, this was not fully communicated to sites

initially. This resulted in results being left on devices and staff being unaware. The process to

manually upload is very simple, but a task to remember to complete on a regular basis. It also means

results can be lost if a survey is updated and old off-line results are not manually uploaded.

With some exceptions, most sites had not fully established ownership of the devices and person/s

responsible for updating and managing the devices when data collection commenced. The fast pace

of the pilot left sites unsure as to where to place the devices right up until the release of the first

survey, this made identifying the key people to manage the devices a last minute decision for some

and a significant amount of support was required at the early stages. As the pilot progressed device

ownership improved, with key person/s identified to be responsible for managing all the devices.

This has worked much more effectively with the latest version of the survey being updated quickly

and easily with minimal intervention.

DATA COLLECTION- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE


Factors influencing the sites ability to roll out the survey quickly was mostly influenced by barriers

typical and often unavoidable within organisations, staff sickness, resources etc…. The tight timeline

of the pilot required pressure to be placed on the sites to get the data coming in as quickly as possible

to inform the initial stages of the pilot and allow further evaluation of the core questions. Under

normal circumstances with less time constraints, less support contact would be required, reducing

costs in bringing other sites on board.

Other issues arising discussed above, such as ownership, where to place devices, training etc… could

be detailed as part of the larger resource pack, provided post initial recruitment, increasing

awareness at the early stages and thus minimising their impact as barriers to implementing the

survey in their chosen service areas.

Although some advice could be provided on the issue of Wi-Fi availability, detailing strategies used by

other pilot sites, this issue would need to remain with the site to manage. Unfortunately it does

impact how ‘real time’ the results are for some services who are not able to access Wi-Fi on a daily

basis, but is an organisational issue different for each service that will need to be managed internally.

The steps to load the survey initially, and update in the future are simple enough for a person with a

novice level of IT literacy to complete, by following step by step instructions, with telephone support

if required. We would recommend that although a higher level of IT support within each organisation

would be required to manage the technical aspects of the device, e.g. passwords, security etc… a

person/s should be identified at the service level as well. The time taken to gather in devices and

have an ‘IT’ person update and send out again is wasteful, as the process of updating is simple and

can be completed at the service level easily.

The functionality for results completed offline to automatically upload when a Wi-Fi connection is

made, is a recommended software development task. We would consider this development essential

for a wider roll out. Once this functionality is available, the staff in service areas would only need to

update surveys when required.

The level of support required for the pilot could be significantly reduced on a wider scale, by ensuring

sites have identified staff, to take ownership over these tasks. Instructions at each step were emailed

out and feels fragmented. The person/s identified at the early stages of data collection have also

changed as the pilot progressed. Before the end of the pilot period the instructions for the task’s


listed below will be collated, updated and provided in one pdf document to all sites. This can be

updated in a wider roll out situation to reflect any software changes and incorporated into a manual

suggested earlier.

Loading the survey software and settings

Manually uploading offline results

Updating the survey

Managing setting on the device

Other considerations e.g. anti-virus, security etc…

4 . ADDITIONAL QUESTIONS

All sites were given an opportunity to add in additional questions specific to their service area. They

were all provided with a discussion guide to support sites in the design and implementation of

optional additional questions following on from the core question set. The content of this guide is

mostly reiterated below. In order to accommodate these additional questions a new survey was

required for each site.

Sites were not expected to come up with the ‘perfect’ question. They were mostly able to identify

the domains of patient experience, service or intervention to be evaluated, CBG assisted at the

second site visits and in collaboration with the HDC to design the appropriate question/s. Some sites

were very clear on their questions and needed little or no support in developing.

The bank of questions provided in the discussion guide were mostly selected from two sources:

Discussion guide of sample questions used to facilitate feedback when designing the core

questions

Suggested questions from the first pilot site visits and consumer, family / whānau groups

Some questions were omitted or amended to reflect the feedback given to date, and remove repeat

questions. These questions provided a starting point and were amended further to perfect the

wording and ensure relevance to the service.


Where some questions were considered irrelevant or too ‘medical’ to be applied to all pilot sites for

the core question set, we still included in the bank of questions. For sites wishing to evaluate an

inpatient service or treatment clinic, for example, they may well be useful. The word ‘hospital’ could

be replaced with residential centre, clinic or other service. Likewise a ‘group’ or therapy’ could be

replaced with a specific name.

There were a number of limitations set, listed below:

A maximum of 2 additional questions

All questions will be designed in collaboration with CBG and pilot sites and will ideally take

into account consumer, family / whānau feedback recently provided

All questions will need to be approved by CBG in collaboration with the HDC

Sites can have only 2 surveys running at the same time, it is not possible for the pilot to have

different additional questions on different devices.

- Core question set only.

- New survey with the core questions set plus additional questions.

Response options for the additional questions will be the same as the core questions, using

the smiley faces rating scale or selecting from a pre-determined list. Free text responses will

not be available.

ADDITIONAL QUESTIONS- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE

We would recommend the option to add in additional questions that are outside of the national core

set remains. Developing a core set of questions relevant to a range of organisations and service

areas has been challenging, and although they have been developed to be as applicable to all, for

some sites especially those outside of mainstream secondary care services, they may never be ideal.

Additional questions designed with the specific service or client group in mind, targeting key

programmes or initiatives will be a significant motivation for organisations to stay engaged with the

survey and drive change. It also allows for results from the core questions to be explored in more

detail, allowing services to target areas of need more specifically.


The option of additional questions creates the need for a survey per site, which comes at an

additional cost. A survey containing the core questions only can always be running and some sites

may prefer to have this, with no additional questions. The advantage of a separate survey per site, is

customisation with their own logos, tailoring the wording on the front pages to name their

organisation and other surrounding text. How much control is retained over the level of

customisation would need to be further discussed.

The limitations placed on adding in additional questions listed above we consider reasonable and

should mostly remain, with the exception of the following:

A maximum of 2 additional questions

All questions will be designed in collaboration with CBG and pilot sites and will ideally take

into account consumer, family / whānau feedback recently provided

Once a new survey is created there is no limit to the number of questions and therefore, no

additional cost incurred per question, but the feedback in the first round of qualitative research was

clear that consumers wanted the survey to be short, especially if being asked to complete each

episode of care. It would be worthwhile waiting to see if the independent evaluation identifies

length as a barrier to completing the survey, given it has increased since the first version. There may

be some scope to extend beyond two questions.

We would recommend that the final choice of question be approved by the HDC for inclusion, but

control over this should be limited, mainly ensuring anonymity is maintained. The survey

management team could provide an advisory role in developing the questions if required, using the

feedback from the First and Second Qualitative Reports and other experience in survey question

design. The bank of possible questions listed in the pilot could be reviewed and provided to sites as a

guide.

5 . LANGUAGES

The surveys have been made available in 7 different languages

English


Māori Tongan Samoan Chinese (simplified) Korean Hindi

The software offers different languages, but NZ International was used, which offers a high quality

translator services with extensive quality checks. The languages were then programmed using skip

logic. The translations are limited to the questions only. The front page and text introducing the

survey, as well as the final page remains in English only, due to software restrictions.

The languages were not made available until the third version of the survey at the end of July, due to

continued development of the questions. There was a significant amount of work and time involved

with lining up translators to be available to translate the revised set of core questions and additional

questions within the time frame proposed. This quick turnaround may not be guaranteed in the

future and time would need to be allowed for any additional translations in the future to occur.

This is going to be particularly relevant if sites wish to add in or change additional questions. Having

them translated into multiple languages means they cannot have the questions implemented as

quickly as if English only. There is also a significant cost as most quality translator services work on a

minimum word count for the minimum base charge per language. The cost to get one question

translated will be the same as approx. ten or more.

LANGUAGES- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE

Translations for any further changes to the core questions will become less as the final changes are

made and will not, therefore, incur a significant issue or on-going cost. Any further translations for

additional questions could be implemented in batches and set opportunities be given for question

review. It is not sensible or cost effective for one or two questions to be sent off for translation ad

hoc. The other option has already been briefly discussed with the HDC and advisory group;

translating the bank of questions mentioned earlier in this report. In developing a final list the sites

could have opportunity to add other questions. Any questions developed outside of this list would

need a process of development, authorisation, and translation. This process could be managed in its

entirety by the sites, and the questions provided for programming.


Ideally the survey front page should allow a language selection and then all text, information as well

as the questions continue and finish in that same language. If this is possible using the current survey

software would need further discussion, but we would recommend this as a future development.

6. SURVEY SOFTWARE

Having reviewed the NHS technical solution and a range of other options, detailed in our proposal,

including building a system from scratch, our preferred option at that time was a service provided by

another local New Zealand company. The company was “Kinross” and the product is called “AMS”.

Another New Zealand company called droidSURVEY came to our attention at a late stage, they

offered all the same desired functionalities listed below, and had a significant advantage over AMS

for increased customisability and ease of use, for this reason we decided to adopt the droidSURVEY

platform.

https://www.harvestyourdata.com/

FEATURES

Does not require a particular operating system

Device independent

Collects data both online and offline, so that if a Wi-Fi or mobile broadband connection fails

or is unavailable data can is stored and uploaded when the connection is restored.

Collects a wide range of data types including single and multi-choice selections (sliding scale

or smiley face selection), text, numbers, email capture, images, photos, videos, signatures,

barcodes and locations.

Provides skip logic functionality without the need for any coding language (to implement

questionnaire branching).

Multiple surveys can be loaded on to a device and you can quickly switch between them

when it's offline, without losing any of your captured results.

Each purchased survey can be loaded on an unlimited number of iOS and Android devices,

no charge per device or question


https://www.harvestyourdata.com/

Can manage all devices centrally

Supports data protection that is implemented at the software level and works with the

hardware and firmware encryption to secure data collected in the event that a physical

device is stolen or lost.

All export options needed for downloading data collection results in the SPSS format or in

CSV (comma separated value) format, which is 100% compatible with Microsoft Excel.

Automate the downloading of results with the ‘Developer API’ and integrate data with other

software.

Real time charts providing analysis of results that are displayed in a customized dashboard,

automatically refreshing every minute with the latest questionnaire data. (this link can be

embedded in other online sites, newsletter or displayed as a live link in waiting rooms)

You can set up surveys to record time stamped GPS co-ordinates for each survey result

No programming is required to create your survey or form.

Automatically restart the survey after a certain amount of idle time that is specified. You can

choose to display an optional warning message prior to the survey restarting.

Control whether the device saves or discards partial results.

User interface that automatically proceeds to the next question after a user answers a single

select question. Users can quickly move to the next screen without having to select the next

screen button.

DEVELOPMENTS MADE

There were a number of developments we needed initially before the surveys could be

implemented.


Landscape option

Customised response option displaying 5 smiley faces on one screen

RECOMMENDED FURTHER DEVELOPMENTS

The table below summarises recommended software developments that are needed to improve the

overall look of the survey and ease of use from a management perspective and at the user level. The

cost and time frame for these developments to be done would need further discussions with

droidSURVEY and at this stage of the pilot a further review of other systems including developing a

tailored system from scratch that meets all the requirements.

Requirement Details

Improved editing capability Increased editing capability for the 'look' e.g.

different font sizes, colour on different screens,

currently selections are the same for all. Also

ability to change positioning/ spacing.


Original option available Current option

Improved look for landscape Although the survey was developed to use

landscape mode, many other features designed

originally to look good in portrait mode were not

changed. e.g. positioning of multi select to the

right of the question rather than underneath, and

text for first and final pages split across the page.

Preview screen available for landscape For the same reason above, this functionality only

displays in portrait. When programming it would

save a significant amount of time to be able to

preview the survey on screen, rather than

repeatedly loading to the device to test.

Manually remove results from droid database The real time results on the public page accesses

the droid database before the data is exported to

our own systems for further analysis. Occasionally

false results are entered by sites for demo

purposes and we cannot manually remove

without asking droidSURVEY

Automatic upload of completed offline surveys

when Wi-Fi connection is made

This was a feature we originally thought possible

and is a priority development for the future.

Should be the default option that can be turned

off if required.

The next button for text response questions

needs to be more prominent

Currently there needs to be instruction to tap the

‘Next’ button as it’s not immediately obvious.

Combining droid databases The real time results on the public page start

again when a new survey is published, you cannot

combine the databases.

Translations available for all survey screens not

just the questions

Option on the front pages to select language and

all screens to follow in that language

6. ANALYTICS

Feedback on the nature of reporting was minimal in the early stages. All sites were given the

opportunity and provided with sample analytics for discussion at the site visits. Generally feedback

on what we were presenting and proposing seemed adequate at that stage of the pilot. Sites were


not ready to address analytics in detail at this stage. The focus was more on the questions design

and where to place the devices within services.

For others lack of experience in analysing data or looking at graphs and charts made consideration of

the analytics overwhelming and there was a reluctance to offer any feedback at the site visits. In

most cases the person who would be responsible and more interested in the analysis was not

present.

The public results update every minute and can be viewed www. patientexperiencesurvey.co.nz

With the latest version, sites now have their own surveys and the live overall results per site will be

displayed in the same format, by going to the same website and clicking on the heading ‘Public

Results per Organisation’ and selecting their organisation. Any site that does not want their results

accessible to other sites can opt to have a separate link which they can embed in their own website.

The sample analytics for the organisations were presented to the advisory group and the HDC and

were implemented once a reasonable amount of data was returned from each site to ensure

anonymity. We have used an analytic software solution called Yellowfin, presented in real time

dashboards via a secure log for each site. Reports can be easily exported, emailed and printed. Links

can also be created to allow users to embed in other sites, newsletters or other means of circulating

the results.

At the second site visit feedback was gathered on the analytics and at this stage a number of

suggested improvements and requests were identified. Having a visual starting point made this

process easier to conceptualise. The majority of these improvements have been made and we will

continue to work through the rest for completion before the end of the pilot period.

ACCESS TO RAW DATA

Currently the sites have not been given the raw data for any of the questions. Concerns around risk

management of the raw data for the text response questions led to this being withheld initially. The


http://www.patientexperiencesurvey.co.nz/

following paragraph was communicated to the public in the leaflet and is stated prior to the text

response question on the survey screen.

“Any additional comments as text feedback will not be made fully available to services, but

presented in themes analysed automatically by a word search using computer software. This is to

ensure anyone completing the survey or staff members are not identified. Any requests for assistance

will not be viewed by services and cannot, therefore, be followed up. If you feel you are in distress or

need help please contact your local mental health services, GP or nearest Accident and Medical”

This makes this question on screen look very crowded, but it was decided that this was required until

further decisions were made on this issue.

ANALYTICS - RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE

Yellowfin would continue to be our preferred analytical software solution and can be expanded to

allow users with a higher level of technical expertise to utilise additional functionality, in particular

“self service” business intelligence, including the creation of site specific reports – tables, reports

and dashboards. Further customisations can be made very easily to the existing suite analytics

easily as required in the future.

Discussions on access to the raw data has already progressed and final decisions on this will be made

by the HDC following further discussion with participating sites and management of risk. The

advisory group consensus indicates that the raw data, in particular for the text response questions

can provide very rich and useful information that can be instrumental in driving change.

The use of the results and how effective they will be at driving change should be a key focus

following on from the pilot. Working with sites to encourage the dissemination of the results at

multiple levels should be a key priority. Included in this, should be access to the results for the public

to keep the people motivated to complete and see how the results are being used to improve the

services. The trend data will allow an overview of the uptake rates week by week, overall and across

services. This information should help inform sites as to where their devices may be best placed and

enable them to continue to improve survey administration and delivery.


To date, sites have not described an unwillingness to complete the survey more than once, although

there are real concerns that people will become less enthusiastic as time goes on. A longer period of

data collection is required for this to be evaluated. The survey is still progressing and recent

improvements to the questions may increase the likelihood that people will be willing to complete

more than once, or on a regular basis. Keeping up provider interest in administering the survey is

also going to be a challenge and sites have already started to think about moving devices around

more often, and advertising results. Advice on long term management of real time surveys will likely

be sought post pilot period and each approach will be dependent on the organisation, the services

and even personalities involved, but the analytics will play an important part for all.

7. SUMMARY OF QUESTION MODIFICATIONS

The table below show the original core questions implemented in the first version of the survey and

the modified questions implemented in the current version at the end of July. Please note question

five was omitted from the first version in error. A second version was published in June to include

this question, an evaluation question and an opening question asking if they are answering as a

consumer or family / whānau member. Additions to version2 listed below.

1. I am completing this survey as… a consumer or service user or person seeking support family, whānau or friend

5. Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement….

I have the support I need for the future

12. Did the questions in this survey cover things about the service that are important to you? Yes completely Some things Not really Not at all

13. What else would you like to have been asked about? (text option if ‘Not really’ or ‘Not at all’)

First Set of Core Questions, April 2014 Modified Core Questions, July 2014

The people I see within the service value my opinions and are compassionate, non-

Thinking about your most recent experience with the service/people who support you, how much


judgemental and respectful of my cultural needs.

do you agree or disagree with the following statement….

I feel respectedI am sufficiently consulted and involved in decisions about services to meet my identified needs

Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement….

I am involved in decision makingThe services I use communicate well with each other and are fully informed about my needs.

Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement…. (Don’t know option)

The people I see communicate with each

other when I need them to

My family / whānau or someone close to me has been given information and support to be involved in my care and recovery.

Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement…. (Additional Option: I didn’t want them to be involved)

My family / whānau are given information and encouraged to be involved

The services have supported me and my family move forward in recovery.

Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement….

I have the support I need for the futureI would recommend this service to friends and family if they needed similar care or treatment.

Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement…. (Optional)

I would recommend this service to friends and family if they needed similar care or treatment.

Is there anything you want to say about your recent experience with the service or anything you think we can improve on?

Thinking about your most recent experience with the service/people who support you… (Optional)


What don’t you like?

Thinking about your most recent experience with the service/people who support you… (Optional)

What do you like?

8. CONCLUDING COMMENTS

This has been a very rewarding project in a number of ways. There have been a number of technical

challenges that have been interesting and satisfying to resolve. The idea of measuring and acting

upon real time patient experience data has been universally supported by a wide range of providers.

During site visits, advisory group meetings and in the meeting of all participants, we have observed

stakeholders develop ideas for leveraging the potential of the infrastructure, and observed support

for widespread adoption of the system within organisations.

Having youth services and primary care on board as pilot sites has challenged the core questions

significantly to be relevant outside of mainstream mental health services. Consequently the

developed core questions are now more applicable to wider range of services.

CBG is looking forward to the next stage of this project and expanding the number and range of

providers using this tool. One challenge will be developing a coordinated management structure for

real time feedback, and defining the role of this system in relation to other national services.


final - maramahdcrtf.co.nz/.../cbg-final-pilot-report_rtf_hdc-septe… · web viewcbg health...

Documents