post implementation review - amazon...

D 4.6

Authors:

Jill Evans, Dr Louise Moody

HDTI, Coventry University

Project co-funded by the European Commission within the ICT Policy Support Programme

Dissemination Level

P Public ��

C Confidential, only for members of the consortium and the Commission Services

Post Implementation Review

Project Acronym: DISCOVER

Grant Agreement Number: 297268

Project Title: Digital Inclusion Skills for Carers bringing

Opportunities Value and Excellence

of 2

REVISION HISTORY AND STATEMENT OF ORIGINALITY

Revision History

Revision Date Organisation Description

#1 HDTI Report template and draft strategy

#1.0 12/03/15 HDTI Full draft v1.0 sent out for feedback

20/03/15 All Feedback received

#1.1 30/03/15 HDTI Full draft v1.1 sent out for feedback

31/03/15 All Feedback received

#1.2 01/04/15 HDTI Full draft v1.2 sent to project coordinator

02/04/15 Digital Birmingham Feedback received

#1.3 08/04/15 HDTI Final version circulated to project partners.

Statement of originality:

This deliverable contains original unpublished work except where clearly indicated

otherwise. Acknowledgement of previously published material and of the work of

others has been made through appropriate citation, quotation or both.

of 3

Table of Contents

EXECUTIVE SUMMARY 6

1 INTRODUCTION 8

1.1 Aim 8

1.2 Purpose 8

1.3 Data collection 9

1.4 Structure 9

1.5 Intended Audience 9

2 PILOT PHASE 2 METHODOLOGY 10

3 PILOT SITE DESCRIPTIONS & VARIATION 11

4 CHANGES BETWEEN PILOT PHASE 1 AND APPLIED TO PILOT PHASE 2 18

4.1 Changes to the DISCOVER service (reported in D4.5) 18

4.2 Suggested changes to the pilot project methodology 19

4.3 Changes to Pilot site delivery changes (informal observations) 20

5 A SUMMARY OF THE IMPACT EVALUATION 22

6 USABILITY ASSESSMENT OF DISCOVER 23

6.1 Aim 23

6.2 Method 23

6.3 Results by Country 28

6.4 Discussion 37

7 SYSTEM DATA 40

7.1 Pilot Phase 1: System Data 40

7.2 Pilot Phase 2: System Data 40

7.3 Results 41

7.4 Discussion and conclusions 44

8 LESSONS LEARNT 45

of 4

8.1 Method 45

8.2 Results 45

8.3 Discussion 61

9 THE EXIT STRATEGY 62

10 CONCLUSIONS 64

10.1 Efficacy of the DISCOVER solution and supporting processes 64

10.2 Improving the the quality of life of carers 64

10.3 The exit strategy 65

10.4 Final conclusions 65

APPENDIX A: PILOT PHASE 2 USABILITY SURVEY FOR CARERS 66

APPENDIX B: LESSONS LEANT – NARRATIVE DESCRIPTION 70

of 5

Tables

Table 1: Number of Participants (carers and cared for people) recruited for Pilot Phase 2 [by country] ............................................................................................................................................. 10

Table 2 Pilot sites comparison: National context & similar services available .......................................... 12

Table 3 Pilot site comparison – service and responsibilities ..................................................................... 13

Table 4 Pilot Site Comparison: Target group(s) for the pilot and for the service post pilot ....................... 14

Table 5 Pilot Site Comparison: Pilot running ............................................................................................ 15

Table 6 Pilot Site Comparison: Pre- and Post-Implementation- Changes during pilot implementation ................................................................................................................................. 16

Table 7 Pilot Site Comparison: Future development and business plans ................................................. 17

Table 8 Number of Participants (carers and cared for people) recruited for Pilot Phase 2 [by country] ............................................................................................................................................. 24

Table 9 Participant summary per country ................................................................................................. 24

Table 10 Education levels of pilot participants .......................................................................................... 25

Table 11 Location of caring role for pilot participants ............................................................................... 25

Table 12 Use of DISCOVER .................................................................................................................... 25

Table 13 System use statistics for each pilot site learning zone component............................................. 42

Table 14 Help section view counts ........................................................................................................... 43

Table 15 Lessons Learnt: Carer Recruitment ........................................................................................... 46

Table 16 Lessons Learnt: Carer retention ................................................................................................ 48

Table 17 Lessons Learnt: Working with Stakeholders .............................................................................. 50

Table 18 Lessons Learnt: 4 different pilot sites / countries / services ....................................................... 51

Table 19 Lessons Learnt: Pilot process ................................................................................................... 52

Table 20 Lessons Learnt: Data collection tools ........................................................................................ 54

Table 21 Lessons Learnt: Data Collection Processes .............................................................................. 55

Table 22 Lessons Learnt: Service Content............................................................................................... 56

Table 23 Lessons Learnt: Service Development ...................................................................................... 57

Table 24 Lessons Learnt: Pilot Delivery ................................................................................................... 58

Table 25 Lessons Learnt: Business Plans ............................................................................................... 60

of 6

EXECUTIVE SUMMARY

This report (deliverable 4.6) details the findings from task 4.5 the Post Implementation Review.

The Digital Inclusion Skills for Carers bringing Opportunities, Value and Excellence (DISCOVER)

project was funded by the European Union Competitiveness and Innovation Framework

Programme. It has run from April 2012- March 2015 and involved 9 partners from 5 countries (UK,

Netherlands, Greece, Ireland and Spain).

The DISCOVER service was developed to provide online support to carers. The service provides

support to users through a portal to information about their caring role and access to learning

about digital skills. It has been piloted in two phases in the UK, Netherlands, Greece and Spain.

The first phase evaluation (Pilot Phase 1) sought to inform the iterative development of the service

and test the pilot methodology. This report details some of the changes to the DISCOVER service

and pilot site delivery that were made as a result of the first pilot. Following on from Pilot Phase 2,

the report comments on the effectiveness of the service from the user perspective, the

development of the service, the success of the pilot implementation with reflection on site-based

differences and the lessons learnt from the pilot operation. This deliverable also reflects on the

impact evaluation results reported in D2.7 and D5.3 the Business Development Plan that has

emerged from the project.

Pilot Phase 2 ran from 01/01/2014 to 15/02/2015 dates. During this time 371 carers and 18 of the

older people they care for were enrolled and trained in the use of the system. Data was collected

from carers to allow an impact evaluation and assessment of the usability of the service. In

addition semi-structured interviews were undertaken with pilot site leads, system data was

captured to understand how the service was being used, and lessons learnt were captured

through interviews and project partner’s workshops.

DISCOVER was developed for, and piloted in 4 different countries. The importance of tailoring

both the service and the pilot to the local country context and requirements was recognised. As a

result of the site by site differences, much of the data collected has been considered on a country

basis, rather than being combined together. The business plans are specific to each country pilot

site.

The impact evaluation (see D2.7) provides a final analysis of the skill and quality of life gains made

by participants using DISCOVER. Overall it was concluded that many carers learned new skills and

gained new knowledge through engaging with the DISCOVER Skills Zone. Carers reported that they

valued finding out about local and national services, enjoyed leisure activities such as brain games

and as a result of DISCOVER felt more supported in their caring role. Carers’ feelings of wellbeing

were found to improve during pilot phase 2. Both informal and formal carers would recommend

DISCOVER to other carers and formal carers would also recommend it to their employers.

Under task 4.5 usability testing of DISCOVER was undertaken. This explored differences in end-

user perception of usability within each country. The service was considered on a country by

of 7

country basis given the variations in content. Broadly the findings suggest that participants feel

the system was enjoyable to use and met their expectations. Greek and UK-based participants

appear to be most satisfied with the service. There did not appear to be any major usability issues,

and ratings for accessibility were positive. Participants strongly agreed that they would not pay for

the service. Caring skills were mildly rated as having improved. Feedback indicated that

communication with others had not been improved as a result of using DISCOVER. The analysis has

led to recommendations for future development both of the service and platform to ensure that

system usability does not discourage use.

A review of system data was undertaken to provide insight into actual use of the system during

the pilot. The data indicates that the number of registered users varied across pilot sites. The first

piece of content / course provided by the service has the highest number of hits, perhaps

suggesting that use is not continued through the content, and content is explored sequentially.

The lessons learnt through the piloting of DISCOVER have been captured. Issues arising have been

documented, alongside the potential impact and how the project team responded to address

these issues. The issues are wide ranging and cover recruitment, retention, working with

stakeholders, and working piloting in different countries etc. It is intended that this log provides

guidance for similar projects and further exploitation of the DISCOVER service.

In conjunction with D5.3 the report summarises the ability of the DISCOVER solution to be

competitive in the market. A business model for each country is presented and factors affecting

the exit strategy at each site are briefly summarised.

The report concludes that overall DISCOVER, as a funded project has achieved the primary aim of

creating a new service to deliver digital skills training for carers. The service has been developed

and tailored in a country specific way to cater for local context and to ensure sustainability. The

pilot results indicate that the service is valued by users and that the service offers adequate

usability. Recommendations are summarised for the further development and exploitation of the

service.

of 8

1 INTRODUCTION

This report provides a post implementation review of the DISCOVER service. DISCOVER has been

developed over the period April 2012- March 2015 to provide online support to carers. The service

provides support by providing a portal to information about their caring role and access to

learning about digital skills. It has been piloted in two phases in the UK, Netherlands, Greece and

Spain. The first phase evaluation (Pilot Phase 1) sought to inform the iterative development of the

service and test the pilot methodology. The second phase (Pilot Phase 2) sought to explore the

efficacy of the solution and the impact on learning.

This report represents the output from Deliverable 4.6 and reports on the DISCOVER pilot

implementation. It will comment on the pilot processes including country / pilot variations. It will

reflect on the opportunity that the service offers to improve the quality of life of carers (linking to

D2.7). In conjunction with D5.3 it will comment on the ability of the DISCOVER solution to be

competitive in the market. Factors affecting the exit strategy at each site will be discussed. In

order to guide future development and exploitation, the lessons learnt from the project are

mapped.

1.1 Aim

The aims of this work package and deliverable are:

• To provide a post implementation review of the DISCOVER service

• To detail changes to the service since Pilot Phase 1

• To evaluate the usability of the DISCOVER service as is stands during Pilot Phase 2

• To provide a review of the pilot running and operations during Pilot Phase 2

• To collate lessons learnt during Pilot Phase 2

As a result of the above, this deliverable will complement deliverables D2.7 and D5.3 by collating

the results from the Pilot Phase 2 evaluation and drawing conclusions around the: • Efficacy of the DISCOVER solution • Effective supporting processes • Achievement of the desired outcome to improve the quality of life of carers through

learning, informational access and support outcomes for the end-user • Ability of the DISCOVER solution to be competitive in the market as a solution • An exit strategy that ensures the Discover solution gives support to formal and informal

carers for three years, after the completion of the project. The business part of the exit

strategy is reported in D5.3 Deployment Plan.

The final results of this post review evaluation provide a success indicator and reinforcing evidence

for the effectiveness of the DISCOVER service.

1.2 Purpose

The purpose of this report is to provide a post implementation review of the DISCOVER service. It

is a success indicator of DISCOVER effectiveness, providing measurable and quantifiable results to

allow assessment of the efficacy of DISCOVER.

of 9

The report will comment on the effectiveness of the service from the user perspective, comment

on the development of the service, the success of the pilot implementation with reflection on site-

based differences and the lessons learnt from the pilot operation.

The report will outline the final usability evaluation undertaken on the DISCOVER system, and

relate the findings to the feedback and subsequent technical developments arising from D4.5

(Service Evaluation and Adaptation Report). This allows us to determine where the strengths and

limitations of the DISCOVER service lie from a user perspective. The report also links to the future

business plans for each DISCOVER site discussed in D5.3.

It is intended that that the report will guide future exploitation of the project and provide useful

reflections for future EU projects.

1.3 Data collection

The findings reported here have been derived from the Pilot phase 2 implementation. The pilot

site evaluation has involved the following methodology to evaluate the DISCOVER solution and the

experiences of the project personnel (as per D4.3 Pilot and Service Methodology Report):

• Pilot users: impact questionnaires (reported in D2.7)

• Pilot users: usability questionnaires (reported here in D4.6)

• Interviews and focus groups with carers and the older people they care for (reported in

D2.7)

• Interviews with project partners (reported here in D4.6)

• Analysis of system and technical files, logs and statistics (reported here in D4.6)

• Review of lessons learned (reported here in D4.6)

1.4 Structure

This deliverable will be structured as follows, with detailed subheadings given in the contents

section:

• Pilot phase 2 methodology

• Pilot site descriptions and variations

• Pre and post implementation changes

• A summary of the impact evaluation results

• Usability assessment of DISCOVER

• System usage data

• The exit strategy

• Conclusions

• Recommendations

1.5 Intended Audience

This deliverable is intended to be used as a reference document that guides further development

and exploitation of DISCOVER. It also aims to provide lessons learnt from this project that may be

transferable to other similar projects that are piloted across the EU.

of 10

2 PILOT PHASE 2 METHODOLOGY

The Pilot Phase 2 DISCOVER evaluation was undertaken at 4 pilot sites: UK, Netherlands, Spain and

Greece. The Pilot Phase 2 methodology is detailed in full in D4.3.

In summary the pilot lasted 54 weeks from 01/01/2014 to 15/02/2015.

During this time 371 carers and 18 of the older people they care for (total participants: 389) were

enrolled and trained in the use of the system. The country by country break down of participants

is summarised below in Table 1.

Table 1: Number of Participants (carers and cared for people) recruited for Pilot Phase 2 [by country]

Greece Spain The Netherlands* UK

Informal carers 65 67 40 96

Formal carers 40 37 8 18

Cared-for people 6 2 2 8

Total 111 106 50 122

Total number of participants (carers and cared-for people) across all 4 pilot sites: 389

*from Month 31

Data was then collected from carers in several formats:

1. Impact evaluation questionnaire (n= 260; representing a 67% return rate)

2. Usability questionnaires (n= 207; representing a 53% return rate)

In addition to the data collected directly from end-users the following were undertaken:

• semi-structured interviews were undertaken with pilot site leads (n=4)

• system data was captured to understand how the service was being used

• lessons learnt were captured through a series of interviews and data collection exercises

during project member workshops

of 11

3 PILOT SITE DESCRIPTIONS & VARIATION

As the project evolved, it was clear that the situation and the nature of the caring population in

each of the pilot sites countries (UK, Netherlands, Spain and Greece) differed. The needs and

requirements of the intended end-users and the way in which the service would be used varied.

Therefore, the DISCOVER service and content was adapted to local country context and

requirements.

This section aims to summarise the differences in the pilot sites and provide an overview of the

main contextual points at each. It aims to show where there were content differences in the

DISCOVER service and highlight differences in the piloting approach. As a result of the site by site

differences, much of the data collected has been considered on a country basis, rather than being

combined together. The business plans are specific to each country pilot site taking into account

local context.

Tables 2-7 provide an overview of the cultural context within which each pilot has been run.

Deliverable D2.7 provides a fuller overview of the cultural differences and local variations

observed throughout the project. This deliverable (see section 8) will describe key learning points

from the pilot findings, including those stemming from localised differences.

Table 2 Pilot sites comparison: National context & similar services available UK Spain Netherlands Greece

National context

• Healthcare in the UK is mainly

provided by the National Health

Service (NHS), and formal care is

largely provided by statutory

services.

• Anyone can receive up to 6

weeks’ funded in-home support

for reablement or enablement

purposes which is usually

provided by a Local Authority.

• Formal care is funded from

personal budgets and

governmental allowances e.g.

attendance allowance.

• Informal carers can gain support

from several local and national

organisations such as Carers UK,

Age UK, Carers Trust and others.

• There are a number of local

charities and organisations within

the Birmingham region where

piloting took place (e.g. Dementia

Information and Support for

Carers, Mencap, Birmingham

Carers’ Hub, Birmingham City

Council, My Care in Birmingham).

• There are laws and rights for people

with disabilities in Spain but no

comprehensive approaches to

support carers.

• Services are provided to the

community by residential homes,

day care centres, home assistance

services, multiservice centres

• Formal care is funded from personal

budgets and governmental

allowances

• INTRAS is a non-profit organization

specialising in mental health and

ageing

• It focuses on co-development,

assessment & validation of services

& technological solution for users, to

address their existing and emerging

social and health care needs.

• It consists of 9 centres in 6 different

provinces in Spain with more than

90 psychiatrists, psychologists and

professionals from social and

economic fields

• It delivers services for elderly,

disabled people and carers, whilst

creating models of practice.

• Care in the Netherlands is currently

within a period of significant change.

As of Jan 2015 care will be controlled

by Municipalities of which there are

around 400. It was previously the

responsibility of national

government and national funded

institutions.

• The providers of care, welfare,

housing and labour are involved in

this transition. Clients

(organisations), voluntary and civic

organizations will play an

increasingly important role.

• Through the new law solutions will

only be offered to people who are

not able to pay themselves. The

municipalities receive 60% of the

current budget to provide these

customized solutions.

• All aspects of care outside the

responsibility of the Health

Insurance, now fall under the new

system and it is as yet unclear how

this will be rolled out and affect

existing carers and those receiving

care.

• There are a large proportion of

migrant workers within Greece

delivering care, therefore it is a

challenge to recruit due to language

barriers.

• However, it is expected that this

group would not be willing or able to

pay for a service like DISCOVER so it

is considered to be more important

to engage with formal caregivers

who may be representative of future

customers.

Similar services available

• Although there are other services in

the UK providing elements of the

DISCOVER service, such as digital

learning or carer support groups,

there is no known competitor

providing a whole service which

provides all aspects that DISCOVER

offers.

• Informal carers can gain support

from several local and national

charities and organisations such as

regional and national caregivers

associations (e.g. FAFAL)

• However the support provided can

be irregular (e.g. yearly workshops,

standard information on the web).

• A priority in The Netherlands is

anything that can help support older

and vulnerable people to maintain

their mental health and reduce

loneliness.

• Several organisations are delivering

this support, using portals and

websites. None have a Learning

Zone included.

• There are no known services offering

the same type of support to carers in

Greece.

of 13

Table 3 Pilot site comparison – service and responsibilities

UK Spain Netherlands Greece

Short description of the DISCOVER service

• The UK version of the DISCOVER

service has included all teaching and

learning elements including quizzes,

course activities and virtual case

scenarios.

• Activity completion rates are

available via the Moodle admin

facilities

• Quiz attempts/completion are also

available.

• Forums were introduced for Pilot

Phase 1 at the request of UK

participants.

• The Spanish version has included

most teaching and learning elements

including quizzes and virtual case

scenarios.

• Activity completion rates are not


facilities

• Quiz attempts/completion are

available.

• The Dutch version includes the

portal and the Learning zone

subjects to find relevant websites

using Moodle

• The Dutch version of the DISCOVER

service has included limited/no

teaching and learning elements, with

the removal of quizzes and course

activities.

• Virtual case scenarios remain and

have been well received.

• No course activity or quiz

completion statistics are available

via Moodle.

•

• The Greek version of the DISCOVER

service has included all teaching and

learning elements including quizzes,

course activities and virtual case

scenarios.

• Activity completion rates are not


facilities,

• Quiz attempts/completion are

available.

• Forums were introduced for Pilot

Phase 2 at the request of Greek

participants.

Partner responsibilities during the pilot

• Partner responsible for training

carers: OU/Digital Birmingham

• Partner responsible for

recruitment/carer engagement:

OU/Digital Birmingham

• Tech support: BCU

• Maintenance: OU


carers: INTRAS



INTRAS, with the support of

stakeholders organizations such as

AFACAYLE Alzheimer Association


• Maintenance: INTRAS


carers: ASTRA



ASTRA/ stakeholder organisations.

• Tech support: BCU, ASTRA

• Maintenance: ASTRA

.


carers: AUTH



AUTH


• Maintenance: AUTH

of 14

Table 4 Pilot Site Comparison: Target group(s) for the pilot and for the service post pilot


Target group(s) for the pilot and for the service post pilot

• The UK pilot was aimed at both

formal and informal carers

• The aim was to reach 75% informal

and 25% formal carers. Due to the

recruitment method and priority on

reaching informal carers the actual

distribution was 84% and 16%.

• In PP1 carers were reached mostly

through carer support organisations

• In PP2 carers were reached through

carer support organisations, word of

mouth and care homes. In addition

the Citizen Panel market research

database was used to ensure the

target of 100 carers was achieved.

• Moving forward will be marketed

primarily at informal carers and the

care home market within the formal

carers sector (see D5.3 for more

details).

• The Spanish pilot was aimed at both

formal and informal carers, and

there was a roughly even split

between the two.

• Initial feedback (PP1) was obtained

from formal carers who were known

to INTRAS and had a relatively high

level of computer literacy.

• In PP2 recruitment processes were

adapted to use gatekeepers to reach

carers who were not existing

contacts of INTRAS.

• The level of computer literacy of

these carers was lower although still

perhaps not representative of a

typical carer.

• The Dutch pilot was initially aimed at

stakeholders (in PP1) and some

carers in order to discuss the

concept of the DISCOVER service

whilst the platform was in

development.

• In PP2 carers were recruited through

snowball sampling via the

stakeholders, and represented a mix

of formal and informal caring

perspectives.

• The Greek pilot was aimed at both

formal and informal carers; there

was more success in reaching formal

than informal carers.

• Recruitment and retention was

largely successful, with 105 carers

completing the usability

questionnaire.

• Carers were reached via the

University , through the Alzheimers

Association and Related Diseases

Association

• DISCOVER was advertised at

workshops for other projects to

increase exposure; carers were also

reached through their continuing

education on other university

courses

• The type of people who would be

interested in paying and joining a

course would not be the immigrant

carers, so it is accepted that these

were not the carers reached in the

pilots.

•

of 15

Table 5 Pilot Site Comparison: Pilot running


• Participants were recruited with the

help of a number of organisations

e.g. Carers Centres, Mencap,

Birmingham Citizen’s Panel, AgeUK

and were based in Birmingham and

its surrounding areas, and Milton

Keynes. For more detail see D2.6.

• DISCOVER was demonstrated to

participants by OU and BIRM

partners, and participants were then

encouraged to try the service for

themselves.

• Some sessions were delivered in

groups and some on a one-to-one

basis.

• During PP2 all participants were

given an individual login however,

some used an admin login.

• There was no further face to face

training following the induction

session. Participants were asked to

access the Skills Zone from home.

OU contacted participants by email

to encourage further use.

• Carers accessed DISCOVER via group

workshops where they were guided

through the service and the various

features week by week.

• The workshops were delivered over

10 weeks but none of the carers

attended every session. There were

approximately 5 to 8 participants per

session.

• Face to face support and

engagement was necessary to keep

carers motivated but this was

challenging.

• Caregivers were recruited from

interventions provided for those the

participants were caring for as well

as other activities such as workshops

for carers

• They were also recruited through

dissemination and project exposition

in different regional events involving

and targeting carers.

• DISCOVER was presented to

stakeholder groups during

discussions generally about the

provision of carer support in the

Netherlands.

• Stakeholders were shown the

service and encouraged to sign up

for themselves when the site was

ready.

• Carers were guided through the

process with remote support from

ASTRA when needed.

• Support was provided by telephone

and email for those who chose to

create accounts.

• Carer feedback was obtained by

telephone interviewing all individual

testers of DISCOVER who were

available, able and willing to be

interviewed.

• ASTRA used the project protocol

with questions added to obtain

participants’ opinion about the

potential use of DISCOVER

components available in the UK, but

not implemented in The Netherlands

• DISCOVER was run through specific

workshops – system is demonstrated

and initial steps are taken e.g.

signing up.

• All participants attended a workshop

and all had individual logins.

• The master login was demonstrated

and then participants signed up to

the service

• If they did not have an email

address this was created in the first

session.

• Some carers used DISCOVER

between sessions, some waited for

the second workshop to use it.

• Generally the questionnaires and

evaluation materials were combined

with the delivery sessions to

facilitate better response rates.

• Within each workshop there were

between 10 and 20 participants in

order to allow everyone to take part

in a manageable way.

• Some users continued to use it in

their own time

of 16

Table 6 Pilot Site Comparison: Pre- and Post-Implementation- Changes during pilot implementation


• Though there was a good level of

engagement with carers and

participants showed enthusiasm,

they were less engaged with the

process of the pilot and continued

participation.

• Time was a recurring issue; with

carers feeling they didn’t have time

to complete questionnaires or in

some cases use the service.

• About half of the carers completed

the impact evaluation questionnaire

through a variety of approaches:

paper and electronic versions face to

face contact, and by phone

• Reminders were given using

telephone and email.

• Carers who felt unable to complete a

questionnaire were asked key

questions by phone, which increased

the richness of the data.

• Carers were supported during the

pilot through regular email and

phone contact. They were offered

face to face sessions.

• Many were not able to leave the

people they cared for so take up

after the ‘getting started workshops’

was low.

• Initially (Pilot Phase 1)

experimentation and end-users

analysis focused on expert end-users

given their previous collaboration

testing other technologies,

approaches, or those with medium

or advanced digital abilities.

• For the second pilot (considering the

technology and approach advances

& improvements offered at this

stage) real users with very

heterogeneous profiles and digital

abilities took part.

• 77 carers completed the impact

evaluation questionnaire through a

variety of approaches: paper and

electronic versions face to face

contact, and by phone

• Some new collaborations were

established with recognised

gatekeepers supporting the project

by signalling potential interested

read-users with which the pilot team

established contact. In the PP1

participants were mostly recruited

directly by INTRAS considering the

stage of the intervention prototype.

• Dutch carers expressed that they do

not wish to partake in structured

learning, but would rather find

information related to their needs at

their own pace.

• The DISCOVER service was adapted

in The Netherlands and teaching and

learning tools, and reference to

learning removed; the learning zone

was renamed the Knowledge

Database

• The original plan for the Dutch

evaluation was to build face to face

groups and foster a community

engagement approach in order to

encourage carers to engage with

DISCOVER.

• There were challenges faced with

this approach so the methodology

was adapted to include distance

participation; carers were reached

remotely and ‘virtual groups’ were

created.

• The evaluation activities were largely

completed by telephone.

• Dutch carers are reticent to

complete questionnaires and prefer

to give feedback via qualitative

means. This was reaffirmed by the

low questionnaire completions

• The Greek pilot developed specific

cases in other systems (for example

Open Labyrinth virtual patient cases)

• This provided a more experiential

case based learning approach.

• Some of these also translated into

Second Life but this could not be

financially maintained (300 EURO

per month) so Open Sim is now

being used instead but certain

aspects need reworking.

• This is more sustainable as there is

no cost – this has only been

developed in Greece presently.

• Users cannot run this on their own

when they first use the service, as it

needs a powerful machine to run.

• Initial feedback suggests that carers

like this approach, especially the

younger carers.

of 17

Table 7 Pilot Site Comparison: Future development and business plans


Future plans for development

• DISCOVER will be used to strengthen

the existing work of BCU with

regards to course delivery.

• It is clear that the service must be

free at the point of delivery, as

carers do not have the means to pay.

• elearning is an important

component of the service, though

some carers will require support to

use the Skills Zone initially.

• In the UK BCU will continue to

maintain the platform.

• Personalised/customisable content

• Feedback suggests that mapping

services and providing information

about non-care-related services

would be of interest to people.

• Promoting national and regional

awareness about the importance of

caring for caregivers

• Promote awareness of how the

needs of rural caregivers can be met

by ICT solutions – continue to open

the eyes of the Goverments to

ensure a good internet broadband in

such areas

• DISCOVER is seen as a tool to allow

people to find information about

subjects relevant to them (e.g.

dementia) without being

overwhelmed by the number of

search results returned by Google.

• Users show a willingness to continue

to use DISCOVER

• Should be customisable allowing

users to suggest or create content

for themselves.

• Teaching and learning can exist but

in the form of embedded learning,

so that it appears to be more than a

learning tool.

• For people to carry on using

DISCOVER after the life of the

project, they should be able to

create content and maintain profiles,

scoop.it topics and so on.

• More content is required to there is

continuation for existing users of the

site.

Business plans (see D5.3 for detail)

• At this stage it is envisaged that BCU

in collaboration with Birmingham

City Council will host, maintain and

develop the DISCOVER platform.

• In the short term these costs will be

absorbed by BCU. Birmingham City

Council has the contact with

potential users that will allow

effective targeting of formal carers

through partners CM2000 and NCC

Resources.

• DISCOVER will be free at the point of

use

• Currently it is assumed that people

will not pay for DISCOVER.

• DISCOVER would be best placed as a

component of a training course, e.g.

a dementia course.

• This somewhat reduces the risk

associated with the business plan by

embedding DISCOVER within a

larger offering as a resource to use

between sessions, and a means of

accessing information

• The service must mainly support

carers in their care provision – the

priority for Dutch carers is to be able

to give care as good as possible, and

as a secondary priority to be able to

improve their own lives.

• It is also important not to forget the

needs of the cared for, so the

community approach will be

important in the business plan.

• One of the plans in response to this

is to release DISCOVER as a life long

learning scheme or module and

combine it with other courses, and

consider accreditation

• In the form of CPD, or element of an

undergraduate course involving

blended learning, seminars and

some activities over distance

Maintain and advertise effectively.

• Gain university approval (see D5.3

for further details).

4 CHANGES BETWEEN PILOT PHASE 1 AND APPLIED TO PILOT PHASE 2

Pilot Phase 1 aimed to inform the iterative development of the DISCOVER service, and to develop and test

the pilot methodology. The results are summarised in D2.6 (Impact Evaluation) and D4.5 (Service

Evaluation and Adaptation Report). In this section we summarise some of the changes to the DISCOVER

service that were made as a result of the Pilot Phase 1 results reported in D4.5.

4.1 Changes to the DISCOVER service (reported in D4.5)

Changes that were made to the service content or appearance as a result of feedback from users as

reported in D4.5:

Feedback from

users

Suggested Changes Implemented changes

Font size Increase the font size and keep font sizes

consistent. Ensure there is a way for users to

manually adjust the font size – if this is already

in place, ensure that it is made obvious how to

do this.

In the UK the site was developed according to

Accessibility guidance W3C Level A and many

features reach Level AA. This is being adopted

by the other country services in due course.

Navigation Users found the site difficult to navigate –

ensure that menus are clear and simple, and

keep a ‘breadcrumb trail’ to allow users to

return to their starting point.

The navigation has been revisited at each site.

There are country variations on the

transparency of the navigation and depth of

the structure which are being considered

locally.

Help function The help function was seen as useful but not

always able to assist. If users submit a

question, ensure that it is answered and where

possible, add to the help content periodically

to keep it up to date.

A help button was added on the right hand site

navigation to make the access to help more

transparent, the help content has been

improved

Login requirement Introduce a ‘quick-links’ function to allow some

content to be seen without a login. Some users

do not wish to register with DISCOVER in order

to access information, so allow some to be

open content.

It was important to maintain the login

requirement for tracking of usage during the

pilot. This is an issue that will be revisited on a

country by country basis depending on the

business model to be adopted going forward.

These changes relate to those recommended in D4.5. There are wider content changes to the system and

local variations that are detailed elsewhere as a result of Workpackage 3 (Service Adaptation and

Customisation).

of 19

4.2 Suggested changes to the pilot project methodology

Changes that were made to the project methodology on the basis of feedback or challenges reported in

D4.5:

Challenge/limitation Suggested Changes Implemented changes

Consider the

implications of 4 sites

and 4 methodological

approaches

The results indicate that DISCOVER varies in

content and usability at each site. Equally the

implementation of the evaluation approach

has varied at each site. The project should

consider whether the approach will be more

aligned for pilot phase 2 allowing direct site

comparisons to be made.

Research during the project into local context

and needs highlighted the importance of

solutions that are tailored to local needs and

delivery model. In Pilot Phase 2 there was

effort directed at capturing and clarifying the

differences between sites, rather than trying

to develop a singular unified approach.

Questionnaire

numbering (scale)

Usability questionnaire Visual Analogue scale

numbering was confusing to some

participants – reverse this so that 10

becomes strong agreement, 0 becomes

strong disagreement.

The questionnaire scale was reversed so as to

reduce confusion.

UK – giving

questionnaires out to

take away/telephone

follow-up not

successful

More support is needed for UK participants

to return questionnaires, as telephone

follow-ups had limited success. Suggested

change is to add more narrative to the

questionnaire explaining its purpose and

reduce the overall length of the

questionnaire.

The overall length of the usability

questionnaire was reduced, with open-ended

questions kept to a minimum and more

narrative added to explain the purpose of the

questionnaire. Assistance and support was

provided by the pilot personnel.

Translation Technical support team and local pilot staff

to check local sites for translation issues –

some content remaining in English and some

only partially translated.

Translation issues were identified and

rectified as an ongoing task within the pilots.

Recruitment Where possible, aim to introduce groups of

carers face to face before they begin to

network online, as Dutch feedback suggested

that this would increase engagement with

the service.

Aim to recruit not just from carer

associations, but a wide range of caring

experience. Aim to recruit those with a

spectrum of digital skills and experience.

Face to face carer groups were challenging

due to limited availability on the part of

carers and the high numbers required in the

pilot phase 2 sample. Some groups were

achieved however methods were adapted to

include increased telephone contact and

one-to-one interviews.

Digital skills were considered to be more

representative of the target group in pilot

phase 2, although still showed relatively high

levels of digital skills suggesting that the very

digitally illiterate may not have been

represented.

Retention Ensure that support is available to

participants throughout the pilot phase,

including support to complete questionnaires

but also support when difficulties arise using

the DISCOVER platform.

Pilot personnel support was provided via

telephone, email and face to face contact.

Depth of data The initial data from pilot phase 1 indicates

that there was some disagreement that

digital and caring skills had been increased,

and that participants were not using

DISCOVER widely on mobile devices. To

The question ‘what are your favourite/least

favourite things about DISCOVER’ was

changed to ‘what was difficult/easy about

using DISCOVER’, to gain more usability-

focused data.

of 20

ensure that this is not due to usability issues,

the questionnaire may need to be adapted

for phase 2 to explore these issues further.

It is suggested that the question ‘what are

your favourite/least favourite things about

DISCOVER’ is changed to ‘what was

difficult/easy about using DISCOVER’, to gain

more usability-focused data.

Crossover of impact

evaluation and usability

Define clearly which aspects of the service

are to be evaluated in which questionnaires,

to reduce the overall questionnaire sizes by

removing duplication.

Questionnaires were adapted to remove

superfluous or repetitive questions.

Length of usability

questionnaire

Questions to be revisited to reduce the time

taken to complete the questionnaire. More

guidance for questions may be required to

reduce confusion with Likert scales and ease

completion (thereby further reducing

completion time).

Length of time taken to complete the

usability questionnaire was reduced by

making the changes previously mentioned,

which reduced the overall number of

questions and thus the completion time.

System data Revisit system data logs from moodle to

determine how these can be used as an

evaluation tool – a more realistic and

achievable plan must be put in place to use

this data more effectively if possible as it has

the potential to support the usability

evaluation data by placing responses in

context of usage levels.

The use of Moodle does not readily permit

the analysis of system data, however

methods of doing so were explored for pilot

phase 2 and some system data has been

acquired and reported.

Language used in

questionnaires

Questionnaires to be reassessed for language

and clarity, as informal feedback suggested

that some of the language was too technical

e.g. participants not understanding terms

such as ‘social media’. Non-English pilot

personnel to reassess the translation of

questionnaires to ensure that question

meaning is not lost in translation.

Fuller explanations were added as necessary,

and confusing questions were removed from

the questionnaire.

4.3 Changes to Pilot site delivery (informal observations)

Table 6 above details some of the changes that were made to pilot running at each site. At a global level

the pilot was altered in the following way following learning from Pilot Phase 1:

1. Pilot sites developed relationships with trusted organisations to aid recruitment of carers and

encouraged stakeholders who supported Pilot Phase 1 to recommend other carers to join Pilot

Phase 2. Face to face contact was valued most, then email and telephone.

2. Face to face focus groups and training workshops for carers as potential users, were vital to

encourage initial engagement with DISCOVER, as well as to further sustain their engagement with

the service. For example, in Spain, carers recruited for Pilot Phase 2 were offered a

of 21

space/workshop weekly to interested carers with low digital abilities with the objective to

reinforce their self-esteem and ability to be autonomous in the use of the Skills Zone.

3. In Kerkrade, in the Netherlands, the majority of the informal (and even formal) carers do not know

anything about ‘computers’ and are averse to using it, according to the focus group and according

to phase 1. So for Pilot Phase 2, carers invited to join groups that are mainly offering things they

dearly want: meet each other, get answers to difficult questions they have, find experts when

needed. This exercise has been named ‘Leuk voor Elkaar’ (Being nice, kind & compassionate) to

one another. The DISCOVER Skills Zone was presented to all ‘Leuk voor Elkaar’ groups, together

with other digital resources and applications.

4. Participants were recruited via gatekeepers local to the pilot site organisations and through direct

approaches.

5. Carers were reached via flyers and leaflets, telephone invitation, social media, personal invitation

by trusted contacts, and word of mouth.

6. Due to the larger sample, the main differences between Pilot Phase 2 and Pilot Phase 1 have been

focused on the monitoring and evaluation methodology. The key points on pilot procedure were

specifically:

o collection of the questionnaires (impact and usability)

o 2 focus groups (formal and informal carers participated together) were led by a

psychologist and content manager, involving 7-8 carers (at the end of Pilot Phase 2)

o follow-up activities on a weekly basis, mainly face-to-face and/or phone contacts.

o Big Blue Button teleconference platform was used with more experienced users (Spain

only)

o internal analysis of feedback provided by caregivers

The development of DISCOVER has been iterative with end-user involvement to ensure that the service

meets the needs of the intended user-group. The changes to both the service and the pilot approach

between Pilot Phase 1 and 2 have been significant to ensure the needs of carers are met for efficient

service delivery. The lessons learnt through both pilots are elaborated upon in more detail in section 8 to

further explain the project progress and learning and also inform future projects in this area.

of 22

5 A SUMMARY OF THE IMPACT EVALUATION

The DISCOVER solution sought to provide learning for carers so that they might improve their digital skills

literacy, care skills, and benefit from online social interaction.

D2.7 provides a detailed Final Impact Evaluation Report and offers a final analysis of the skill and quality

of life gains made by participants using DISCOVER.

In summary the results indicated that:

• Despite feeling very time poor and pressured, many carers learned new skills and gained new

knowledge through engaging with the DISCOVER Skills Zone

• Carers valued finding out about local and national services

• Carers valued leisure activities such as brain games

• Carers joining DISCOVER felt more supported in their caring role

• Carers feelings of wellbeing improved during Pilot Phase 2

• Both informal and formal carers would recommend DISCOVER to other carers and formal carers

would also recommend it to their employers

• Stakeholders saw DISCOVER as a valuable addition to care services currently available that had

potential to improve both care services and the lives of carers and the people they care for.

These are very promising findings for the efficacy of the service. The pressure on carers is significant and

it has been clear that asking participants to find time to commit to the use and evaluation of the service is

challenging for them. However, despite these pressures 371 carers engaged with the service and 67%

found the time to respond to the impact evaluation survey about the system. This represents a good

response rate for survey-based research and is taken to be indicative of the recognised value of

DISCOVER.

It is of particular significance that carers’ feelings of wellbeing improved over the Pilot, so there was not

only perceived skill and learning benefits, but there was a reported positive psychological impact to

service and pilot involvement.

The perceived value of the service from the intended end-user group alongside the willingness to

recommend DISCOVER is significant in terms of seeking future investment for the sustainability of

DISCOVER.

of 23

6 USABILITY ASSESSMENT OF DISCOVER

This section describes the usability testing of DISCOVER. Usability testing is undertaken with

representative end-users of a system to detect any usability problems, and determine the end-users

satisfaction with the product. Poor usability can lead to poor use of a system despite its valuable content,

because it is hard and frustrating to use. This is particularly important to consider when developing a

service for older users who are not familiar with computer-based systems.

A usability survey was conducted in Pilot Phase 1 of the DISCOVER project and resulted in

recommendations for how the system might be further developed to improve usability. This was reported

within D4.5 Service Evaluation and Adaptation Report and these issues are summarised in section 4

above. Here the findings of the second round of usability testing during Pilot Phase 2 are detailed and

differences in end-user perception of usability within each country are explored. This evaluation will lead

to recommendations for future development both of the service and platform to ensure that system

usability does not discourage use.

6.1 Aim

In Pilot Phase 2, the usability evaluation aimed to quantify user perceptions of DISCOVER usability. It was

intended that this feedback could be used to guide further improvement of the system, to report on the

current status of the service, and to inform the business plan by providing guidance on future

developments as suggested by the users.

6.2 Method

The usability survey was developed and approved by Coventry University. The pilot evaluation as a whole

was approved by the Open University Human Research Ethics Committee. The pilot methodology as a

whole was designed by the OU to ensure consistency across the 4 sites. The usability surveys were run

locally by the 4 pilot sites and the results delivered to the HDTI, Coventry University for analysis.

6.2.1 Recruitment

Participants were recruited via gatekeepers local to the pilot site organisations. Carers were reached via

flyers and leaflets, telephone invitation, social media, personal invitation by trusted contacts, and word of

mouth. Recruitment strategies differed by pilot site to adapt to local circumstances, and are discussed in

more detail in each site-specific section. Pilot personnel and project partners who were interviewed took

part voluntarily.

6.2.2 Participants

The number of participants included in Pilot Phase 2 are detailed below in Table 8.

of 24

Table 8 Number of Participants (carers and cared for people) recruited for Pilot Phase 2 [by country]

Number of participants (n= 389)

Greece Spain UK The Netherlands

Informal carers 65 67 96 40

Formal carers 40 37 18 8

Cared-for people 6 2 8 2

Total 111 106 122 50

Not all of these participants took part in the usability assessment of the system as can be seen by the

participant number reported in Table 9. Approximately 53% of all pilot participants completed the survey.

The response rates were very good for Greece, and lowest for the UK.

Table 9 Participant summary per country

Number of participants (n=207)

Greece Spain UK Netherlands

Number completing the survey 105 65 19 18

Male 26 17 7 6

Female 79 48 12 12

Mean length of time caring (years) 4.3 2.7 7.7 9.2

Range (months and years) 1m – 20y 4m – 7y <1y – 39y 1y2m – 20y

Age of participants (years)

18-30 16 7 1 1

31-44 18 18 3 3

45-54 35 19 3 3

55-64 25 13 7 5

65-74 7 5 4 3

75+ 4 3 1 3

A summary of participant characteristics per country is given in Tables 10-12 and described below. This is

based on the data provided in the demographic section of the survey; some participants had chosen not

to complete these questions, therefore these figures are based only on valid responses to the

questionnaires.

of 25

Table 10 Education levels of pilot participants

Number of participants

Highest level of education completed Greek Spain UK Netherlands

No school 0 2 0 0

Elementary/junior school 7 8 3 0

Secondary/high school 31 22 6 6

Higher education 57 26 10 12

Other 8 6 0 0

Number of participants with formal care

training 26 1 1 5

Table 11 Location of caring role for pilot participants


Where do you conduct care-related work most of the time? Greek Spain UK Netherlands

At the home of the person you are caring for 66 30 18 12

At a residential care home 6 7 0 1

At a care day or drop-in centre 9 0 0 1

For trips only, e.g. Helping with shopping for socialising 8 1 1 0

Did not answer 16 0 0 0

Table 12 Use of DISCOVER


Do you use DISCOVER: Greek Spain UK Netherlands

At home 75 35 9 12

Under supervision e.g. in a training session or workshop 18 0 2 1

Both of the above 12 0 4 1

Neither – I have seen DISCOVER but not used it 0 6 1 0

Greece:

105 participants completed the usability questionnaire in Greece. Of these, 26 were male and 79 were

female. 95 participants were currently caring for someone, and 8 were not at the time of data collection

(two participants did not divulge this information). 26 participants had formal training as a carer. Length

of caring ranged between 1 month and 20 years, with a mean of 4.3 years, indicating a wide range of

caring experience. Table 10 gives the highest level of education attained by the sample, and indicates that

there was a large proportion of University students or graduates within the sample. There were a wide

range of ages in the 18-64 range, with fewer participants in the older age brackets (see Table 9). Most

participants were responsible for caring at the home of the person receiving care. The mainly used

DISCOVER at home (n=75).

of 26

Spain:

65 participants completed the survey in Spain. 17 were male and 48 were female. 56 participants were

currently caring for someone, and 9 were not at the time of data collection. 29 participants had formal

training as a carer, 25 did not. Length of caring ranged between 4 months and 7 years, with a mean of 2.7

years, indicating a wide range of caring experience alongside a reasonably even mix of formal and

informal carers. The majority of participants had completed either secondary education or University

education. There were a wide range of ages in the 31-64 range, with fewer participants in the older age

brackets (see Table 9) and the 18-30 range. Most participants were responsible for caring at the home of

the person receiving care. They exclusively used DISCOVER at home (n=35) if they had used it at all,

whereas 6 participants reported that they had only seen DISCOVER but not used it. In Spain DISCOVER

evaluation was largely facilitated in workshops, and there is missing data from this question, it can be

assumed that a substantial proportion of those who did not respond either to this question or to the

questionnaire itself used DISCOVER under supervision.

UK:

19 participants completed the survey in the UK. 7 participants were male and 12 were female. 19

participants reported that they were currently caring for someone. One participant had formal training as

a carer, 4 did not. Length of caring ranged between 1 and 39 years, with a mean of 7.7 years. 10 of the

sample indicated completion of Higher Education, including vocational qualifications. Participants were

spread evenly across age ranges (see Table 9). Most participants were responsible for caring at the home

of the person receiving care. The participants reported using DISCOVER at home (n=9), with 2 participants

using it under supervision and 4 in both settings.

The Netherlands:

18 participants completed the survey in the The Netherlands. 6 were male and 12 were female. All 18

participants were currently caring for someone at the time of data collection. Five participants had formal

training as a carer, and 13 did not. Length of caring ranged between 1 year and 2 months and 20 years,

with a mean of 9.2 years. Those who completed the usability questionnaire had reached either secondary

or higher education. There were a wide range of ages, with participants representing all age brackets (see

Table 9). Most participants were responsible for caring at the home of the person receiving care. They

mostly used DISCOVER at home (n=12)..

6.2.3 Design and Procedure

Usability evaluation questionnaires were developed by the Health Design & Technology Institute (HDTI,

Coventry University) and AUTH, and then agreed with wider project partners. The questions were based

on established usability principles such as error rate and correction, navigability, learnability and

accessibility. See appendix A for example.

of 27

The questionnaires contained 35 questions relating to the carers’ computer literacy, the ways in which

they had used the DISCOVER service, how satisfied with the service they were, how easy they found it to

use, their feedback on usability principles such as accessibility and navigation, and their suggestions for

improvement and future development. Questions relating to ease of use and satisfaction were answered

using a 10-point scale where a higher numerical score indicated a more favourable opinion. Questions

relating to suggestions for improvement were open-ended allowing participants to give as much or as

little information as they wished. All 35 questions were optional.

The usability questionnaire had been updated following feedback obtained in pilot phase 1, notably that

the 1-10 scales were confusing and that the anchors should be inverted, and the observation that the

open questions were not providing clear feedback. Scales were reversed and closed questions were

substituted for some of the open questions, to reduce the overall length of completion and provide

clearer feedback on some of the usability indicators.

6.2.4 Analysis

The data was analysed on a country by country basis due to the differences in the service available and

the way in which the pilots were run. The sample sizes completing the survey also differed from country

to country. As this was an exploratory survey investigating perceptions, descriptive statistics rather than

tests of statistical inference were undertaken. As is appropriate with ordinal data making use of Visual

Analogue scales, distribution free methods such as a frequencies, and medians were calculated.

of 28

6.3 Results by Country

In the following section the results of the usability evaluation are described for each pilot site. The tables

and graphs report the median scores given on a visual analogue scale where 1 represented a negative

score and 10 represented a positive score.

6.3.1 Greece

The Greek data is summarised below and graphically in Figures 1-3:

Ease of use/ ease of learning

• The ease of learning scores were towards the higher end of the rating scale which indicated positive

feedback

• Users could easily find the information that they required, and could use DISCOVER successfully every

time.

• The help function and the ease of correcting mistakes were relatively less highly rated, but still scored

well overall.

Errors

• No errors were reported via the usability questionnaire and no suggestions for improvement were

offered.

Satisfaction

• Satisfaction levels for DISCOVER were generally in the higher range (7-10) and suggest that the Greek

participants were happy with the service.

• The most positive scores were for the desire to continue to use the DISCOVER service

• Participants would recommend the service to other people.

• Participants’ knowledge about their caring role had been improved as a result of using DISCOVER.

• A lower average score, was given to the improvement of carers’ ability to use computers.

• Though computer literacy improvement was relatively less well rated, it was still rated in the higher

end of the scale

• Higher ratings were given for carers’ enjoyment of using the service and perceived beneficial effects.

• Though still in the higher end of the scale, the relatively less positive scores related to communicating

with others and a willingness to pay for the DISCOVER service which was the lowest score and closest

to the mid-scale.

• Fourteen participants had used some form of social networking since using DISCOVER, which they

reported to mostly be Facebook.

• One of these participants had also used Viber since using the DISCOVER service, and one reported

using online forums, though did not specify which.

of 29

123456789

10

I could

easily find

the

information

I needed to

get started

with

DISCOVER

I was able

to find the

help

function on

the system

when I

needed it

The help

function

provided

the

solutions

when I had

difficulties

I could use

DISCOVER

successfully

every time

It was easy

to find the

pages or

sections I

wanted to

look at

The

instructions

in

DISCOVER

were easy

to follow

When I

made a

mistake, it

was easy to

correct or

undo

It was easy

to track my

learning

progress

and see

what I had

learned so

far/what I

had left to

learn

It was easy

to pick up

where I had

previously

left off

I liked the

visual

design of

the

DISCOVER

system

I wa able to

create an

account to

use

DISCOVER

without

any

problems

Figure 1: GREECE - Median ratings of agreement to ‘ease of learning’ questions

(1 =negative score and 10 =positive score)

1

3

5

7

9

How beneficial

do you think

DISCOVER is for

you?

How enjoyable

was it to use

DISCOVER?

DISCOVER has

met my

expectations

Using DISCOVER

has improved

my ability to use

computers

Using DISCOVER

has improved

my professional

care skills

The text on the

DISCOVER

website as at a

size I could

easily read

The buttons on

the DISCOVER

website were

labelled clearly

The colours

used in the

DISCOVER

website did not

cause me any

problems

Figure 2: GREECE - Median ratings of agreement to satisfaction related questions


1

2

3

4

5

6

7

8

9

10

DISCOVER has helped me

to communicate with

others more

DISCOVER has improved

my access to information

and knowledge about my

caring role

I would continue to use

DISCOVER if it was

available

I would be willing to pay

for DISCOVER

I would recommend

DISCOVER to other people

Figure 3: GREECE - Median ratings of agreement to

satisfaction related questions (1 =negative score and 10 =positive score)

of 30

6.3.2 Spain

The Spanish data is summarised below and graphically in Figures 4-6:

Ease of use/ease of learning

• Responses to the ease of learning statements were consistently in the mid to high range, indicating

that there was moderate agreement with the statements

• There was also moderate agreement with the statement ‘I could use DISCOVER successfully every

time’, indicating that participants were able to use the system easily.

• Participants mostly agreed on the effectiveness of the help function, and for those who tried to access

it (n=24), there was agreement that this function was easy to find.

• There was further agreement regarding the ability of the help function to provide solutions to the

problems encountered.

• Agreement was slightly lower (though still towards agreement rather than disagreement) regarding

the ability to self-correct mistakes, suggesting that participants attempted to correct their own errors

and were more able to do so.

Errors

• Six participants stated that at some point during their use of DISCOVER they had experienced a page

error.

• Of those who reported this, one was able to easily navigate away from the error or correct it,

however five were not.

• Those who had experienced errors were asked to describe them to the best of their ability; two

participants’ issues related to the content they were seeing being in English and therefore they could

not understand it.

• The remaining errors were temporary page loading or access problems, and issues using the VCC.

Satisfaction

• Responses questions relating to satisfaction tended toward a positive response.

• The higher ratings (indicating a more favourable opinion) suggest that participants found the service

enjoyable to use, and that it had improved their access to information related to their caring role.

• Usability ratings showed agreement that the design of the site did not cause any problems.

• Participants moderately agreed that DISCOVER had improved their professional care skills

• Participants disagreed that the service had improved their computer skills (see Figure 5).

• Willingness to continue using DISCOVER and to pay for the service was a source of moderate

agreement

• Moderate agreement with willingness to recommend the service to others.

• DISCOVER had improved access to knowledge about participants’ caring role, but there was no

agreement nor disagreement that DISCOVER had improved communication with others.

• Four participants reported using social media since learning to use DISCOVER, the remainder had not.

of 31

123456789

10

I could easily

find the

information I

needed to get

started with

DISCOVER

I was able to

find the help

function on

the system

when I needed

it

The help

function

provided the

solutions when

I had

difficulties

I could use

DISCOVER

successfully

every time

It was easy to

find the pages

or sections I

wanted to look

at

The

instructions in

DISCOVER

were easy to

follow

When I made a

mistake, it was

easy to correct

or undo

It was easy to

track my

learning

progress and

see what I had

learned so

far/what I had

left to learn

It was easy to

pick up where

I had

previously left

off

I liked the

visual design

of the

DISCOVER

system

Figure 4: SPAIN - - Median ratings of ‘ease of learning’ questions


123456789

10

How beneficial do

you think

DISCOVER is for

you?

How enjoyable

was it to use

DISCOVER?

DISCOVER has

met my

expectations

Using DISCOVER

has improved my

ability to use

computers

Using DISCOVER

has improved my

professional care

skills

The text on the

DISCOVER

website as at a

size I could easily

read

The buttons on

the DISCOVER

website were

labelled clearly

The colours used

in the DISCOVER

website did not

cause me any

problems

Figure 5: - Median ratings of ‘satisfaction’ related questions


1

3

5

7

9

DISCOVER has helped me to

communicate with others

more

DISCOVER has improved my

access to information and

knowledge about my caring

role


DISCOVER if it was available

I would be willing to pay for

DISCOVER

I would recommend


Figure 6: Median ratings of ‘satisfaction’ related questions


of 32

6.3.3 UK

The UK data is summarised below and graphically in Figures 7-9:


• There was a very positive view of the DISCOVER learning experience.

• Participants were able to find the help function, could find the information that they needed and use

DISCOVER successfully every time.

• Font size, button labelling and colours were rated highly as appropriate and acceptable.

• The slightly lower average scores (indicating a tendency toward relatively lower agreement, though

still at the more positive end of the scale) suggested that correcting mistakes and getting started with

DISCOVER were slightly more difficult than general system use.

Errors

• Four participants indicated that they had experienced a page error at some point whilst using

DISCOVER

• One participant expressed that they found the navigation around the site difficult and could not work

out how to get from one page to the next (though they wondered whether this was due to the

capabilities of their computer).

• One participant felt that people with visual impairments may struggle with the site and suggested

bold type on yellow background as an improvement.

Satisfaction

• In the UK, satisfaction scores were more variable than in some of the other pilot sites.

• Participants were in agreement that the service was enjoyable to use, that it met expectations, and

that participants would recommend it.

• Ratings for how well DISCOVER had helped participants to improve their digital and caring skills and

ability to communicate with other carers demonstrated lower ratings.

• Participants strongly disagreed that the service had helped them communicate with others

• They neither agreed nor disagreed that they would be willing to pay for the service.

• There was agreement that participants would continue to use the service if it were available to them.

• Three participants reported that they had used social media since using DISCOVER

• Open ended questions revealed that there were mixed feelings about the usability and

appropriateness of the DISCOVER solution in the UK e.g. “Very straight forward. Don't know why it's taken so long to go online..”

“Couldn't work out where anything was on the website and how to find your way back again”

• There were few suggested changes made: “It was beneficial at the beginning - used to get access to other relevant websites. “

“Suggested using Streetwise to find out about any local information - such as other carers”

“I already know a lot of the things on Discover. I can't see much benefit for professional carers and think it would be much

better for informal carers. I know about services such as respite services, which I tell my clients about.”

“Unlikely to be using Discover very much in future as felt it just wasn't helpful for me or husband - but could be very helpful

for someone else with different needs to us.”

“Thank you for showing me this wonderful system, I would never have known what info was out in the community”

of 33

123456789

10

I could

easily find

the

information

I needed to

get started

with

DISCOVER

I was able

to find the

help

function on

the system

when I

needed it

The help

function

provided

the

solutions

when I had

difficulties

I could use

DISCOVER

successfully

every time

It was easy

to find the

pages or

sections I

wanted to

look at

The

instructions

in

DISCOVER

were easy

to follow

When I

made a

mistake, it

was easy to

correct or

undo

It was easy

to track my

learning

progress

and see

what I had

learned so

far/what I

had left to

learn

It was easy

to pick up

where I had

previously

left off

I liked the

visual

design of

the

DISCOVER

system

I was able

to create

an account

to use

DISCOVER

without any

problems

Figure 7: UK - Median ratings of 'ease of learning' statements

(1= Negative score; 10= positive score

123456789

10

How beneficial

do you think

DISCOVER is for

you?

How enjoyable

was it to use

DISCOVER?

DISCOVER has

met my

expectations

Using DISCOVER

has improved

my ability to use

computers

Using DISCOVER

has improved

my professional

care skills

The text on the

DISCOVER

website as at a

size I could

easily read

The buttons on

the DISCOVER

website were

labelled clearly

The colours

used in the

DISCOVER

website did not

cause me any

problems

Figure 8: UK - Median Rating to 'ease of learning' statements

(1 = negative score; 10 = positive score)

1

3

5

7

9

DISCOVER has helped me

to communicate with

others more




caring role


DISCOVER if it was

available


for DISCOVER

I would recommend


Figure 9: UK - Median ratings of 'ease of learning' statements

(1= negative score; 10 =positive score)

of 34

6.3.4 The Netherlands

The data from the Netherlands is summarised below and graphically in Figures 10-12:


• There was a relatively less positive view of the DISCOVER learning experience in comparison with the

other pilot sites.

• Participants most highly rated the ability to find the help function and correct mistakes when they

were made.

• Font size, button labelling and colours were rated highly as appropriate and acceptable.

• The mid-range average scores (indicating a tendency toward relatively lower agreement, though still

at the more positive end of the scale) suggested that participants could use DISCOVER successfully

and could find the information they needed.

• There were lower ratings compared to the other pilot sites for how enjoyable it was to use DISCOVER,

how beneficial the service was and whether the service met expectations.

• The lowest rating regarded whether DISCOVER had improved the Dutch participants’ ability to use

computers. Responses indicated that participants disagreed with this statement strongly.

• There was higher agreement that DISCOVER had improved care skills but this was still around the mid

range and therefore is inconclusive.

Errors

• Seven participants experienced difficulties when making an account for themselves on DISCOVER (it is

unclear how many people in total made their own account and how many had an account made on

their behalf).

• When given an opportunity to describe the issues they experienced, the information given was that

the participants were unable to log in. One user described having tried a number of different methods

to log in, all of which failed, and wondered whether cookies may have been causing a problem.

• No participants used the questionnaire to report any page errors.

Satisfaction

• Satisfaction scores were lower than in some of the other pilot sites.

• There was a lack of agreement that the service had increased knowledge of caring skills, or that users

would recommend the service to others.

• Participants declined to indicate whether the service had helped them communicate with others

• They strongly indicated that they would not be willing to pay for the service.

• There was some level of agreement that Dutch participants would continue to use the service if it

were available to them.

• Participants declined to answer whether they had used any social networking sites since using

DISCOVER.

of 35

A narrative usability report was provided by the Dutch pilot team, who had conducted interviews with

users to gain feedback on the system. The main conclusions formed were as follows.

• The content of the Dutch Discover Learning zone is interesting for informal carers and it is well

presented, but is not necessarily different from existing Dutch services.

• The service was usable for those who had some degree of computer literacy already, but those who

did not struggled with the learning zone approach.

• Help and support was needed to allow these less experienced users to access DISCOVER.

• For those who were able to gain access to DISCOVER and assess the content, there were no major

usability issues.

• Some participants felt that they already knew the information being presented to them, and felt that

links to external content e.g. via Scoop.it should be curated by the users and not the team running the

service.

• There was a general view that people preferred information being presented to them by experts and

health professionals via personal contact.

• The falls prevention simulation was seen as interesting but the information was available and

preferred through local courses.

• The simulation of assistive technology in the home, however, was seen as useful.

of 36

1

2

3

4

5

6

7

8

9

10

I could easily

find the

information I

needed to get

started with

DISCOVER

I was able to

find the help

function on

the system

when I

needed it

I could use

DISCOVER

successfully

every time

It was easy to

find the pages

or sections I

wanted to

look at

The

instructions in

DISCOVER

were easy to

follow

When I made

a mistake, it

was easy to

correct or

undo

It was easy to

pick up where

I had

previously left

off

I liked the

visual design

of the

DISCOVER

system

I wa able to

create an

account to use

DISCOVER

without any

problems

Figure 10: DUTCH - Median rating 'ease of learning' statements

(1= negative; 10 = positive)

123456789

10

How beneficial

do you think

DISCOVER is for

you?

How enjoyable

was it to use

DISCOVER?

DISCOVER has

met my

expectations

Using DISCOVER

has improved

my ability to use

computers

Using DISCOVER

has improved

my professional

care skills

The text on the

DISCOVER

website as at a

size I could

easily read

The buttons on

the DISCOVER

website were

labelled clearly

The colours

used in the

DISCOVER

website did not

cause me any

problems

Figure 11: DUTCH - Median Ratings of satisfaction statements

(1= negative score and 10 =positive score)

123456789

10




caring role


DISCOVER if it was

available


for DISCOVER

I would recommend


Figure 12: DUTCH - Median ratings of satisfaction statements

(1= negative score and 10 =positive score))

of 37

6.4 Discussion

There were a number of observed differences across the pilot sites in Pilot Phase 1 that where possible

were responded to in terms of system design and usability evaluation approach that were addressed for

Pilot Phase 2.

6.4.1 Comparison to Phase 1 Usability Results

It is difficult to make comparisons between Pilot Phase 1 and 2 because of the change in the way the Pilot

was run and the composition of the participants groups however some broad conclusions are drawn

below.

In Pilot Phase 2, there was greater consistency in some aspects of the evaluation. For example,

participants in the Netherlands showed more willingness to complete the usability questionnaire and as

such there are comparable data allowing for commentary on the same issues as the remaining pilot sites.

In general, the feedback was consistent across the pilot sites and having the data from the Netherlands

allowed for this consistency to be observed.

The service

Phase 1 results: Currently the DISCOVER service is running in each pilot country, with service users

comprising in the main of evaluation participants and pilot personnel. There are differences in the

structure of the web platforms (in particular in The Netherlands), and partners are curating content

relevant to the needs of their carers.

Overall, the ratings for the ease of use of the system indicate that consistently participants agreed that

they could use DISCOVER, and it met their expectations. Though there were some slight differences (e.g.

the Spanish participants’ ratings were slightly less positive than the other sites), generally the feedback

indicated that participants were able to use DISCOVER without major problems.

Phase 2 results: indicate that the findings largely match those of phase 1, with participants generally

feeling the system was enjoyable to use and met their expectations. Once again, the Spanish participants’

responses were relatively lower and thus less positive than the other sites. There did not appear to be

any major usability issues, and ratings for accessibility were very positive indicating that participants were

able to use the site without visual difficulties.

Despite some reported positive views of the service overall, participants were strongly agreed that they

would not pay for it, and were neither in agreement nor disagreement that their digital skills had

improved as a result of using DISCOVER. Caring skills were mildly rated as having improved. Feedback

indicated that communication with others had not been improved as a result of using DISCOVER.

of 38

Challenges and enablers to usability

Phase 1 results: Participants felt that the content and information available was generally adequate

(though there were some who wished to see additional content or the existing content simplified), and

that it was presented in an interesting and engaging way. In particular, the virtual patients and interactive

scenarios were popular with carers.

Barriers to usability included font size, page layouts and navigation. Users at times found it difficult to find

the content they wished to view, and to find their way back to the homepage. Some consistency is needed

in font sizes and menu items to improve usability.

Phase 2 results: Based on ratings of the usability and accessibility questions, it appeared that difficulties

in this area were resolved for phase 2. No issues were reported resulting from colours or font sizes,

though there were less positive ratings for the ability to find help when it was needed. Additionally the

ability to resolve problems using the help function on the website was not highly rated. Participants were

more positive about being able to correct or undo any mistakes they made, suggesting that from a

usability perspective they were able to use the site successfully. This indicates that the problems

remaining unresolved by use of the help function may not have been usability issues, and may have been

content related. It is unclear from the findings what the exact nature of the difficulties were, as sufficient

descriptions were not provided by participants.

6.4.2 Conclusions

The analysis has been made on a country by country basis because the content / service vary for each. In

terms of responding to these findings, a more qualitative approach would be useful to explore how the

service might be enhanced locally to further usability.

Once again, usability questionnaires were completed at one time point in time only, so there is not

transparent pre- and post-use comparative data. However, usability feedback does not necessarily

require repetition and a comparison between PP1 and PP2 provides insight into changes across the two

iterations.

The findings show broad acceptance of the system with overall positive ratings for ease of use. The Greek

participants were most satisfied with the system and its usability.

The results pick up the following issues for future consideration particularly in respect to the UK,

Netherlands and Spain:

• Lack of willingness to pay for the system by the individual user

• Poor user belief that DISCOVER has improved their ability to communication

• Poor user belief that DISCOVER has improved their ability to use computers

The differences in the way in which the Pilots were run meant that the Dutch and UK participants were

more likely to have short periods of use, e.g. no face to face training in UK (different to Greece and

of 39

Spain). To achieve real improvement in digital skills participants would need to undertake sustained

learning, which should be considered carefully in the future.

It is not clear how participants assessed their own learning within the context of completing this survey.

However it may be that they need a different form of evaluation to reflect on possible skills gained. For

instance, unless a user has undertaken one of the basic digital skills quizzes, they would not feel that they

have learned anything about it although their media literacy may well have improved.

Findings reported in D2.7 indicated that participants were more inclined to use the service to look up an

answer to a question. This fits with the Dutch experience that participants were not motivated to ’learn’

using the system but recognised its value. Therefore the value of the system may be as a quick

information resource rather than in digital/computing skills development.

Whilst the system has good usability and has scored well in terms of impact (D2.7) we need to consider

further how to address user perception in terms of the usefulness of the service and the impact it has had

on their technical skills.

of 40

7 SYSTEM DATA

The Impact Evaluation summary (section 5 ) and Usability assessment (section 6) provide feedback on the

effectiveness of the DISCOVER service from the perspective of the end-users and the results are largely

positive. This data is however reliant on knowledge acquisition and self-reports. In order to add to this,

the Pilot Methodology (D4.3) indicated that a review of system data would be undertaken in order to

provide insight into actual use of the system during the pilot.

7.1 Pilot Phase 1: System Data

The DISCOVER service is run through the Virtual Learning Environment Moodle. It was intended to collect

the following data regarding how the system is being used in respect to the number users; the number of

visits to the site / user; the most popular content access; frequency / patterns of use; learning accessed /

completed; and the number of tests completed / user.

However, it became apparent in Pilot Phase 1 that not all of this information would be available, and

analytics were not readily extractable from Moodle (see D4.5 for details). The use of Moodle as the

Virtual Learning Environment in its current format and version does not allow for as detailed analytics to

be acquired as intended. As this was not possible an amended plan for a limited analysis of system data

was developed for Pilot Phase 2.

7.2 Pilot Phase 2: System Data

Since the completion of Pilot Phase 1, different mechanisms for extracting the system data have been

explored. The data that has been extracted is as follows:

• Number of registered users on each course component of DISCOVER

• Number of quizzes attempted and completed

• Number of course activities completed (this was available for the UK only)

• Page views for the virtual case scenarios

• Number of posts made to forum pages

• Use of the help function

It was not possible to extract a clear database of all system data because of limitations within Moodle.

Although statistical data is theoretically available via back-end admin access in Moodle itself, it is

unreliable. Attempts to download this database failed on numerous attempts due to technical issues

associated with Moodle for which there were no solutions readily available.

7.2.1 Data capture and reliability

The administrator access to Moodle allows the observation of registered user numbers on each

component page within the site. These numbers include all registered users and therefore are inclusive of

project team members, spam accounts, facilitator accounts, and accounts which have not been used to

log into the service.

of 41

Accounts were largely provided by CETEMMSA to the pilot leads, to be then provided to participants for

use. Participants also had the opportunity to use an online form to create their own account but reports

from pilot leads suggest that this option was rarely taken.

Each system page within the learning zone was accessed to establish the usage figures for that page on a

country by country basis. These are given in Table 13.

These figures give a rough indication of the numbers of people accessing DISCOVER. Data reliability has

however been affected by a number of factors. It is known that at some stages of the project there were

users logging in via the generic facilitator accounts therefore this data is somewhat unreliable in

determining system use. So extraction of the system data was challenging due to Moodle limitations, lack

of clarity in the data (which were genuine users and which were not), and the lack of ability to filter by

dates and times.

These figures may also have been compromised due to some attacks on the site during the life of the

project, but are unlikely to have altered the results significantly as the findings are based as far as

possible on active use of the site.

Quizzes as learning activities were not made available in all sites (The Netherlands opted not to include

learning activities such as quizzes) however the Moodle administration allows for the observation of the

number of attempts and completions of the quizzes. Again, this includes all facilitator accounts and as

such as not precisely representative of carer use.

Course activities were available in the UK and the Moodle administration allowed for the capture of these

figures via a downloadable spreadsheet. This was not an option on the system for any other pilot sites

therefore are reported only where the option was available.

7.3 Results

Table 13 provides an overview of the usage figures for each of the local sites within DISCOVER. This data

was captured in January 2015.

The main courses within each service are listed on a country by country basis. The table then indicates

the number of registered users to the content, the quizzes and activities completed.

There are a number of gaps and n/a in the table. This indicates that the content was not available for that

pilot site. Quizzes for example were not available to the Dutch participants due to their expressed

reticence to complete formalised learning activities. Activities completed statistics do not appear to be

available for all areas of the site, though it is not clear why. This is a Moodle option within the

administration view and was available only for two areas of the site.

Table 13 System use statistics for each pilot site learning zone component

Course Registered Users

Quizzes

attempted

Quizzes

completed

Activities

completed

Views -

Assistive

technologies

in the home

(VCC page)

Views -

Reducing

the risk of

falls (VCC

page)

Views -

Dementia

care (VCC

page)

Forum

topics

UK

Using the web 231 189 148 113 3

Connecting for care 202 31 28 21 1

Enhancing care skills 196 n/a n/a n/a 257 670 165 n/a

Making your skills count 189 n/a n/a n/a n/a

Spanish

Uso de la web 71 234 161 n/a n/a

Conectar a los cuidadores 47 7 5 n/a 1

Mejorar la capacidad del cuidador 50 n/a n/a n/a 155 255 n/a

Competencias personales 43 9 7 n/a n/a

Dutch

Kennisbank 64 n/a n/a n/a n/a

Verbeter Zorg vaardigheden 22 n/a n/a n/a 61 133 n/a

Greece

Χρήση Διαδικτύου 170 1465 1371 n/a n/a

Επικοινωνία Περιθαλπόντων 90 152 139 n/a 1

Ενίσχυση Δεξιοτήτων Περιθαλπόντων 113 n/a n/a n/a 101 450 n/a

Αξιοποίηση Δεξιοτήτων Περιθαλπόντων 58 n/a n/a n/a n/a

The data demonstrates that:

• The numbers of registered users varied across pilot sites

• The number of registered users may not be as high as the number enrolling in the pilot in some

countries

• The numbers of registered users varied across content, so the content is not equally accessed

• The first piece of content / course has the highest number of hits, perhaps suggesting that use is not

continued through the content, and content is explored sequentially

• The number of registered users accessing the areas of the learning zone varied between countries

• ‘Using the web’ was the most commonly accessed course in all pilot sites. It is noted that this is where

many carers began their exploration of the site and perhaps did not spend the same amount of time

exploring other pages

• ‘Making your skills count’ was generally the least commonly accessed (but to date has less content).

• There is very little forum use

• The quizzes, (intended as a means to check learning by checking for understanding of the topics in the

learning zone) had less uptake in the UK in comparison to Spain and Greece. This is in keeping with

the proposed business models (see D5.3 for details). In Spain and Greece, Greece in particular, there

is a good level of support for structured and blended learning and thus it is not unexpected to see a

greater uptake of the quizzes. There are more quizzes available to be taken in Greece than any of the

other pilot sites therefore the figures are high. In contrast, in the UK a more casual learning structure

is preferred, therefore the relatively lower uptake and availability of the quizzes supports this.

• Blended learning and formal carer professional development constitute a larger component of the

Greek and Spanish business models therefore structured learning such as quizzes would be expected

to be more acceptable in these pilot sites.

Table 14 provides a summary of access to help content over Pilot Phase 2. The counts show that the help

section was accessed and used. The most frequently accessed are the main help page, the ‘skills Activities

Help menu page’ and the Scoop.It Topics.

Table 14 Help section view counts

of 44

7.4 Discussion and conclusions

The available statistics provide an overview of system use to complement the usability and impact

questionnaire results.

The data provides some overall ‘headlines’. For example the low use of Forums; forums were introduced

at the request of users in the initial impact evaluation stages, and communication with others was a key

identified need which forums aimed to meet. The lack of uptake of this feature suggests that user

feedback should be interpreted with caution – D2.7 provides more detail on this issue.

The data has been drawn from Moodle directly via back-end administration access. The inability to collect

the required data necessitated a different approach, which was to use the admin access to the system to

determine figures for basic system use.

There is an unresolved challenge with this data in the differentiating of registered users from pilot

participants, from those that are interacting with and developing the system. The registered users

contains all admin accounts and accounts made by project partners, therefore this is not an indication of

DISCOVER users which is presumed to be a much lower figure. There is currently no way to extrapolate

the ‘real’ DISCOVER users from these numbers. The data set generated on a use by use basis is huge, and

to date we have not developed a rapid system for sifting out irrelevant account details.

The varied limitations of collecting data in Moodle are well documented, and there are a number of

projects and initiatives aiming to provide workable solutions. However, as yet there are no solutions

which would have provided a feasible solution in the timeframe available for this project. Future

projects could consider accessing a development such as the MOCLOG project to attempt to extract more

valuable data from Moodle (http://moclog.ch/project/). There are also other sources of data potentially

from the supporting database and Google Analytics.

It is recommended that further interrogation is needed to provide a robust picture of usage, but this

initial snapshot has allowed us to demonstrate that the system has been used by a good number of users

across the four pilot sites and that in the future we need to align and develop the content to encourage

users to explore the system with increasing depth.

8 LESSONS LEARNT

In reviewing the implementation of DISCOVER, this section aims to draw together the lessons learnt

across the service evaluation.

8.1 Method

All pilot partners were invited to give feedback on the running of the pilots, and reflect on the challenges

and successes of the evaluation process. Feedback was given via:

• Semi-structured interviews with pilot site personnel

• Email review of lessons learned with the pilot team

The aim was to understand the pilot delivery locally, usability issues, changes to the protocol and the site,

as well as information informing the business plans described in D5.3.

D4.5 described lessons learned from Pilot Phase 1 and the suggested changes to the methodology, site

and data collection tools in order to improve Pilot Phase 2. Table 15 describes the changes made and

combines lessons learnt across the project in respect to the piloting of DISCOVER.

8.2 Results

In-depth discussions about lessons learned throughout the pilot process took place within the pilot

personnel interviews. It was evident that when asked to think about lessons learned, the partners

showed a strong tendency to think in terms of limitations and challenges, and what could be done

differently in the future to avoid them.

Tables 15-20 summarise the identified lessons. We identify issues that have arisen during the project, the

potential impact and how we have or would respond to them. Many of the responses were implemented

in Pilot Phase 2 and as such represent successful improvements to the pilot processes, which are shared

in order to provide guidance for future similar projects. They have been summarised under the following

themes:

• Table 15 Lessons Learnt: Carer Recruitment

• Table 16 Lessons Learnt: Carer retention

• Table 17 Lessons Learnt: Working with Stakeholders

• Table 18 Lessons Learnt: 4 different pilot sites / countries / services

• Table 19 Lessons Learnt: Pilot process

• Table 20 Lessons Learnt: Data collection tools

• Table 21 Lessons Learnt: Data Collection Processes

• Table 22 Lessons Learnt: Service Content

• Table 23 Lessons Learnt: Service Development

• Table 24 Lessons Learnt: Pilot Delivery

• Table 25 Lessons Learnt: Business Plans

A narrative explanation of some of the key issues is also given in Appendix B.

Table 15 Lessons Learnt: Carer Recruitment

Issues arising during the

project

Potential impact Responses

C-1 Alteration of the pilot sample

from specified sample in

methodology due to:

* Lack of time

* Lack of existing ICT

knowledge

* Access to ICT facilities

* Relevance of the service

→ Results possibly not

representative of the originally

intended target market.

• There is a need to be flexible in the approach taken and respond to the potential

market and participant pool

• Conduct pilot focus groups and/or opinion polls in advance of initial recruitment, to

provide information against which to compare results and determine how

representative the sample is

• Offer practical support for the project to reassure potential participants that they will

have help if they need it

• Design a protocol with fewer questionnaires and more practical/experiential

evaluation

C-2 Difficulty gaining access to

carers

→ Difficuloes recruiong to

target

• Use a number of different channels to reach carers, through letters, meetings and

phone calls and email invitations.

• Use face to face meetings wherever possible to build trust and understanding

Work with gatekeeper organisations with existing links to, and relationships with,

carers

C-3 Achieving recruitment targets → Changes to the sample

composition; participants were

not representative of originally

intended user group and/or

were not in the correct ratio

between formal and informal

• Review progress and adjust recruitment methods

• Plan time for recruitment, engagement and the impact of losing access.

• Significant effort was given to this i.e. recruiting gatekeepers as early as possible,

building relationships and active recruitment could take 12 months in its own right.

• Acknowledge that it is hard to engage organisations / individuals in the social care

sector because giving care always tops all other interests

• Consider the sample size needed and what is achievable and realistic

C-4 Method of carer recruitment

changed during the pilot -

more reliance on gatekeepers

required than was initially

thought.

→ Improved access to carers

via the gatekeepers and

stakeholders

→ More effort for pilot sites

• Early engagement with gatekeepers and focus on bigger organisations that can

support such efforts.

• The pilots were planned with carers' needs in mind and that this helped encourage

participation

• Place emphasis on stakeholders as well as carers, to increase buy-in and cooperation

which could facilitate recruitment

• In early pilots (e.g. phase 1 in this case) engage with people with better computer

literacy to gain quick and effective feedback. Use this to make improvements to

make later phases (phase 2) more efficient

C-5 Variation in recruited

participants against the

number accessing the system.

Lack of analytical system data

available to verify

→ Results possibly not

representative

→ Lirle reliable system data to

analyse for evaluation

• Forward planning for analytics and usage statistics to give an accurate overview of

system use

of 47

C-6 Sample limited in terms of

language e.g. migrant workers

(Greece only - Spain have

migrant workers but without a

language barrier, NL and UK

do not have a significant

proportion of migrant

workers)

→ In Greece there may have

been important carers'

perspectives missed by not

engaging with migrant workers

who make up a large portion of

carers.

• Include content within the system for migrant workers in the countries in which this

is applicable.

• In the future consider including more basic language in this section, and providing

facilitation to allow these carers to participate and subsequently widen the market

• Seek to understand the similarities and differences between migrant and native

workers where this is not known (e.g. in Spain, where there were fewer differences

than in Greece, for example the lack of a language barrier)

C-7 Participants reluctant to

participate due to concerns

about not being able to

maintain their commitment

→ Hard to recruit to target

→ Increased arrioon rates

• We explained the processes and stages involved in participation at each step, whilst

reiterating offers of support and listening to concerns

• Ensured that carers could voice their concerns or fears at any point

• We ensured a flexible approach which takes into account carers' needs and lifestyle

and the methodology was adapted to their needs

C-8 The need to better sell the

project/ service in order to

recruit (this was particularly

evident in the Netherlands

and not observed across all

sites)

→ Carers and stakeholders lost

at point of recruitment due to

the unfinished site

• In the Netherlands for projects such as this, wait until the service is as finished as

possible before presenting it

• Explain clearly wherever there are unfinished or beta versions of site components,

and check that participants have understood this

• Respond to any technical queries or difficulties, and ensure that they are resolved as

quickly and adequately as possible

• Highlight that participation will help to improve a service which is aimed at carers -

carer involvement is key to ensuring that the service meets the needs of the users

• Disseminate the project wherever possible as a positive participation opportunity,

and highlight the benefits to carers of taking part

C-9 Face to face contact and

establishing a relationship is

very important to recruitment

(better than phone and email)

→ Makes recruitment more

resource intensive

→ More effecove recruitment

• Ensure face to face contact is embedded into the methodology from the outset of

carer projects

• Allow carers to share their stories and interact socially, to avoid the feeling of being

overburdened by attending project sessions

C-10 Carer scepticism of projects

and research activities in some

countries

→ High arrioon rates re the

questionnaires

• Ensure that the project aims and objectives are well explained to the participants,

and that they know their time is appreciated.

• Reassurance was provided alongside University level ethics approval

of 48

Table 16 Lessons Learnt: Carer retention


project


CR-1 Challenges in keeping carers

motivated through the pilot

→ Poor retenoon of carers • Change approach to involve face to face / group sessions that carers value.

• Design pilot to cater for participants that spend significant amount of time giving

care

• Provide flexible support and that if a carer requires technical help, or guidance

with the use of the service, they must be able to access this in a reasonable time

frame

CR-2 Limited content on the site

made retention difficult

→ Poor naturalisoc usage of the

site, loss of interest by carers

• This was addressed with more content complete for Pilot Phase 2

• Ensure regular updates throughout users' time as a participant OR explain that

they are only required to look at limited content.

• Make connections with other organisations who could provide updated validated

content

CR-3 Participants wanted to see a

nearly finished product

→ Hard to maintain parocipant

interest and commitment to the

pilot

• Ensure adequate explanation of pilot process and unfinished elements of the site.

• Keep site stable throughout evaluation periods.

• Design pilot to ensure access to material, e.g. by using a formal test population in

the early stages, and engage carers later or through more group/f2f settings with

support to give benefit beyond the project.

• Consider the time taken once content is created/curated in the UK, to then

translate and replicate the site in non-English pilots. This time should not impact

on the time given to evaluation periods

CR-4 New content is needed on an

ongoing basis to encourage

continued use

→ Loss of carers once the

content has been explored

• Working with gatekeeper organisations to continuously recruit new carers (high

turnover). Regular development of new content.

• Ensure carers can find information at the beginning.

• Once their need for answers is satisfied, they may not use the Skills Zone again or

use it less often- so recognise in evaluation

of 49

CR-5 Recruitment of carers may

have resulted in carers with

greater IT literacy than

originally intended in some

countries.

→ Pilot phase 2 adapted to this

issue and may have been more

representative

→ This provided useful

feedback as digitally literate

participants are more able to

voice their feedback than those

who are inexperienced

• Could be an advantage for the actual service development because we got

feedback from both groups.

• It may make sense to have people with higher IT skills in the early test phase

because they could give useful feedback, where the digitally illiterate could not.

• Consider pre-screening for computer literacy to give a better indication of the

existing computer skills of the sample and analyse results accordingly..

CR-6 Lack of incentive for carers to

take part in the pilot

→ Resources required for

recruitment

→ Lower numbers retained

• BIRM and the OU found that offering travel cost reimbursement and/or

refreshments helped to attract participants to introductory workshops

• Project staff visited many organisations at their own venues to take the burden of

travel away from carers.

• We didn't offer any incentives due to budget, consistency and fairness

• Provide information as to how not only the service but participation in the project

could positively benefit carers.

• Promote services which could provide additional support to those with increased

needs, e.g. people with very limited computer skills/access who may require

additional help such as public computer access to use the service.

• Consider including this as a clear section on the website.

CR-7 Burden on carers in terms of

the time required to give to

the pilot

→ Resources required for

recruitment

→ Lower numbers retained

• Appeal to significantly more carers than are needed for the sample, to account for

refusals and drop-outs.

• Clearly explain the differing routes to access for DISCOVER - that it can be accessed

from home or within a group, etc

Provide reassurance and support to encourage carers to talk to researchers

instead of dropping out if they have commitment issues

At the outset, show carers how participation could positively help them by

providing information and support

of 50

Table 17 Lessons Learnt: Working with Stakeholders

Issues arising during the project Potential impact Responses

SH-1 Gatekeepers added their own

questionnaires to our data collection

measures

→ Increasing parocipant

workload

→ Decrease in the number

of questionnaires completed

Ensure that gatekeepers do not introduce own questionnaires - if this is necessary,

streamline the data collected to avoid repetition

SH-2 Working with gatekeeper

organisations may have made

samples unrepresentative

→ Parocipants willing to

engage with technology and

greater IT literacy

→ Recruitment through

specialist support centres

may have meant

participants were not

applicable to the content

available - validity /

reliability of pilot data

• Consider alternative or parallel approach in future projects. For example, we

could have advertised in local newspapers and websites to seek 'trial

participants' e.g. in the job section, small ads etc. Note there could be a cost

implication.

• Clarify to the gatekeeper organisations what the aims and objectives of the

project are, so that they can help more effectively. Ensure inclusion and

exclusion criteria are clearly defined

of 51

Table 18 Lessons Learnt: 4 different pilot sites / countries / services


S-1 The decision to allow carer driven, flexible

content through content curation

(Scoop.It)

→ Resource implicaoons for

curating content for 4

languages and lacking the

ability to directly translate

content from Scoop.it (instead

of content written by the

DISCOVER team increased the

workload on all pilot sites).

• It changed the effort from translating and adapting 1 agreed course

into finding the relevant resources and create quizzes for these.

• However, the long term benefit is that we do not need to maintain the

content because it is maintained by others.

• Localisation is required in a multi-lingual and multi-cultural project. A

more detailed plan at the beginning may have helped, with more input

from the different countries involved.

• Use the exit strategy to clearly define the updating and maintenance

issues for the service

• Regular videoconferencing for pilot sites to report developments

Helpdesk support for pilot sites to enable to them to make local

changes more efficiently and effectively

S-2 Different country participants getting a

different experience

→ Hard to compare the

participant experiences

• Analyse country specific data separately and make comparisons within

the conclusion

• Results are likely to not be directly comparable due to pilot site

differences

S-3 4 data sets → Resource implicaoons

S-4 4 business models → Resource implications for

developing and implementing

the business models

S-5 The project aimed to achieve a

sustainable service in 4 pilot sites and

explore a joint organisation/business

model going forward. It became clear that

the 4 pilot site organisations have very

different needs and aspirations, therefore

4 different business models have been

developed.

→ Hard to make comparisons;

but appropriate as the sites and

local contexts different.

of 52

Table 19 Lessons Learnt: Pilot process


project


PP-1 Encouraging use of the

system outside of scheduled

sessions was challenging

→ Effects the validity /

reliability of the data

collected- DISCOVER used

in a controlled setting

rather than at home

• Different approach meant the carer experience was different and this has been

documented.

• Consider different approach to recruitment / different target group to verify online

learning impact.

• Design sessions to include lab usability as well as living lab evaluation

PP-2 Participants were guided

through specific tasks which

enabled them to assess the

various aspects of DISCOVER

→ Parocipants were not

exploring DISCOVER

themselves; (this did

provide a standardised

protocol for a group);

participants time wasted if

topics were not relevant to

them

• Consider group approach and face to face support from the outset.

• Consider recruiting a different target group to test online learning.

• Try to keep carers' interest active by providing learning activities tailored to their

needs

• Establish what people are interested in, and guide them towards that instead of

prescribing which content they should look at

PP-3 Participants were guided

through all areas; therefore

valuated by participants to

whom the content is not

directly relevant.

→ Affects the validity /


collected, or changes the

nature of the feedback. If

people are not using the

system in a 'real' setting,

the feedback is less valid

to questions pertaining to

this type of us.

• Interpret the results in the context in which they were observed - if people had not

used DISCOVER at home or in unsupervised/naturalistic settings, we can only make

limited conclusions about its use in this environment.

• Consider more learning evaluation methods e.g. experiential learning

• Collect more data which could be used to determine differences in use patterns, e.g.

accurate system data

• Embed data analytics into the system from the start for better analysis of system use

PP-4 DISCOVER used in an artificial

way to satisfy pilot

requirements

PP-5 Participants encouraged to

look at topics they would not

chose to explore

of 53

PP-6 The workshop approach

resulted in a range of skill

levels and attendance present

→ Parocipants wanted to

progress at different rates

- which may have affected

perception of the system

• Project staff and developers need to interpret the feedback against the characteristics

of the test population.

• Provide facilitation at the outset, which will then allow for more confident users to

progress on their own but could be continued for those who require ongoing support

• Encourage peer support for participants joining workshops at a later stage - existing

attendees could support new participants to learn the system

PP-7 Staggered recruitment led to

participants varying levels of

exposure to DISCOVER

→ Usability and impact

results may be variable as

a result

• Monitor dates of recruitment and attendance to allow for analysis of effects

PP-8 Hard to motivate participants

and encourage them to

continue use

→ Poor retenoon of

participants throughout

the pilot (repeated access

to the site)

• The project has increased our understanding of how hard it is for these carers to cope

with the demands of their role.

• To reach them learning and support needs to be embedded into activities they must

do anyway or take place at venues that they naturally attend, potentially at their

home through formal help.

• It could make sense to focus on prevention, i.e. encouraging people to use DISCOVER

that are new to a caring role when the condition of the care receiver is still good

enough so the carer has time to find information.

• Send notifications and news according to carers' interests and promote

communication and topics which are of known interest to participants

PP-9 Groups important to

encouraging use of

DISCOVER, can be virtual but

better to start in person

→ Group acovioes can

increase continued

participation by carers

• Ensure group activities form a part of the project plan

• Demonstrate the value of group activity online, and how to do this

• Allow sufficient time to organise groups

PP-10 Possible facilitator effect on

the feedback of the site

→ Affects the validity /


collected

• Accept as limitation of pilot methodology.

• Ensure facilitators are aware of any biases of their own or changes in expertise as they

grow more experienced in delivering the training

• Include reflection from pilot site personnel in evaluation content

of 54

Table 20 Lessons Learnt: Data collection tools


DCT-1 Loss of meaning as content is

translated

→ Validity of data collected

→ Comparability between

sites • Ensure adequate translation resources are allocated

• check for continuity of meaning and cultural differences DCT-2 Translation of the tool → Time and resources for

the pilot sites

DCT-3 Translation of the findings → Time and resources for

the pilot sites

DCT-4 Too many questionnaires → Lack of parocipant

motivation to complete;

poor sample sizes

• Online surveys so that participants can then have the system in front of them

when they complete the questionnaire.

• Consider using less questions per questionnaire

• Manage the gap /time between questionnaires, so that participants are not

tired of survey completion

• Evaluate usability through lab tests and not a questionnaire

DCT-5 Open questions of limited use due

to lack of detail provided and loss

of meaning in translation

→ Poor data quality in Pilot

1; adjusted for Pilot 2

• Usability should be at least partially evaluated in a lab setting.

• If a questionnaire is the only viable way to capture this information,

questionnaires should focus solely on usability issues and be quantitative.

of 55

Table 21 Lessons Learnt: Data Collection Processes


DCP-1 Online surveys planned but not

practical with target sample

→ Time and resources required for

data entry; poor response rates; no

ability to track completion rates • online questionnaires would reduce data entry time

DCP-2 Participants did not see the

relevance of the questionnaires

→ Low compleoon rates • Consider reducing the overall questionnaire lengths and including

more qualitative feedback for impact

• Questionnaires either alongside looking at the platform or as a task

embedded within the platform itself

• Use back-end system data to answer some questions, which reduces

the burden of completion on participants and provides an objective

rather than subjective view of site use

• Completion of the questionnaires at the workshops (as opposed to

sending participants the questionnaires separately)

DCP-3 Participants not attending

workshops

→ May have impacted compleoon

rates

• Facilitate participant access to workshops

• Consider use of incentives

DCP-4 Carers using one log in, rather than

being individually registered

→ Hard to track the number of site

users and their patterns of use

• Ensure users do not use a generic login by not giving out the

password

• Support users to make their own accounts

DCP-5 More robust mechanisms need for

collecting system data

→ Reliability of system log

information

• Analytics to be considered from the beginning and embedded into

the system and project plan

DCP-6 Ongoing usability testing required - → Usability issues idenofied which

were not formally captured

→ Issues present at end stages of

project which could have been

identified earlier

• Lab usability testing should be included in data collection methods

for website development projects

DCP-7 System use data difficult to establish

due to admin logins and spam

accounts which could not be

extracted

→ System use data and figures not

representative

• Consider spam and admin accounts and develop methods to remove

these from analytics

of 56

Table 22 Lessons Learnt: Service Content


SC-1 Low use of forums → No social benefit gained

from using the site

• Improve invite text

• Practise forum for learning how to use one.

• Review competing interests - is building a community of carers within the Skills

Zone an aim of the service? For instance this is not a priority for Birmingham, as

building a community requires support and moderation.

• Building a community of learners is different from a community of carers.

• Consider disabling the forum about care topics and leave one for discussing

learning progress, suggesting new topics etc.

• Explore whether forums are not needed or whether carers lack confidence to

use them

SC-2 One size does not fit all → User needs are individual

and varied; challenge to

reconcile requirements into

one service, and one

evaluation methodology

• Consider smaller target group (only carers that deal with a certain condition)

• Ensure objectives in line with carer needs

SC-3 Greece - formal carers more likely to

follow an accredited learning

pathway than informal carers

→ Impact on acceptance

and perceived usefulness of

the system, as well as

learning

• Iterative feedback to guide the usability and content.

• UK feedback from stakeholder stated that in rural areas approx. 10% of

informal carers move into formal care work. Potential to focus on a smaller

target group and specialise the service.

SC-4 Informal carers want to learn at their

own pace

SC-5 Adjustments needed to increase the

acceptability of the content to Dutch

carers

SC-6 Dutch carers resistant to structured

learning and teaching

• Present a complete service where possible

• Offer a product in which the learning aspect is embedded: first motivate, then

do-and-learn, then share

SC-7 Spousal carers do not necessarily see

themselves as carers and so don't

think they need the service

• Change marketing approach, not everybody thinks they are a 'carer'.

of 57

SC-8 In Spain people do not tend to have

broadband; mobile internet is not

sufficient resource to use the service

→ Poor usage

→ Poor usability

• Highlight that the service can run on poor broadband connections

• Wherever possible work with local organisations who can provide access to ICT

facilities and support users of the service

• Consider loans of equipment for people to try the service out and if they find it

useful to buy their own equipment

SC-9 Carers want non-caring content that

supports social inclusion

→ Social inclusion supported

by means other than online

forums

→ Added value to the

DISCOVER service

• Review feedback and manage scope of content provided during project and

later as a service.

• Emphasise the communication aspect of the service (e.g. forums, social media

participation)

• Created interactive map for carers to share social content

• Highlight the opportunities to communicate with people in similar situation

• Provide opportunities around overall psychological wellbeing such as brain

games etc

Table 23 Lessons Learnt: Service Development

Development


D-1 Difficult to develop content as the

site architecture wasn’t clear at first

→ Affected speed of

development

• Consider agile approach. Start small with a suggested architecture then amend.

• The project was deliberately user driven but getting feedback took time and

created a chicken and egg situation at the beginning.

• Clearer definitions of the learning zone architecture at the beginning

D-2 Different needs for different sites

→ Time and resource

implications

• DISCOVER use case and scenarios description from each partner

D-3 Replicating problems not always

possible due to different computer

systems

• Maximum and minimum system requirements definition

of 58

Table 24 Lessons Learnt: Pilot Delivery

Issues arising during the project Potential Impact Responses

PD-1 Setting up and running the pilot is

time consuming

→ Workload

→ Staff satisfaction

→ Timely compleoon

• The pilot team should have transparent roles and specific strategy.

• Stealth monitoring for User Activity Tracking

• Improved planning from the outset

• the pilot methodology was tested in Phase 1 with a smaller sample

• Consistent methodology delivery with support for pilot personnel

PD-2 Differing levels of resources and

staffing time between pilot sites

→ Workload

→ Staff saosfacoon


→ Variable quality and

quantity of site content

• Transparent roles

• Work allocation reviews

PD-3 Changing requirements during the

project (e.g. the need for translation,

for pilot partners to be involved in

content development and technical

changes)



PD-4 Match of roles to expertise (e.g. Pilot

partners required to support

development of the content



PD-5 Technical language barrier between

partners / sites → Increased ome taken to

make some developmental

changes

→ Some autonomy given to

pilot sites allowing them to

make changes for

themselves

• 1:1 Skype support for technical development to the pilot sites.

• Presentations of developments to each site

• All significant decisions, recommendations and guidelines need to be

annotated and translated adequately (translated and verified) to avoid

misunderstandings or inappropriate translation of procedures or content.

• Need to take into consideration in the Budget the efforts this procedure

entails.

of 59

PD-6 The need for pilot partners to be

involved in content development and

technical changes

→ Reduced burden on

technical team by giving

small changes to pilot sites

to complete

• Each pilot site needs a team with different skill sets. The pilot sites must

involve local experts in content development.

• Content plan from the beginning

• Exit strategy plan with definition of the system transfer in each pilot partner

infrastructure

• Partner meetings especially involving the responsibilities of the pilot activity in

each pilot site (researchers and supportive staff)

• Training in the procedures for pilot implementation: recruitment, running,

follow-up, data collection.

PD-7 Lack of clarity over partner roles → Staff saosfacoon


• Wwork package meetings at each new stage of the Project and guidance notes,

as provided by WP2 / or when a new major task begins to explain the tasks

clarifies roles of each partner.

• Work package leads to clarify WP tasks and manage accordingly

• Appoint a content manager from each site, to manage content plan

presentations and definition of tasks

PD-8 Changing deadlines and timescales → Confusion for project

partners

→ Missed deadlines

→ Reduced time for phase 2

and thus reduced time

between questionnaire

completions

• Milestones definitions and maintenance of a strict schedule

• Increase communication with delivery teams to ensure changes are

understood

of 60

Table 25 Lessons Learnt: Business Plans

Potential impact Responses Issues arising during the project

BP-1 Investors need to see a nearly

finished product

→ Gaining investor buy in • Balance the need for feedback from critical friends and engage with trusted

stakeholders only at the prototype stage.

• Wait for wider engagement until the product/service is well defined, complete

and functional to stand up to scrutiny of the wider market place.

• Ensure communication is tailored to explain clearly if a prototype is being

reviewed or the final product is launched into the market.

• An easy-navigated site, with good interface and fully operational services

without bugs, missing content, deadlocks etc makes clear the scope of the

project and its expected impact

8.3 Discussion

During the running of the pilots, a number of observations were made which necessitated changes to the

methodology, approaches to carers, and interpretation of the data. Tables 15- 25 provide an overview of

the lessons learnt during the pilot stages of the project. The issues arising are detailed alongside their

potential impact and how they were tackled. The tables have indicated where and when these responses

should be considered and how they can be implemented to improve key project outcomes such as

meeting deadlines, maintaining interest from carers and identifying differences across pilot sites which

can be mitigated. A narrative explanation is given in Appendix B which gives some further context to

explain some of these issues.

There are some lessons that apply to specific country site, however in the main, the pilot sites shared

similar experiences and were able to share their learning with one another to facilitate effective problem

solving across the project.

It is clear that an adaptive approach was required in order to make changes to the methodology and

approach to maximise the results and the likelihood of project success. Adaptability was also required to

adjust the service and pilot approach to local country conditions.

Working with carers is challenging; this group has huge pressures on them (hence the need for the

project). However, this makes participation in a project such as this difficult. The recruitment and

retention process was therefore carefully considered and adapted in each country to ensure this target

end-user groups were closely involved in the development and testing of the service.

The involvement of gatekeeper organisations and face to face working was particularly important to

recruit carers and to maintain their engagement through the pilot. The approach to pilot implementation

and working with carers was adapted as the project developed to ensure that participants ranging in skill

levels and attendance could progress at different rates and stay involved despite their challenging

circumstances.

The challenges of working across 4 countries, with 4 different versions of DISCOVER are detailed, these

lessons may be broadly applicable to other similar EU project. It was important to have 4 versions of the

service to meet local needs; however this increased the technical workload and local resource to support

this significantly. Furthermore the translation of site content has been noteworthy given the volume of

content generated. Careful allocation of resources is need for projects of this size where 4 versions of

materials emerge.

of 62

9 THE EXIT STRATEGY

In evaluating the success and potential of the DISCOVER service, this section aims to summarise the

findings in relation to exploitation developed in work package 5.

The overall market conditions for a continuing sustainable business model show a consistently

challenging situation. Each of the DISCOVER partners has developed an implementation plan that utilizes

their existing advantages, resources and infrastructure to the full, while using their local knowledge of the

targeted market segments to provide a tailored solution that is most likely to work in their specific

context. These are detailed in D5.3.

For example AUTH in Greece uses their existing university experience to provide training, set within a

recognizable course structure, utilizing existing local infrastructure and accreditation to target a gap in

provision for formal carers. Successful operation of this business model should lead in the medium term

to a Massive Open Online Course (MOOC) to dramatically increase the potential for new users and

further provision for informal carers. The key benefit of the Greek model is the potential first-mover

advantage available in this area. Political policy trends around care in Greece show that a national

strategic plan is imminent in the next political period, focusing on the accreditation of caregivers and

caregiving. The AUTH service, partly as a result of work on DISCOVER, is well-placed to offer this

accreditation ahead of competitors. By the time the National Strategic Plan is announced, the service will

have all necessary infrastructure and logistics in place to deliver a service.

On the other hand, ASTRA-com has taken a very different approach, which suits their culture and context,

by setting up a co-operative organization which is embedded into the community. This business model

has already attracted €740,000 of funding (€350,000 of external funding, more than matched by the co-

operative members). ASTRA-com’s approach is much more focused on face to face community based

learning, whilst encouraging remote attendance and use of community building fora and communication

tools to facilitate the sharing of knowledge, and experience.

One of the key elements of the AUTH, BIRM, INTRAS and ASTRA-com approaches is that they are all scale-

able, in that growth is built into the design of their business models. In the case of AUTH and INTRAS

exploring the use of MOOCs and cloud based provision allows significant growth without exhausting

scarce resources. In the case of ASTRA-com the replication of the co-operative model in other local

communities, built around a simple but sound infrastructure allows growth in a very different but equally

feasible way.

In terms of risk, all Partners address the highest risk for these circumstances, which is that when the

project funding ceases the project also ceases. Each of the Partners has extended the life of the

DISCOVER solution by creating either a minimal cost sustainable entity (such as BIRM) whereby time to

develop a more robust ongoing collaboration, or transfer to a stronger (new) partner is gained, or they

have created a viable new organization with strong prospects of future growth and success. AUTH and

INTRAS have demonstrated this approach, and ASTRA-com have developed the most positive business

model by creating a co-operative organization that has already gained a sizeable amount of funding to

guarantee its medium term future.

of 63

Whilst working across Europe, the consortium has witnessed and recorded many common challenges

faced by carers, the cared for, governments and support agencies. Across all the DISCOVER partner

countries, we have seen: care services overstretched by Europe’s ageing demography; rising social and

economic disadvantage and vulnerability for informal carers; and a seemingly continent-wide need for

training and development in both care and digital skills for all carer groups.

After three years of research, development and partnership, DISCOVER is well-placed to play a part in

addressing these common European challenges and needs. DISCOVER, as a funded project has achieved

the primary aim of creating a new service to deliver digital skills training for carers, in a context

appropriate manner and in a form which is sustainable.

of 64

10 CONCLUSIONS

This report has offered a post implementation review of the DISCOVER service. It is a success indicator of

DISCOVER effectiveness, providing measurable and quantifiable results to allow assessment of the

efficacy of DISCOVER. It has drawn together information from a range of sources and aimed to summarise

some of the key findings from the evaluation of DISCOVER.

10.1 Efficacy of the DISCOVER solution and supporting processes

The DISCOVER service was developed to provide online support to carers. The report has shown that the

content of the service was adapted to local conditions, and the pilot methodology and approach tailored

within each country. The service has thereby provided support to users by providing a portal to

information about their caring role and access to learning about digital skills. As a result of the findings

from Pilot Phase 1 and informal feedback as the service developed, DISCOVER has been successfully

adapted to local circumstances.

Usability testing of DISCOVER looked at differences in end-user perception of usability within each

country. 207 participants gave feedback. The service was considered on a country by country basis given

the variations in content. Broadly the findings indicated that participants feel the system was enjoyable to

use and met their expectations, and improved in usability from Pilot Phase 1. Greek and UK-based

participants appear to be most satisfied with the service. In order to further improve usability and

acceptability of the system it would be advisable to undertake detailed interviews and walkthroughs with

representative users to further improve the ease of use in a targeted way.

Issues arising during the pilot have been documented, alongside the potential impact and how the project

team responded to address these issues. The issues were wide ranging and cover recruitment, retention,

working with stakeholders, and working piloting in different countries etc. The lessons learnt through the

pilot reflect the iterative process that has been involved in both the development of the service and

supporting processes and the pilot operations to ensure local needs are met.

The importance of tailoring both the service and the pilot local country context and requirements was

recognised early in the project and the project plan adapted accordingly. It is intended that the log of

lessons learnt provides guidance for similar projects and further exploitation of the DISCOVER service.

10.2 Improving the quality of life of carers

The report has summarised the usage rates of the service and shown that it is in use in the 4 pilot

countries. 371 carers and 18 of the older people they care have engaged with the system in Pilot Phase 2,

with a further 140 participants active during Pilot Phase 1.

260 participants completed the impact evaluation which indicated that carers:

• learned new skills and gained new knowledge through engaging with the DISCOVER Skills Zone

• valued finding out about local and national services, enjoyed leisure activities through DISCOVER

• felt more supported in their caring role as a result of DISCOVER

of 65

• feelings of wellbeing improved during pilot phase 2

• would recommend DISCOVER to other carers

• formal carers would also recommend it to their employers

Whilst the impact evaluation data was very positive, a couple of key messages from the usability data

should be reflected upon particularly in respect to the UK, Dutch and Spanish data which suggests that

participants had doubts as to whether DISCOVER has improved their ability to use computers and

communicate with others. This should be explored further as to whether it relates to the participants

existing competence in these areas, or their perception of their own learning during the pilot.

Overall it is concluded that the service was effective from the user perspective in terms of learning

outcomes, and willingness to engage with the system. Individual recommendations regarding further

refinement to enhance the service have been made in D2.7 in terms of the learning outcomes from the

service.

10.3 The exit strategy

A market analysis was undertaken under work package 5 and reported in D5.3. It focused on the ability of

the DISCOVER solution to be competitive in the market for three years, after the completion of the

project.

The high level findings of that report and the implications for each country have been summarised here.

Factors affecting the exit strategy at each site are briefly summarised. The report concludes that overall

DISCOVER, as a funded project has achieved the primary aim of creating a new service to deliver digital

skills training for carers. The service has been developed and tailored in a country specific way to cater for

local context and to ensure sustainability. The pilot results indicate that the service is valued by users and

that the service offers adequate usability. Recommendations are summarised for the further

development and exploitation of the service in D5.3.

10.4 Final conclusions

The conclusion of this post review evaluation is that there is strong evidence to point to the success of

DISCOVER as a service to provide effective online support to carers. The results presented show improved

skills, acceptance by the intended end-user group and good usability in each of the pilot countries within

which it was piloted. The business models demonstrate a route for sustainability and exploitation in each

country. A comprehensive overview of the recommended responses to observed challenges throughout

the project, have been given which not only reflect successful project working but inform best practice

for future projects.

of 66

APPENDIX A: PILOT PHASE 2 USABILITY SURVEY FOR CARERS

We would greatly appreciate your feedback and kindly ask you to complete this questionnaire. The answers you provide will

allow us to understand if the DISCOVER service was useful, easy to use and provided a satisfactory service. DISCOVER refers to

the digital training service for caring for older people that you have been using.

Please try to answer all of the questions, or select ‘N/A’ if the question isn’t appropriate or applicable to you. Your answers will

be added to the answers of other people using DISCOVER and will help us to make improvements for the future. Your feedback

is highly valued and we appreciate your participation.

Statement of Privacy: The information provided will be kept completely confidential and participants’ identity will remain

anonymous.

Demographic information

1. Gender Male Female

2. Age group 18-30 31-44 45-54 55-64 65-74 75+

3. Education

level

No

school

Elementary/junior

school

Highschool/Secondary

school

Higher

Education

Other

4. Are you providing care for someone? YES / NO

5. (If you answered yes to Q4) How long have you been a carer? ............ years ............. months

6. (If you answered yes to Q4) Do you have any formal education or training in caring for older people?

YES / NO

7. (If you answered yes to Q4) Where do you conduct care-related work most of the time?

At the home of the

person you are caring

for

At a residential

care home

At a care day or

drop-in centre

For trips only, e.g. helping with

shopping or socialising

Computer literacy

8. Do you use DISCOVER:

At home

Under

supervision e.g. in

a training session

or workshop

Both Neither – I have only seen

DISCOVER but not used it

of 67

Ease of learning

Please rate your answers on the scale by selecting the number which best represents your agreement.

9. I could easily find the information I needed to get started with Discover

strongly agree 10 - 9 - 8 - 7 - 6 - 5 - 4 - 3 - 2 - 1 strongly disagree

10. I was able to find the help function on the system when I needed it


or N/A – I didn’t need the help function (please tick)

11. The help function provided solutions when I had difficulties


or N/A – I didn’t need the help function (please tick)

12. I could use DISCOVER successfully every time


13. It was easy to find the pages or sections I wanted to look at


14. The instructions in Discover were easy to follow


15. When I made a mistake, it was easy to correct or undo


16. It was easy to track my learning progress and see what I had learned so far/what I had left to learn


17. It was easy to pick up where I had previously left off


18. I liked the visual design of the DISCOVER system


19. I was able to create an account to use DISCOVER without any problems


or N/A – an account was made for me (please tick)

If you had problems, please describe them as well as you can:

…………………………………………………………………………………………………………………………………………………………….

of 68

20. Did you experience any page errors? (Page errors are when a page within a website fails to load

properly, and an error message is displayed.) YES / NO

21. (if yes) I understood what the error description said and it was easy to correct the error or get away

from the error page YES / NO

Please describe the error(s) in as much detail as you can remember, so we can try to prevent it from

happening again.

…………………………………………………………………………………………………………………………………………………………

Satisfaction

22. How beneficial do you think DISCOVER is for you?

Very beneficial 10 - 9 - 8 - 7 - 6 - 5 - 4 - 3 - 2 - 1 not beneficial at all

23. How enjoyable was it to use DISCOVER?

Very enjoyable 10 - 9 - 8 - 7 - 6 - 5 - 4 - 3 - 2 - 1 not enjoyable at all

24. DISCOVER has met my expectations


25. Using DISCOVER has improved my ability to use computers


26. Using DISCOVER has improved my professional care skills


27. Since using DISCOVER, have you used any social networking sites? (Social networking includes sites

like Facebook and Twitter, or using online forums or any other online method of talking to people or

sharing information) YES / NO

If yes, which one(s)? ……………………………………………………………………………………………………………………………..

Accessibility

28. The text on the DISCOVER website was at a size I could easily read


29. The buttons on the DISCOVER website were labeled clearly


30. The colours used in the DISCOVER website did not cause me any problems

of 69


31. Did you have any problems with the accessibility of the DISCOVER website (e.g. text style or size,

colours, language, pictures, layout)? If so, please give details so that we may make improvements in

the future .

……………………………………………………………………………………………………………………………………………………………….

.

32. DISCOVER has helped me to communicate with others more


33. Discover has improved my access to information and knowledge about my caring role


34. I would continue to use DISCOVER if it was available


35. I would be willing to pay for DISCOVER


36. I would recommend DISCOVER to other people


37. Is there anything you think should be changed to make DISCOVER more user friendly?

……………………………………………………………………………………………………………………………………………

……………………………………………………………………………………………………………………………………………

……………………………………………………………………………………………………………………………………………

of 70

APPENDIX B: LESSONS LEARNT – NARRATIVE DESCRIPTION

Translation

Throughout the pilot phases, partners were required to translate the content of the DISCOVER site into

their native language and make any adjustments felt to be necessary to bring the content in line with the

expectations and needs of the carers they were working with. This was a crucial task since there is a need

for content to be presented appropriately, but one which led to some difficulties. The translation work

was seen to be time consuming and difficult; at times the translation work was being completed by

project partners rather than qualified translators, which took longer than expected and required

checking. Additionally, extra care had to be taken to ensure that meaning and context was not lost during

the process. Due to the evolving nature of the DISCOVER site, the translation was an ongoing task.

In addition to translating the content of the web pages, new Scoop.It links were required in the correct

languages – since Scoop.It curates external content, this could not be translated. This required additional

time to find equivalent content to the UK Scoop.It topic list. However, this was an opportunity to embed

country-specific content into the local sites and ensure that the needs of the carers in each site were

being listened to and catered for, taking into account any cultural differences or differences in areas of

interest. This was an opportunity to provide a tailored solution within each pilot site and engage with

carers more fully.

The site and content were not the only areas which required translation; the questionnaires and

subsequent responses were also needed in the correct language. Care had to be taken to ensure that the

meaning of the questions was not lost in order to elicit appropriate equivalent responses, and these

responses then needed to be converted into English for analysis. This was an opportunity to identify areas

where meaning may be lost in translation, and to remedy this where appropriate.

In future projects it would be prudent to consider the effort required for this task ahead of time to allow

for the time and expertise required to be incorporated into the project planning. Though translation was

included in the project plan, the effort required exceed what was expected and therefore impacted on

the resources for other areas of the project.

Content

The DISCOVER content was developed and updated continuously throughout the pilot phases, and as

such may have resulted in some participants experiencing a slightly different site than others during the

same phase. Pilot partners did not feel that this would have had a significant impact on their feedback,

but some did wonder whether there was a facilitator practice effect. In addition, to some extent it was

felt that this caused difficulties when approaching gatekeeper organisations or potentially considering

investors/business partners, due to not having a finished product to demonstrate.

The site content was developed in close collaboration between the technical partners and pilot partners,

however in the beginning stages this was felt to be somewhat challenging. Some pilot partners felt that

devising content before knowing the site architecture was challenging and may have been easier with the

site structure already defined. This was a further challenge due to the different needs of the different

of 71

pilots, for example the Dutch partners felt strongly that a teaching/learning structure was not appropriate

for their carers and therefore would not be a suitable way to organise the content.

The content, though developed consistently throughout the pilot period, was limited due to resources

and the requirements of the trial; a finite number of topics and virtual learning environments were to be

tested. Feedback suggests that participants found this content useful, but that they would not be likely to

revisit the same content multiple times. New content will be needed to maintain engagement and

learning, which could be added either by the users or by the DISCOVER team, dependent on the business

model moving forward.

Recruitment

Recruitment was consistently challenging for the pilots due to the nature of the population from which

the participants were drawn. Carers were seen as open to ideas and services which might help them, but

protective of their limited time and therefore wary of committing to taking part in the research activities.

This pattern of behaviour was seen across all pilot sites, however there were differences in the ratios of

formal to informal carers and ease or difficulty of reaching carers.

One concern which was common across some of the pilots, in particular in Spain and Greece, was that

since carers are a hard to reach group with little time to commit to research participation the participants

reached in the pilots may not be representative of the wider population. In Spain there was an

acknowledgement that the baseline computer literacy and access of the participants (particularly in pilot

1) may be higher than the average carer, due to the need to recruit carers already associated with

INTRAS. Pilot Phase 2 was considered to be more representative, with recruitment networks having been

improved due to learning from Pilot Phase 1. In the UK however, this was not seen as a problem but

rather an opportunity to gather feedback from more digitally literate users who would be more able to

describe their experiences with DISCOVER than those who were less digitally experienced.

In addition, there was consideration of the fact that many carers are migrant workers with limited

language skills in their country of employment. This would lead to difficulties using the site and therefore

makes this group all the more difficult to reach. This was an issue limited to Greece in the main, as there

are a number of migrant workers providing care in Greece.

A strong message from the carers’ feedback via interviews and focus groups was that people are over-

burdened in their caring role and need to feel that anything which requires their time must offer them

significant value to justify this commitment. Those who could not commit to taking part in the pilots

offered positive feedback on the DISCOVER service having briefly looked at it (more details can be found

in D2.7). It was observed that some participants derived a positive experience from being involved even

minimally in the project, and appreciated their opinion being sought.

The challenges in recruitment were met in some cases by working with gatekeeper organisations who

were able to either embed DISCOVER within their existing courses or to offer support and routes to

recruitment via their own networks. This was seen as helpful in order to meet project requirements and

targets, however it was noted that this may have been unrepresentative of the target audience of

DISCOVER. It is possible that the people who could be reached this way already had an interest in helping

of 72

with projects of this nature, and the baseline computer skills suggested that (particularly in pilot phase 1)

participants from these sources were already computer literate to a certain degree. Gatekeeper

organisations also at times introduced their own questionnaires to the data collection measures which

meant that participants were completing more questions than originally intended. This was seen to

potentially discourage some from continuing, and may have been a source of frustration to some

participants. Overall, gatekeeper involvement was seen as vital to the success of the recruitment and was

seen as a positive and efficient way to meet recruitment targets. Phase 2 involved engaging with

gatekeepers much earlier in the process which yielded more success.

Use of DISCOVER

Much of the work during the pilot phases was completed in workshops and other face-to-face sessions.

Participants were guided through specific tasks which enabled them to assess the various aspects of

DISCOVER. Whilst this allowed for a standardised protocol to be followed, this also potentially meant that

some users were not experiencing DISCOVER in a natural way. Due to the value of carers’ time, they fed

back that they found it useful to look at the areas related to their caring role, but less so for areas they

felt were irrelevant to them. This also could have affected the results in that the topic areas may have

been partially evaluated by people with no existing knowledge of the area. For future projects of this

kind, it is recommended that topics be assigned to participants depending on their caring role; i.e., those

recruited from a dementia centre could primarily be asked to evaluate the dementia topics. This

approach was implemented in pilot phase 2, based on the learning from phase 1. Additionally, it was

recommended by pilot partners that sufficient time is given to carers to socialise and share their

experiences with each other during the group sessions, as this was seen to have a positive effect on

participation and provide added value to carers taking part. For future projects involving carers this is

seen as a crucial part of building rapport with participants, and was considered by pilot sites as an

important lesson and one which could improve participation.

A second issue with the workshop approach was that the range in skill levels and attendance caused

participants to progress at different rates. Participants who joined a series of workshops later on in the

course would be further behind than those who had joined earlier, and some participants worked

independently to look at the areas which interested them instead of staying with the facilitator in the

prescribed route through DISCOVER. ‘Buddy learning’ was proposed as a means to turn this challenge into

an opportunity, by allowing participants who had already attended some sessions to support and guide

newer participants.

Retention and attrition

There was a high level of participant attrition across almost all of the partner sites, which can be

somewhat accounted for by previously mentioned issues such as a lack of time from the carers’

perspective and frustration with the number of questionnaires. The feeling from some of the pilot

personnel was that participants were reluctant to feel like they were committing themselves to more

than they could manage. Signing up to the project was seen as something of a contractual agreement and

as such for some participants began to feel like pressure. There were a number of suggestions which

were seen as ways to mitigate this issue, and were put in place in some pilot sites. Completion of the

questionnaires at the workshops (as opposed to sending participants the questionnaires separately) was

of 73

an effective way to collect the data, but did not address the issue of participants not attending

workshops. There were discussions about hosting the questionnaires online to cut down on data

processing/entry time, which it was felt could also make completion easier for the participants. Adequate

resources were not available within the scope of the project however this is a point for consideration in

future similar projects. Not only could online questionnaires have reduced data entry time, but

participants could have completed the questionnaires either alongside looking at the platform or as a task

embedded within the platform itself. It is possible this approach could have yielded more completions.

The overall lesson learned by the pilot personnel regarding retention of participants was a feeling that

interesting people in the project in the first instance is not the biggest challenge; keeping people

motivated to continue in the project and complete the relevant research activities even when there is no

direct benefit to them is the challenge which must be met.

Pilot partner workloads and requirements

There were a number of issues raised during the course of the pilots which would be useful learning

points for future projects.

Differing levels of resources and staffing time: Pilot partners were able to commit different number of

staff members to the project and the running of the pilots, which may have resulted in variance in

individual workloads.

Changes in requirements: There were changes to the tasks and requirements during the life of the pilots,

such as the need for translation for which effort was possibly underestimated, and the need for pilot

partners to be involved in content development and technical changes.

Lack of technical expertise: Pilot partners were involved in the development of the content of the local

sites and as such were required to complete developmental tasks to update the sites. This was an

ongoing task which some partners found to be challenging due to their lack of prior knowledge of this

type of work. The main underlying issue with the technical development was a language barrier; not only

a literal language issue but also a technical language barrier. Some pilot partners felt that whilst they

were well supported in their efforts to update the website and content, the work was very time

consuming for them as they not only needed to complete the task but also learn how to do it.

Additionally, they felt unable to describe problems adequately to the technical team who were

subsequently not always able to understand the problems and replicate them to find a solution. From the

technical side, there was concern that because non-technically trained personnel were updating the site,

there was a risk of accidental errors leading to additional work to remedy them.

It was also clear at some points during the pilots that there was a lack of clarity in terms of partners each

knowing the role of the other partners. In particular it did not seem clear to pilot partners the different

roles of HDTI and the OU.

of 74

The DISCOVER solution

A very clear lesson learned during the course of the project was that carers are very individualised and

have very disparate needs. This was observed across all pilot sites and was seen as a crucial consideration

when developing the service (for more detail see D2.7). The service at this developmental stage was seen

as forcing a compromise between meeting the needs of the users and providing an equitable service to

all, which could be evaluated by a common methodology. It was acknowledged by pilot partners that

there could not be a ‘one size fits all’ solution, and there were a number of factors which could influence

the success or failure of DISCOVER in meeting carers’ needs:

Formal and informal carers: Each pilot site recognised that there were differences in formal and informal

carers across a number of areas and attitudes. In Greece in particular, it was felt that formal carers would

be far more likely to pursue an accredited learning pathway than informal carers. Informal carers, across

the board, were interested in finding out more about their caring role but wanted to do this at their own

pace and were sensitive to feeling undervalued. It was important for the informal carers to be able to

share their experiences and knowledge; for this group, the ability to add and personalise content may be

valuable. Learning was not always viewed positively, but it was possible in some areas to repackage the

same content and increase carer acceptance. It was recognised by the Dutch partners that whilst it is

necessary to make some adjustments to DISCOVER to make it fully acceptable to Dutch carers, making

the system perform a completely different function to its original intention is not necessarily feasible. A

suggested approach in light of this was to see DISCOVER as a ‘give and take’ process – carers need the

opportunity to feed back what they have learned from their caring role, whilst also being able to seek the

new information they need in a supportive environment.

Cultural and geographical differences: In the Netherlands in particular, there were a number of cultural

considerations which necessitated a different approach to the other pilot sites. Carers were against the

idea of structured learning and teaching, and as such the learning zone was rebranded to the

‘Kennisbank’ (Knowledge Bank). In Spain, carers can often be very geographically remote and there is

poor access to broadband internet, which could affect uptake of a service like DISCOVER. Table 16

provides some suggestions as to how to resolve this issue.

Carer identity: It was apparent during the pilots that some carers, particularly informal family caregivers,

do not necessarily identify themselves as such. Where care is given based on the needs of a close family

member, it may be seen as part of familial responsibility and a negative stigma attached to the label

‘carer’. This should be considered when planning carer projects, so as not to miss the feedback of those

who may find the service useful but might not self-identify as the target market.

Ability of carers to communicate their needs: The pilots were focused on determining the needs of the

carers and making developmental decisions accordingly. However, it was clear that in some cases,

feedback from carers regarding what they wanted from DISCOVER was not always supported in their

feedback and behaviour after implementation. For example, in the UK in the prepilot phase carers voiced

strongly that they wished to see forums and other ways to communicate with other carers. As a result,

this was implemented but no forum posts were made. Feedback in pilot phases 1 and 2 suggested that

carers lacked the confidence to use such a feature. Another key lesson from the pilot feedback was that

not all content necessarily must be caring-related. Carers expressed an interest in local activities such as

of 75

walking groups and other social support networks. It was clear, however, that whilst carers may not

always have a strong idea of what they need, their needs equally must not be assumed by developers.

post implementation review - amazon...

Documents