1

Return on

Investment for

the All Wales

Academy

Juste Abramovaite,

Skills for Justice

August, 2015

2

Contents

Introduction to the All Wales Academy and Return on Investment 3

ROI and eLearning – other studies 4

Methodology 6

Findings 7

Training on Data Protection and Information Governance – current picture 7

Additional feedback from local authority representatives 10

Conclusion and recommendations 13

Appendices 15

3

Introduction to the All Wales Academy and Return on Investment

The All Wales Academy (AWA) is a new eLearning platform being launched in Wales which is

planning to provide over 60 eLearning training modules for staff across all 22 Welsh local

authorities. Its vision is to introduce a new way of working by up-skilling employees and

increasing their employability. In addition the academy will reduce training budgets and

make more efficient use of employee time. It is also hoped that the collaboration model put

in place by the project will facilitate collaboration between local authorities in the long-

term.

As it is important for the developed platform to be sustainable and leave a robust legacy

once the project has concluded it was decided that a Return on Investment evaluation was

needed to investigate and identify the benefits of the platform and the collaborative

foundation. This method of evaluation was deemed particularly appropriate as Return on

Investment’s (ROI) aim is to measure and account financial return in real time as well as

taking into account future savings and wider benefits. ROI is therefore not purely focusing

on financial aspects of the project but considers the value in both quantitative and

qualitative terms. Barker (2001)1 has outlined the usefulness of ROI analysis for any learning

products and services:

1 Return on Training Investment: An Environmental Scan and Literature Review (Barker, 2001)

Enables learning to be viewed as an investment rather than expenditure.

Enables informed choices between learning options.

Encourages employers and employees to take learning more seriously.

Compares the cost of training/retraining to new hires.

Contains and/or maximizes learning budgets.

Makes course objectives and content more relevant.

Refines and revises, or eliminates, weak learning programs.

Improve the financial worth of human capital assets.

Attracts attention to a particular problems, e.g. with productivity.

Projects future teaching/learning costs

Improves the efficiency of resource utilization.

Encourages transfer of training.

Measure the effectiveness of education/training and more.

4

It is important to recognise that ROI is different to a simple cost saving – instead of

measuring a reduction in expense (cost saving) ROI is the rate of return on investment so it

takes into consideration how much was spent on investment initially.

ROI and eLearning – other studies

John L. Setaro in his How E-Learning Can Increase ROI for Training2 states that while the

most obvious impact of e-Learning on ROI is the significant cost savings it can produce over

traditional training; well constructed eLearning is not only faster and less expensive than

classroom training but also more effective. Numerous studies have shown that people

learn faster with multimedia training; they more accurately recall what they learned over a

longer period of time and they are better able to transfer what they learned to actual

performance. Studies conducted by the military, education and industry cite a 15-25%

increase in learning with significant increases in retention and transfer of training.

Brandon Hall in his Multimedia Training Return on Investment3 reviewed a number of

studies on multimedia training and ROI. The studies involved companies such as Intel,

American Airlines, Pacific Gas and Electric, IBM, Bethlehem Steel, Bell South, Steelcase, and

Pizza Hut. In all studies reviewed the results indicated significant increases in the quality of

learning when computer based training was compared with traditional classroom

instruction.

However, this does not mean that face to face training is without use; research suggests

that it can be very effective when used as a combined way of learning - blending eLearning

together with classroom learning. Online Training ROI: Five Features That Positively Impact

Your Bottom Line 4states that online training can stand alone as an efficient solution, but

Web based programs deliver optimum returns when used in conjunction with other

techniques to create a blended learning program. In fact, one study cited in above

publication, found that online students had more peer contact with others in the class,

enjoyed it more, understood the material better and performed, on average, 20 per cent

better than those taught in a traditional classroom.

A case study from Northumberland County Council describes how the Council recently

introduced an online Learning Pool in order to provide effective and efficient learning and

development solutions for 6,000 staff across the County. The results were outstanding and

in just one year, the Council had reduced the cost of delivery of training by 80%. With the

2 http://www.swslim.org.uk/downloads/sl1718.pdf

3 Hall, Brandon (1995, July/August). "Multimedia Training's Return on Investment" Workforce Training News

4 https://www.peoplematter.com/system/files/resources/PeopleMatter-White_Paper-Training.pdf

5

increased efficiencies provided, training cost on average £4 per employee per hour. Because

of this overall 80% reduction in costs, the Council is now able to provide even more training

to its staff.

Savings for Northumberland County Council were achieved through a combination of using5:

5 https://www.totaralms.com/case-studies/northumberland-county-council

Online training (over 10,000 hours of classroom delivery

saved).

Self-service functionality for course bookings (over 1,200

hours of administration time saved).

Bespoke content creation (over 20 hours of content

developed).

Reduction in CO2 omissions and fuel savings, all whilst

improving consistency and scalability.

6

Methodology

In order to collect data on the current picture of how much the training is costing at the

moment and how much it is likely to change once the AWA is live, an online questionnaire

was prepared and sent out to 11 local authorities (LAs) in early June6. The questionnaire

asked respondents to provide details on the following:

Unfortunately, potential respondents were reluctant to take part as many felt that they did

not have the answers to the questions that were asked. A change of tactics was required

and an amended questionnaire was sent out in early July7. Questions in this latter

questionnaire focused on Data Protection and Information Governance Courses delivered in

the last financial year. 13 LAs were asked to take part and eight responded to the

questionnaire. In addition, representatives from two LAs were keen to engage in more

detailed discussions following the questionnaire so in-depth phone interviews were held

with these individuals.

6 Letter which was sent out to the LAs with a list of questions is in Appendix 1.

7 An amended letter with a full set of questions is in Appendix 2.

The training costs LAs are facing.

How much of training costs were for face to face training

and how much was for e-learning modules.

The number of people attending training sessions.

The length of each training course.

How training costs will change once the AWA is in place.

If training was accredited.

How satisfied people were with the training they

received.

7

Findings

Initial feedback from most respondents in regards to the questions asked in the

questionnaire was that these were asked too prematurely and should instead be posed in a

year’s time when the AWA is fully operating.

None of the LAs could commit to providing an estimate for future costs once the AWA is

live; most of them stated that it is too early to discuss and that these costs have not yet

been determined. Thus establishing an exact figure of how much money will be saved once

the AWA is fully operating is difficult. However, LAs did provide a picture of what their

training on Data Protection and Information Governance looks like at the moment. All data

collected is represented in Appendix 3.

Training on Data Protection and Information Governance – current picture

From the information provided by LAs it is evident that training differs in both numbers of

people who attend and the costs involved.

Graph 1 below outlines how many people attended the training in each Local Authority

(who provided such information) in the last financial year for Data Protection and

Information Governance Courses. Cardiff had the highest number of attendees overall

(5,126) while Wrexham had the lowest number with just 40 people attending the training.

These figures show a combined number of people attending e-Learning and face to face

training.

“...the ROI may be premature as the system isn't live at

the moment”

(Email from a Local Authority representative)

8

Graph 1

1055

328

5126

40

1700

4591

90

0 1000 2000 3000 4000 5000 6000

Blaenau Gwent

Gweynedd

Cardiff

Wrexham

Denbighshire

Caerphilly

Vale of Glamorgan

Total number of people attended the training

(Data taken from Local Authorities responses to ROI questionnaire (Appendix 2 and 3))

Courses delivered also varied in length. Graph 2 below shows the average length of the

courses for e-Learning and face to face training in all LAs (who provided such information).

On average, the shortest face to face training was delivered in Gwynedd (1.5 hours) and the

longest in Wrexham (6 hours). E-learning was significantly shorter in all LAs and ranged

between 0.14 hours in Cardiff to 1 hour in Caerphilly. Understandably, not all face to face

training can be replaced with a version of an e-Learning module; however partially moving

towards more e-Learning courses being available and delivered would significantly reduce

the time people need to spend away from their desks working. That would not directly

affect the costs of the training; however indirect savings would be substantial. For example,

3 hours per course delivered could be saved by changing it to e-Learning; if 5,000 people

undertook that course each year that would be equivalent to 15,000 hours more of working

time for the Local Authority. If on average, employees are paid £20,000 per annum that

would lead to indirect savings of £153,846 a year8.

8 3 hours*5000 people = 15,000 hours, each year one full time employee works 1950 hours (37.5 hours a week

* 52 weeks = 1950 hours). Then 15,000 hours divided by one full time employee's yearly hours: 15,000/1,950 = 7.69 full time employees. If you take £20,000 as an average pay for one employee, you get 7.69*£20,000 = £153,846 savings a year.

9

Graph 2

(Data taken from Local Authorities responses to ROI questionnaire (Appendix 2 and 3))

The costs of training vary widely across all LAs. On average, to train one person in Blaenau

Gwent costs less than a pound – this Council offers a range of eLearning modules which are

short and very cost effective. In comparison, the average cost of face to face training paid in

Wrexham is £146 per person. The graph below shows the average cost of face to face

training is just under £60 while the average cost of e-Learning is less than £6 which

suggests that there is a potential for substantial savings across LAs once more e-learning is

implemented. Although LAs were unable to provide estimates on how costs will change

once the AWA is live, this suggest that introducing e-Learning across all LAs will be cost

effective in the long run.

10

Graph 3

(Data taken from Local Authorities responses to ROI questionnaire (Appendix 2 and 3))

Additional feedback from local authority representatives

After discussing their views on the AWA, LA representatives raised some interesting points

which would not be captured for some time in the LAs’ data (due to this project being at

such an early stage), however, they could be checked in a year or two:

Duplication – Health and Safety training courses were given as one of the examples

where there is too much choice which often leads to duplication in courses undertaken by

the staff. This in turn leads to extra costs to the LAs – unifying the training courses, while of

course, still keeping needed differences according to each LA, would be the way forward.

For that reason the AWA could be a great opportunity not just to directly reduce the costs

to the LAs but also to make a selection of training courses more unified. This means that

fewer courses would be duplicated, staff would need to undertake fewer courses, overall

direct costs would be lower and time off work taken to attend various courses would

decrease.

“..AWA is a chance to unify things so we avoid all

duplications in training we are facing now... there is too

much choice”

(Email from a Local Authority representative)

11

Welsh language

The LA representatives generally viewed courses in Welsh positively however as there are

not many cases when courses in Welsh language are needed, translation costs can be a

burden to the LAs. Therefore, the AWA which offers training in Welsh would reduce the

total cost.

Implicit training costs

The LA representatives felt that face to face training is more expensive not only because of

how much each training package costs but also how much time staff members need to take

off from work to attend including travel and subsistence cost. The AWA providing eLearning

modules would save staff time in this respect, as staff could undertake training courses

from their computers, tablets or even smart phones.

Accessibility

Some LAs indicated that not all of their staff were previously able to access the training; in

some cases up to one third of all staff working for the LAs in question, would not have

access to a computer. The AWA with its new approach allowing staff to access training on

smart phones and tablets would cater for a more accessible and flexible way to learn.

Additionally, employees with disabilities who would not always be able to attend face to

face training (if it is held outside the office in a non-accessible building) would benefit by

training being delivered via eLearning.

Accreditation

None of the LAs spoken to had accredited their employees for the training courses

attended. Two main reasons given were the extra cost involved and that it was not a

requirement in staff job descriptions. Having collected data on the number of learners

registered and the annual costs of Approved Centre Status for Leadership & Management

and Health & Social Care, it can be concluded that the cost per learner varied a lot between

those LAs that provided data9.

Some LA representatives commented on how expensive it is for them to pay for Approved

Centre Statuses and since there are a lot of duplications across training courses on offer, the

AWA is seen by them as a way forward as it will help LAs to save money on training, reduce

training times by reducing duplications and allow accreditation of employees at much lower

9 All of the data is available in Appendix 4.

12

cost. Furthermore, the Institute of Customer Service suggests that employees’ job

satisfaction could increase once training is accredited as people would feel more satisfied,

committed and motivated10.

Additionally, Steven Schmidt11 found in his study on the relationship between satisfaction

with workplace training and overall job satisfaction, a significant relationship between the

two. Components of job training, including time spent in training, training methodologies,

and content, were determined to be significant in their relationship to job training

satisfaction and trainees were significantly more satisfied with the training they received

when the methodology employed was their preferred one. Therefore, it is possible that the

AWA could not only increase efficiency at work as more staff would be trained but could

also increase employees’ job satisfaction.

10

https://www.instituteofcustomerservice.com/1723/Training-accreditation---TrainingMark.html 11

Steven W. Schmidt, 2007, The relationship between satisfaction with workplace training and overall job satisfaction, Human Resource Development Quarterly, Volume 18, Issue 4, pages 481-498.

13

Conclusion and recommendations

Research findings show that there is a definite need for the AWA and that it could bring

substantial improvements on how training is delivered in the future. It is however worth

noting that not much of the quantitative analysis can be made at this point due to the early

stage of the project. The direct cost of training to all LAs is likely to decrease once all AWA

modules are in use since eLearning has a proven record of being more cost effective than

classroom learning (see the section ROI and eLearning – other studies). Additionally, the

AWA could bring increased accessibility, lower travelling costs, further introduction of

Welsh language courses and reduced duplication across all courses thus enhancing the

overall efficiency of the training provided.

It is recommended that a further financial analysis is conducted in 12-24 months time in

order to fully establish the success of the project. It will therefore be crucial to collect data

for the next year or two in all participating LAs on the following:

- The number of people attending different training courses

- Difference in numbers of those courses being face to face

training and e-Learning

o compare whether more LAs shifted towards more e-

Learning/less face to face training since AWA was

introduced

- All aspects of costs involved including implicit costs such as:

o time taken off work to attend training

o how many people chose to access the training via

smart phone or a tablet

o how many people were accredited for the courses

they attended

o was there a difference in numbers of people with

moving disabilities attending training courses since

AWA was introduced

o was there any improvement in employees work

satisfaction ratings

14

To be able to use future data for financial analysis it is very important that data collection is

standardized across all LAs. At this point, it is possible to observe the positive feedback and

expectations from all LAs that took part in the research. Respondents clearly see the

potential of the AWA and look forward to the positive changes they expect this new e-

learning platform to bring. For the sustainability and success of this project, conversations

and collaboration need to continue between all parties involved and feedback from those

organizing and attending training is important to collect as to ensure that training is

reviewed and improved on a regular basis.

15

Appendices

Appendix 1 – email with the ROI questionnaire sent LA representatives in early June

All Wales Academy (AWA) e-learning project: data for the Return on Investment research

Dear {insert name},

I am following up on an email from Rachel Searle regarding the All Wales Academy (AWA) e-

learning project. As you will be aware, Skills for Justice are currently working with partners

to develop and deliver the AWA and as part of this project, I am responsible for conducting

a Return on Investment (ROI) exercise.

In order to do this, I need to explore in detail the types of training (and the costs involved)

that were in place before the implementation of the AWA and to compare these with the

actual (or expected) training/costs going forward.

Consequently, I would kindly ask if you could provide any data you have available on the

following:

- How many people were involved in training courses before AWA was implemented?

(please give us a figure which only includes the training which is transferable to AWA)

- And/or how many hours of training that was?

- What was the cost of that training?

- How much does it cost to train people now using AWA?

- How were people accredited for the training they undertook?

- Has accreditation changed since AWA?

- Any data available on employees’ satisfaction with their work place?

- Any other information which you think would be of the importance to us to know.

Collecting this information is crucially important in order to see the effectiveness and future

sustainability of AWA, hence, your corporation would be very much appreciated.

16

Appendix 2 – email with the amended ROI questionnaire sent to LA representatives in early

July

All Wales Academy (AWA) Return on Investment Questionnaire

Skills for Justice are currently working with partners to develop and deliver the All Wales

Academy e-learning site and part of the project requires a Return on Investment (ROI)

exercise.

This involves exploring the types of training (and the costs involved) that were in place

before the implementation of the AWA and comparing these with the actual (or expected)

training/costs going forward. This questionnaire focuses on Data Protection and Information

Governance training rather than all training.

Consequently, I would kindly ask if you could provide any data you have available on the

following:

1. Can you list the different Data Protection and Information Governance Courses

offered during the period 1.4.14-31.3.15

2. Using the list of Data Protection and Information Governance training courses you

provided in question 1. Can you list how many people undertook each course between

1.4.14 and 31.3.15?

3. For each course listed in question 1, how long (in hours) was the course?

4. What was the cost of each course listed in question 1, related to Data Protection and

Information Governance?

5. Can you identify if any of the courses listed in question 1 were accredited?

6. All Wales Academy is developing a selection of courses covering Data Protection and

Information Governance, as listed below. We anticipate that some local authorities will use

AWA as part of a blended training approach for Data Protection and Information

Governance. As such, they may reduce their face to face contact training time in this area

or cease to purchase specific Data Protection training e.g. Bob’s Business. Based on your

potential plans for blended learning with AWA in Data Protection and Information

Governance, can you forecast how much it will cost your organisation to train people in this

area over a 12 month period (excluding any contribution to AWA)?

• Sharing Personal Information (already available via AWA)

• Data Protection Awareness (available on AWA w/c 20.7.15)

17

• Freedom of Information - An Introduction (available on AWA w/c 20.7.15)

• Protection of Electronic Communication Regulations (available on AWA w/c 3.8.15)

• Introduction to Information Governance (available on AWA w/c 31.8.15)

7. The AWA board are currently considering the benefits of developing an Assessment

Centre through AWA, allowing Local Authorities to make savings from their individual

assessment centre status (if they wished) and instead operate as satellite centres through

the AWA assessment centre registration. This would allow Local Authorities to still manage

their learners and assessments individually but make economies of scale for centre

approval, registration and certification. In order to calculate projected return on investment

of such an approach, we have decided to focus on the two main areas accredited by local

authorities – Leadership & Management and Health & Social Care. Therefore, can you

please provide:

i. Name of the awarding body your local authority is currently registered with for

Leadership & Management (e.g. ILM or other awarding bodies) ________________ &

annual cost for their Approved Centre Status __________________

ii. Name of the awarding body your local authority is currently registered with for

Health & Social Care (e.g. City & Guilds or other awarding bodies) ________________ &

annual cost for their Approved Centre Status __________________

iii. Number of learners registered & accredited through your centre (1.4.14-31.3.15) for

ILM qualifications _____________ and cost of this (including VAT)_____________________

iv. Number of learners registered & accredited through your centre (1.4.14-31.3.15) for

Health & Social Care qualifications______________ and cost of this_______________

Many thanks for your time. The information you have provided will be used to provide

projected return on investment figures for AWA.

18

Appendix 3 – data on the Data Protection and Information Governance courses collected

from LAs using the ROI questionnaire (Appendix 2)

Area Type of training Number

of

people

attended

per

course

Total

number

of

people

attended

Length of

training

session

Total cost Accredited

(Yes/No)

Blaenau

Gwent

Advanced Fee Frauds 373

Backing up Data 467

Business Continuity 293

Carefully Classified Bus. 540

Data Protection 726

EMail Errors 701

Email Etiquette 570

Freedom of Information 648

GCSX 545

Identity Theft 533

ISO 27001 386

Keeping It Clear 692

Mobile Working 348

PCI DSS 682

Perfect Passwords 748

Phishing Fears 441

Protected Premises 649

Risk Management V2 291

Virus Vigilance 516

19

Web Woes 406

1055 0.2 hours

(all

eLearning)

£9000 No

Gwynedd

DATA PROTECTION

(FACE TO FACE

TRAINING) 29 courses

304 1.5 hours No training

costs

identified

No

DATA PROTECTION (e-

LEARNING MODULE) –

recently produced

24 0.5 hours Learning Pool

essential

service

£16,000 per

annum (excl

VAT)

Shibboleth

Active Sync

Directory

£3,000 per

annum (excl

VAT)

No

328

Cardiff CCTV

Data Protection

The Principles of Data Protection

Email Errors

Email Etiquette

Freedom of Information

Keeping it Clear

PCI

20

Perfect Passwords

Web Woes

Virus Vigilance

Secure Printing

Protected Premises

Phishing Fears

Mobile Working

Identity Theft

5126 0.14 hours

(all

eLearning)

£12,375 BIS

ISO27001

compliant

Wrexham Information Security Law

for Public Sector

Managers, Auditing and

Data Protection

13

6 hours £1,900 No

National Archives

Training

27 6 hours 0 No

40

Denbighshire

Data Protection

eLearning

1300 0.5hours £18,000 No

Data Protection

(bespoke - classroom

training)

200

4 hours £1,200 No

FoI and EiR eLearning 200 0.33 hours Cost not

identified

No

1700

21

Neath-Port

Talbot

No training provided

Caerphilly Protection Information

eLearning - mandatory

every year for all PC

users

2800 1.5hours No

Sharing Personal

Information eLearning

200 0.5hours No

Information Governance

3 courses

73 3 hours No

Protecting Information 809 1.5hours No

Freedom of Information 209 1 hour No

Records management 27 7.5 hours No

Record management

shorter course

263 2 hours No

Data Protection and

CCTV

50 1.5 hours No

Data Protection training 80

FOI training 80

4591

£37,700

Vale of

Glamorgan

Data Protection - face to

face

48 4 hours Cost not

identified

No

Data Protection -

eLearning

42 0.5 hours Cost not

identified

No

90

22

Appendix 4 – data on the Leadership & Management and Health & Social Care courses

collected from LAs using the ROI questionnaire (Appendix 2)

Area and course Number

of

people

£ per

award

Total

cost

Cardiff - City and Guilds

Awards Registered for 2014-15

HSC Diploma Level 2 (Adults) 26 £92 £2,392

HSC Diploma Level 3 (Adults) 19 £110 £2,090

HSC Diploma Level 3 (CYP) 3 £110 £330

HSC Leadership Diploma (Adult Advanced Practice) Level 5 8 £149 £1,192

HSC Leadership Diploma (Adult Manager) Level 5 3 £149 £447

HSC Leadership Diploma (Adult Residential) Level 5 3 £149 £447

HSC Leadership Diploma (CYP Advanced Practice) Level 5 4 £149 £596

Cardiff - ILM 10 £75 £750

Gwynedd - ILM 28 £68 £1,908

Gwynedd - City and Guilds 34 £112 £3,808

Wrexham - City and Guilds 126 £76 £9,582

Wrexham - TMDP Oxford Brookes 3 £2,750 £8,250

Wrexham - ILM 20 £450 £9,000

23

Denbighshire - ILM

Level 3 16 £106 £1,696

Level 5 5 £170 £850

Denbighshire - Health & Social Care qualifications (Anterth Mon in collaboration with

Conwy, Mon and Gwynedd

approx 30-50, at £350-£500 per head 40 £425 £17,000

(costs are likely to rise in the future for age over 25 for level 3 could be £2000 per head)

Neath-Porttalbot - City and Guilds 100 £108.80 £10,180

Caerphilly - City and Guilds 45 £189 £8,500