consistent performance measures - welsh government · sustained destination 4 6experimental...

Version 1 – April 2019

Consistent Performance Measures

Destinations Measure

Explanatory Notes

Version 1 – April 2019

Contents

Introduction 1 Destinations measure overview

Cohort 1

Measure definitions 2

Sustained employment 2

Sustained learning 3

Sustained employment and learning 4

Sustained destination 4

Experimental baseline data 6

Summary 6

Gender and age breakdown 6

Sector breakdown 7

Level breakdown 8

Key data sources 8

Matching process 9

Timelags 10

Annex A – Glossary of terms

11

Version 1 – April 2019 1

Introduction 1 The Welsh Government is developing a new set of performance measures for

further education (FE) colleges and school sixth forms, which give a more rounded picture of learner outcomes in the post-16 sector. The new measures are learner achievement, post-16 value added and destinations; in time these will replace the existing measures. A consultation was undertaken to seek views on the new measures and a summary of responses has been published.

2 There has previously been no systematic approach to looking at the sustained destinations of learners for FE or school sixth forms, and no robust way of identifying progression into employment.

3 The Welsh Government secured agreement to participate in the UK

Government’s large-scale data linking programme. Known as the Longitudinal Education Outcomes (LEO) study, it matches learner records to the Department for Work and Pensions’ and Her Majesty’s Revenue and Customs’ data on employment, earnings and benefits, giving a more comprehensive picture of learners’ destinations than has ever been possible before.

4 In order to have the full picture of learner destinations, including education as well

as employment, we have also established an annually updated matched dataset which brings together records for schools, further education, work-based learning and higher education in one place.

5 Having access to the LEOS data and the educational matched dataset enables

us to develop a measure for destinations and over the last few months we have developed our approach and methodology. We have now published a statistical article which provides analysis of the destinations of the 2015/16 and 2014/15 cohorts of leavers, and to give an understanding of the destination measure and how the data looks for your college or school we have sent you your own baseline data.

6 If you have any queries about your baseline data please contact us at

[email protected].

Destinations Measure Overview 7 Post-16 learners in school sixth forms and FE colleges all undertake a

programme of study, consisting of a combination of qualifications. Learning programmes typically consist of main qualifications such as A levels or BTECs, along with additional qualifications such as the Welsh Baccalaureate, Essential Skills Wales, and/or qualifications required by different industries. Learning programmes are the basic ‘unit’ we use as the starting point for the destinations measure.

Cohort 8 The cohort for the measures is based on leavers with terminated programmes in

the academic year (1 August to 31 July) excluding early drop outs i.e. learners

https://consultations.gov.wales/consultations/consistent-measures-post-16-learning-wales

https://gov.wales/statistics-and-research/consistent-performance-measures-post-16-learning-achievement-learner-destinations/?lang=en

https://gov.wales/statistics-and-research/consistent-performance-measures-post-16-learning-achievement-learner-destinations/?lang=en

mailto:[email protected]


who terminate a learning programme within 8 weeks of commencing without completion. ‘Terminated’ programmes include all those that have ended within the academic year, whether or not the learner has completed the programme. Both full and part-time learners are included in the cohort.

9 In the case where a learner terminates multiple programmes within the academic year, the outcome is reported against the most recently terminated learning programme.

10 The data for the learner cohort is derived from the learning programme start and end dates in the:

Lifelong Learning Wales Record (LLWR) (final freeze is taken in the December following the end of the academic year); and

Post-16 collection, which is collected in the October following the end of the academic year.

Measure definitions 11. A learner’s destination relates to their activity in the following academic year. So

for the 2015/16 cohort, destinations are reported for 2016/17.

12. We have used the same definitions for sustained employment and sustained learning as the Department for Education (DfE) in England. The sustained employment measure allows for a one month pause in PAYE employment to reflect that there may be more less permanency and security with employment than in education. Where the pause is in March, activity in April is checked to see if it is a short pause or a more substantial break.

Sustained employment 12 The definition of sustained employment looks at employment activity in the six

month period (October to March) following the end of the academic year in which the learning took place. For the 2015/16 cohort to be counted as in sustained employment:

A learner must be in paid PAYE employment for at least one day a month in five out of the six months between October 2015 and March 2016; or

A learner must have completed a self-assessed return for tax year 2015/16 stating that they have received income from self-employment and their earnings from a Partnership or Sole-Trader enterprise are more than £0 (profit from self-employment).


Figure 1: The eight possible scenarios that lead to a learner being classified as in sustained employment

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Key In self-assessed employment data for tax year following academic year

In PAYE employment

Not in employment

Not within reference period

Learning programmes must terminate at some point

during the academic year, but can start at any point

Academic Year Academic Year + 1

Reference period to terminate learning Destination reference period

Sustained learning

13 The definition of sustained learning looks at learning activity in the six month period between October and March following the end of the academic year in which the learning aim took place. For the 2015/16 cohort to be counted as in sustained learning:

A learner must be in learning (Further Education, Higher Education or Work-Based Learning) at the same or higher level than the 2015/16 programme for at least one day in in each of the six months between October 2016 and March 2017.

14 The baseline data shows analysis separately where the learner has gone into sustained learning at the same level or at a higher level.

Figure 2: The single scenario that leads to a learner being classified as in sustained learning

[1]

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Key In learning







Sustained employment and learning 15 Some learners will meet the definition of both sustained employment and

sustained learning (e.g. they may have progressed onto an apprenticeship programme; be studying full-time with an evening or weekend job; or be in employment and studying part-time).

Sustained destination 16 The ‘sustained destination’ measure aims to count the proportion of learners with

a sustained outcome, either into learning or employment (or both).

17 For 2015/16 leavers to be counted as having a sustained destination, a learner must either have a:

Sustained employment outcome;

Sustained learning outcome; or

Both a sustained employment and sustained learning outcome.

18 National averages are included to enable you to compare your performance against the average rate achieved across all FE colleges and school sixth forms in Wales.

Figure 3: The twelve possible scenarios that lead to a learner being classified as having a sustained destination

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Key Has a sustained employment outcome

In learning

In employment







19 In recognition that the sustained learning measure above may not reflect the full value of further learning at all levels, a secondary measure is presented showing all learning:

Learning (non-sustained) – learners must be in learning at an equal or higher level to the programme they terminated for at least one day between October and March in the following academic year

Positive destination – learners who progress to a sustained employment destination or learning destination (does not need to be sustained)

Figure 4: The single scenario that leads to a learner being classified as in learning

[1]

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Au

g

Sep

Oct

No

v

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Key Not within reference period





Learning for at least one

day in this period

Baseline data tables 20 In October 2018, each school and FE college was sent a set of baseline data

tables showing the destinations data for their own institution.

21 As the data in the tables had not been suppressed we asked that the data was not shared externally or published. Individual learner-level backing data was not included with the initial tables, but it is our intention that this will be incorporated in future reports.

22 There were four worksheets as outlined below: Summary

2015/16

Learners terminating

(excluding early drop-

out)

Sustained employment

only

National average National

average

National

average

Sustained positive

destination

National average


& sustained higher level

learning

Sustained employment &

sustained same level

learning

Sustained higher level

learning only

Sustained same level

learning only

Destinations - Summary

2016/17

Sustained employment & sustained learning Sustained learning only

Gender and age breakdown

Gender Learners terminating


out)

Sustained

employment only


sustained higher level

learning



learning


sustained learning (same

& higher level)

Sustained

higher level

learning only

Sustained

same level

learning only

Sustained learning

only (same &

higher level)

Sustained positive

destination

Sustained lower

level learning only*

Female - - - - - - - - - -

National average N/A - - - - - - - - N/A

Male - - - - - - - - - -


Total - - - - - - - - - -


Gender2015/16 2016/17


Age Learners terminating


out)

Sustained

employment only



learning



learning


sustained learning (same

& higher level)

Sustained

higher level

learning only

Sustained

same level

learning only

Sustained learning

only (same &

higher level)

Sustained positive

destination

Sustained lower


18 and under - - - - - - - - - -


19 to 24 - - - - - - - - - -


25 to 49 - - - - - - - - - -


50 plus - - - - - - - - - -


Total - - - - - - - - - -


Age

Data sources:

Matched education data from PLASC, LLWR, WED and Higher Education Statistics Authority (HESA).

Matched employment and benefits records held by the Department for Work and Pensions and HM Revenue & Customs.

2015/16 2016/17

23 Gender and age are derived from:

LN15 (date of birth) and LN16 (gender) in the LLWR; and

Date of birth and gender from the Post-16 Collection. 24 Age is calculated from the learner’s age as at 31 August of the academic year in

which their learning programme was terminated.

Sector breakdown

Programme** Learners terminating


out)

Sustained

employment only



learning


& sustained same level

learning


& sustained learning

(same & higher level)

Sustained higher

level learning only

Sustained same

level learning

only

Sustained learning

only (same & higher

level)

Sustained

positive

destination

Sustained

lower level

learning only*

Health, Public Services and Care - - - - - - - - - -


Science and Mathematics - - - - - - - - - -


Agriculture, Horticulture and Animal Care - - - - - - - - - -


Engineering and Manufacturing

Technologies

- - - - - - - - - -


Construction, Planning and the Built

Environment

- - - - - - - - - -


Information and Communication

Technology

- - - - - - - - - -


Retail and Commercial Enterprise - - - - - - - - - -


Leisure, Travel and Tourism - - - - - - - - - -


Arts, Media and Publishing - - - - - - - - - -


History, Philosophy and Theology - - - - - - - - - -


Social Sciences - - - - - - - - - -


Languages, Literature and Culture - - - - - - - - - -


Education and Training - - - - - - - - - -


Preparation for Life and Work - - - - - - - - - -


Business, Administration and Law - - - - - - - - - -


Total - - - - - - - - - -


Destinations by sector

2015/16 2016/17

25 This breakdown is only included for vocational learning programmes, as learners

doing general education (A level or ‘mixed’ programmes) are often studying a combination of subjects that cannot be assigned to a specific sector/subject area.

26 The sector is derived from the programme code which can be found in:

LLWR field LP74; and

the Programme Reference field in the post-16 collection.


Level breakdown

Learners

terminating

(excluding early

drop-out)

Sustained

employment

only



learning



learning


& sustained learning

(same & higher level)

Sustained

higher level

learning only

Sustained

same level

learning only

Sustained

learning only

(same & higher

level)

Sustained

destination

Sustained lower


HE programme - - - - - - - - - -


AS programme - - - - - - - - - -


A2 programme - - - - - - - - - -


Vocational - - - - - - - - - -


Access to HE - - - - - - - - - -


All level 3 - - - - - - - - - -


GCSEs - - - - - - - - - -


Vocational - - - - - - - - - -


Access to FE - - - - - - - - - -


All level 2 - - - - - - - - - -


Vocational - - - - - - - - - -


All level 1 - - - - - - - - - -


Vocational - - - - - - - - - -


Other FE - - - - - - - - - -


All other level - - - - - - - - - -


All part-time - - - - - - - - - -


Level 2

2016/17

Level

Destinations by level

2015/16

Level 4+

Level 3

Level 1

Other level

Part-time

27 The level is derived from the programme code which can be found in:

LLWR field LP74; and

the Programme Reference field in the post-16 collection.

Key data sources 28 The measures are calculated using administrative data sources already held by

the government, placing no additional burden on providers, individuals or employers to collect new information.


29 The key data sources used to calculate the measures are as follows:

Lifelong Learning Wales Record (LLWR): contains data on further education, work-based learning and community learning, collected on a ‘rolling’ basis throughout the year with regular statistical freezes; it provides the official source of statistics on post 16 (non higher education) learners in Wales

Ed

uca

tion

al m

atc

hed

data

set

Post 16 Data Collection: every October, all maintained schools with sixth forms and middle schools with pupils in years 12, 13 and/or 14, are required to report all learning programmes and activities undertaken by learners in the previous academic year

Higher Education Statistics Agency (HESA): an annual collection of a range of UK wide data from universities, higher education colleges and other differently funded providers of higher education; this data is then provided to UK governments and higher education funding bodies to support their work in regulating and funding higher education providers

P45 and P14 employment data (held by HMRC): an administrative dataset covering those who pay tax through PAYE through employer submission of P45 and P14, or through completing a self-assessment tax form. The core purpose of this process is to collect tax from those who are eligible to pay it through this mechanism and so there is not complete coverage. Employers are not required to supply information to HMRC for individuals who earn below the tax threshold; although for large employers these individuals are thought to be included due to methods of data transfer.

Em

plo

ym

en

t d

ata

se

t

Matching process 30 Learner records from the LLWR and Post-16 Collection are linked to DWP and

HMRC data to observe benefit and employment activity, and to other education datasets to observe post-learning activity.

Employment destinations

31 The measures have been produced using a matched dataset of person level administrative data sources from Welsh Government, DWP and HMRC. Learners from the LLWR and Post-16 Collection are matched to DWP benefit records and HMRC P45 and P14 income tax returns using a mixture of National Insurance Number and matching on other personal details. The matching algorithm relies on a number of fields being accurately populated across both datasets and additionally, not all learners will necessarily have any record of employment or benefits for legitimate reasons, so the match rate will never reach 100%. For the learners covered by these performance measures, 98% of the 2014/15 and 2015/16 cohorts have been matched to DWP/HMRC data.

32 Once the match is established, the next step is to merge the different data files (employment, benefits, and learners) on the basis of the person level record linkage defined by the matching. The DWP and HMRC datasets provide a record


of those receiving benefits and those paying tax through the self-assessment and Pay As You Earn (PAYE) systems. Processing rules are then applied to transform the data into useable information on employment and benefit receipt to support all analysis.

Learning destinations

33 The measures have been produced using the “matched education dataset”. Following a tender exercise, a contract was issued by the Welsh Government to London Economics to undertake a data linking exercise, attaching a common anonymised identifier to records from multiple education data collections (covering schools, Further and Higher Education Institutions), so that the various datasets can be linked for statistical and research purposes.

Timelags 34 All data used in this process are drawn from administrative sources, which take

time to process and collate. The time lags between the reference period and availability of the dataset for analysis are as follows:

LLWR data are collated from returns by colleges with the provisional data collected to date generally published on a quarterly basis. Returns are not generally complete until up to six months after the end of the academic year, which runs from 1st August to 31st July.

Post-16 Collection data is collated from returns by maintained schools with sixth forms and middle schools with pupils in years 12, 13 and/or 14 in October, and are generally complete by December.

HESA data are collated from returns by institutions and data for the full academic year are available approximately six months after the end of the academic year.

Employment data are matched to DWP data on a regular basis. There are cleaning rules applied to the data, which identify old records when updated with new information. As new information can come through about a job after it has ended this is a source of constant change and historically the data has been considered complete after approximately six months.

34 The future annual cycle for the destinations measures is publication in September

for learning programmes that ended two years before.

Annex A


Glossary of terms Early leaver/early drop out Learners who terminate a learning programme

within 8 weeks of commencing without completion, are treated as early drop outs.

Higher Education Statistics Agency (HESA)

An annual collection of a range of UK wide data from universities, higher education colleges and other differently funded providers of higher education; this data is then provided to UK governments and higher education funding bodies to support their work in regulating and funding higher education providers

Learner cohort The group of learners included in the calculations.

Learning programme A package of learning that has a clear purpose and planned outcomes, delivered through a main qualification and additional learning activities, as set out in the Planning and Funding Framework.

Longitudinal Education Outcomes Study (LEOS)

UK Government data linking programme which matches learner educational records to the Department for Work and Pensions’ and Her Majesty’s Revenue and Customs’ data on employment, earnings and benefits.

Lifelong Learning Wales Record (LLWR)

The means of recording data on learners, their programmes, activities and awards submitted by FE colleges to the Welsh Government.

National average The average rate achieved across all further education colleges and school sixth forms in Wales.

Post-16 collection The means for all maintained schools with sixth forms and middle schools with pupils in Year 12, 13 and/or 14, to report all learning activities and programmes undertaken by learners in the previous academic year.

Terminated learning programme Any learning programme that is recorded as completed or withdrawn.

http://gov.wales/topics/educationandskills/learningproviders/post-16-planning-and-funding/guide-to-the-framework/?lang=en

http://gov.wales/topics/educationandskills/learningproviders/post-16-planning-and-funding/guide-to-the-framework/?lang=en

consistent performance measures - welsh government · sustained destination 4 6experimental...

Documents