key performance indicators (kpis) background documents

PLANNING AND INFORMATION OFFICE

Office of the Provost and DVC

Key Performance Indicators (KPIs)

The Strategic Information and Business Intelligence (SIBI) reporting tool is an ideal mechanism through

which to publish performance metrics for the University.

Performance reporting is an evolving space. The current KPI dashboard (May 2014) is the first formal

publication of KPIs through SIBI and represents the initial step in establishing an agreed set of institutional

metrics; it is not a definitive or final list. The existing suite of KPIs is expected to evolve as University

leaders develop a shared understanding around which metrics are key to measuring performance and as

more data becomes available for inclusion.

The current module contains a diverse set of KPIs and aims to encourage discussion to guide and inform

future KPI development. This incremental approach is preferable as it acknowledges that KPIs are flexible

and will change in response to institutional requirements and reporting maturity. Feedback on current

performance metrics, or suggestions for potential metrics, can be submitted to the Provost and SIBI

Program Board by emailing [email protected].

To date the focus has been on institutional/faculty KPI development. Indicators for PSUs will be developed

at a later stage.

Background documents

Appendix 1: KPIs – current status and candidate metrics

During the initial KPI development phase SEG considered a draft list of potential KPIs for measuring

institutional, faculty and school performance across a range of areas aligned to the goals of the University’s

Strategic Plan 2011-2015. Some candidate KPIs were non-controversial, others were identified as requiring

further investigation and a third group lacked a robust data source. Some were abandoned following the

SEG review.

Appendix 1 outlines the current status of these candidate KPIs. The indicators already published have been

approved by both the Provost and the Head of the relevant portfolio.

Appendix 1 outlines:

the initial set of KPIs already published through SIBI

KPIs under development for release soon via SIBI

indicators broadly supported by SEG that are candidates for future SIBI development

indicators which require further work to determine the optimal methodology

indicators which may have value but need scoping and costing for data acquisition.

Appendix 2: The KPI framework

The underlying framework was endorsed by SEG in February 2013. Since that time, the development and

release of metrics has shifted to an incremental or phased approach.

Director Planning and Information

May 2014

mailto:[email protected]

Appendix 1

Planning and Information Office

1.1 Initial set of KPIs available in SIBI

Area Indicator Metric Drill to…

Research

Research income Total HERDC research income by year

($m) Faculty

Research outputs Total HERDC weighted research

publication points Faculty

Student Diversity

Indigenous Access, Participation, Attainment, Success Faculty, UG / PG(Cw) /

PG(Res)

Low SES Access, Participation, Attainment, Success Faculty, UG / PG(Cw) /

PG(Res)

Gender % split Female / Male, Access,

Participation, Attainment, Success

Faculty, UG / PG(Cw) /

PG(Res)

Disability Access, Participation, Attainment, Success Faculty, UG / PG(Cw) /

PG(Res)

Regional / Remote Access, Participation, Attainment, Success Faculty, UG / PG(Cw) /

PG(Res)

Student Outcomes UG/PG Success UG/PG(Cw) success rates Faculty, F-1, F-2, Dom/Int

PG research completions HDR completions Faculty, Dom/Int

Financial

Sustainability

Course enrolments Variance to budget Faculty, Dom/Int,

UG/PG(Cw)

Load Variance to budget Faculty, F-1, F-2, Dom/Int,

UG/PG(Cw)

Student revenue Variance to budget Faculty, F-1, F-2, Dom/Int,

UG/PG(Cw)

Internal cohort mix

International students (%) Faculty

Domestic students (%) Faculty

Undergraduate students (%) Faculty

Postgraduate coursework students (%) Faculty

Postgraduate research students (%) Faculty

Brand Overall brand strength Net Promoter score Faculty

Staff Satisfaction

Staff engagement The index value for Passion / Engagement

as calculated by Voice Project.

Progress The index value for Progress as calculated

by Voice Project.

Wellbeing An index value for Peace as calculated by

Voice Project.

1.2 KPIs under development for release soon via SIBI


Student Outcomes UG/PG Retention UG/PG(Cw) retention rates Faculty, Dom/Int

Student Experience *

Overall course satisfaction (SCEQ)

Agreement and broad agreement (%) Faculty, Dom/Int, PG/UG

Good teaching (SCEQ) Agreement and broad agreement (%) Faculty, Dom/Int, PG/UG

Unit of study satisfaction UoS with ≥ 80% satisfaction (% of total) PG/UG

Unit of study satisfaction UoS with ≥ 20% dissatisfaction (% of total) PG/UG

HDR: Overall satisfaction (SREQ)

Agreement and broad agreement (%) Faculty, Dom/Int, PhD/M

HDR: Supervision (SREQ) Agreement and broad agreement (%) Faculty, Dom/Int, PhD/M

HDR: Generic Skills (SREQ) Agreement and broad agreement (%) Faculty, Dom/Int, PhD/M

HDR: Student enrolments in UoS

Number of UoS enrolments Faculty, Dom/Int, PhD/M

Development Fundraising achievement Funds raised (% growth and dollars) Faculty, F-1

Donors (% growth and number) Faculty, F-1, donor type

Alumni Under discussion Faculty

KPIs – current status and candidate metrics

Appendix 1


1.3 KPIs with broad support from SEG and candidates for inclusion in SIBI development.

Yet to be finalised.


Curriculum eLearning UoS with eLearning component (%) Faculty, F-1

Staff Diversity

Indigenous staff Indigenous staff (%) Faculty, F-1

Gender profile Gender indicator (by type and level) Faculty, F-1

Age profile Age indicator (by type and level) Faculty, F-1

Research student satisfaction

Research supervision PREQ quality of supervision Faculty, F-1

Research outcomes

ERA ranking Disciplines rated world class of above (%) FoR

Competitive research income

Category 1: Aust Competitive Grants ($m) Faculty, F-1

Category 2: Other public sector ($m) Faculty, F-1

Category 3: Industry and other ($m) Faculty, F-1

Category 4: CRC ($m) Faculty, F-1

Outputs

Books weighted (A1) Faculty, F-1

Books chapters (B1) Faculty, F-1

Journal articles (C1) Faculty, F-1

Conference papers (E1) Faculty, F-1

Creative works Faculty, F-1

Cross disciplinarity Publications with more than one ERA code Faculty

Community and International Engagement

Linkage grants Number of application partners Faculty, F-1

ARC Funding ($m) Faculty, F-1

Contract research income

Category 3: Australian contract income ($m) Faculty, F-1

Category 3: Other international income ($m) Faculty, F-1

Consultancy income ($m) Faculty, F-1

Contract income ($m) Faculty, F-1

Sustainability

Cost of administrative services Revenue allocated to admin services (%) Faculty, F-1

Directly controllable margin Variance to budget (%) Faculty, F-1

Change on previous year (%) Faculty, F-1

Operating margin Variance to budget (%) Faculty, F-1

Change on previous year (%) Faculty, F-1

Appendix 1


1.4 KPIs that potentially provide performance insight but need further investigation, definition

and/or methodology review. Some can be implemented relatively easily.

Area Indicator Metric Comments

Risk management

WHS Serious incidents/near misses (per 100 FTE)

These have been

proposed by Audit and

Risk Management Office.

These risk indicators

need further discussion.

They are likely to form a

separate dashboard for

restricted FIC and FAC

access.

Lost time due to injury

Teaching Student/staff ratios

ATAR cut offs

Source country diversity Students from major market (%)

Research Research grants (movement)

Ethics and/or misconduct breaches

Finance Load outcomes against targets

Salaries (% of revenue)

HR Staff turnover rates

Business continuity UoS/courses available in other than face to face mode (in the event of a major shutdown or pandemic)

Demand

Quality

ATAR data – cut offs (prior to bonus schemes), median, range

Demand measures need further consideration. Some data needs to be defined and sourced.

Gaukau scores, and other IFee measures

Volume Change in eligible applications (%) UG and PG

Market share Market share (%) UG and PG

Conversion rates Application to offer to enrolment conversion – UG/PG, Domestic/International, research/coursework. By course.

Curriculum

Curriculum review Percentage internal/external review Measure TBC

Quality Assurance Performance + QA

Interdisciplinary courses Measure TBC

Graduate learning

outcomes Employer satisfaction Net Promoter Score (Employer)

Data available late 2013 from Brand Tracking

Research student outcomes

PG research completions Rate (%) [methodology tbc] Yes

Research

Commercialisation Contracts providing commercial return (%) Measure TBC

Quality of publications

Citation rates

For further investigation Impact factor

Nature/Science pubs?


CELT

Students reporting CELT learning experiences (SCEQ) (%) Measures TBC

High quality community relationships (#)

Co-authored publications with overseas academics

Rate of co-authored publications (%) HERDC

Measure TBC

Students as global citizens

Measure TBC

Sustainability

Academic staff promotions Number, rate, success Yes

Productivity Staff achieving objectives in alignment with SP through APD process (%)

Environmental sustainability

Carbon footprint (USM) USM metric to be defined PIO development required

Appendix 1


1.5 KPIs that are potentially valuable, but need scoping for data acquisition and associated budget

estimate.

All require significant investigation

Area Indicator Metric Comments

Graduate outcomes

Quality outcomes More discussion required

All need further investigation

Student satisfaction Graduate survey 1, 3, 5 years

Employer satisfaction Employer survey

UoS with appropriate QA of assessment (%)

Research student outcomes

Graduate destinations (5 yr out)

TBC


Non research consultancy and contract

Number of non-research consultancies and contracts

Positions on external committees

rate per FTE (or % staff). External interests policy as start point?

Honours and awards rate per FTE (or % staff). Use ERA definitions?

International course accreditation

Internationally accredited courses (#)

Staff

Support for staff APD completions, academic (%)

GPD completions, professional (%)

Honorary/Visiting academics

Doctoral qualifications Staff with doctoral qualification (%) (??)

Other qualifications Staff engaged in continuing professional dev (%)


Planning and Information Office Office of the Provost and DVC Level 6, Quadrangle

T +61 2 9036 6490 M +61 434 568 744 F +61 2 9351 7301 E [email protected] sydney.edu.au

ABN 15 211 513 464 CRICOS 00026A

SUBMISSION TO SEG

28 February 2013

INSTITUTIONAL KEY PERFORMANCE INDICATORS (KPIs)

Authorisation: Stephen Garton (Chair SIBI Board) 2013/02

SEG members will be aware of the current business intelligence project under development, the Strategic Information and Business Intelligence Program (SIBI) – a collaborative project between Planning and Information Office and ICT. See: http://sydney.edu.au/staff/planning/sibi/index.php One of the important deliverables of SIBI is to establish coherent reporting against agreed institutional and faculty objectives through desktop, ‘one touch’ enquiry. We are now at the stage in SIBI development where discussion of potential institutional Key Performance Indicators (KPIs) will help drive SIBI progress, focus business readiness requirements and prioritise technical development. In order to stimulate this discussion a proposed KPI Framework for the University is now submitted to SEG for review. This draft Framework was reviewed and endorsed by the SIBI Board, chaired by the Provost, in November 2012 and includes feedback from that discussion as well as feedback from other stakeholder groups within the University. The KPI framework will be an evolving model, that will necessarily be adapted over time to reflect changing University priorities. At this point, SEG members are asked to review this draft and provide feedback on the proposed set of KPIs as measures of institutional and faculty performance that reflect the University’s current reporting requirements, noting that these KPIs can (and will likely) be amended over time in response to internal and external drivers. For example, TEQSA Standards (expected at the end of 2013) may necessitate change to the set of measures for learning and teaching outcomes (eg peer review of our graduates). The framework needs to be sufficiently flexible to be able to respond to such changes in order to guide data gathering requirements and future SIBI technical developments. To date the focus has been on institutional/faculty KPI development – KPIs for PSUs will follow. There are still a number of areas in this framework that need to be finalised, therefore SEG input at this stage of development would help provide direction. For those measures that are relatively straightforward the SIBI team can commence development of reports while discussion continues around the less straightforward measures. Recommendation That SEG review the proposed set of KPIs as the next step in ongoing delivery of performance reporting through SIBI, and provide feedback for finalisation of the overall KPI Framework. S Harrison Director Planning and Information 15 February 2013

Planning and Information Office Office of the Provost and DVC


T +61 2 9036 6490 M +61 434 568 744

ABN 15 211 513 464 CRICOS 00026A

Reporting Performance: An Institutional KPI Framework Consultation Papers

The establishment of a framework for monitoring institutional performance is a companion development to SIBI progress and implementation. We are now at the stage where discussion of a draft set of Key Performance Indicators (KPIs) will help drive SIBI progress, focus business readiness requirements and prioritise technical development. The breadth of performance reporting needs to be relevant and manageable. At the same time, however, we need to maximise the SIBI return on investment – so that some extended data sets will be included in the SIBI reporting capabilities. Developing the Key Performance Indicator Framework There are multiple sources and drivers for KPIs, providing a list of hundreds of candidate indicators. The sources include: The University’s Strategic Plan (2011-2015) The current suite of institutional KPIs (approved by Senate in 2010) Workslate projects The University’s Risk Framework The original 27 faculty KPIs (implemented in 2007) The previous Balanced Score Card project Indicators developed for each PSU/Portfolio (HR, Marketing, DVC(E) for instance) TEQSA regulatory requirements Individual faculty requirements The draft KPI Framework (Attachments 1 and 2) has been prepared following initial consultation with relevant portfolios and PSUs and following criteria agreed by the SIBI Board earlier in 2012 (Attachment 4). The detailed draft list of KPIs and measures (Attachment 3) is seen as a starting point for broad based discussion and consultation. Rationale The final institutional KPI framework must include indicators that are evidence-based measures of the University’s performance, in the context of the Strategic Plan, not a series of measures that simply count outputs. It is also essential that the framework incorporates measures over which the University has performance control, not those where outcomes are substantially governed by external environments, and that they truly add value to monitoring institutiona/faculty performance and to informing decision making. The aim is to provide some level of flexibility, so that Indicators can be shifted between Level 1 and Level 2 in response to changing University priorities. Availability of data KPIs are proposed in this Framework on the basis of meeting the University’s needs, rather than on the current availability of data. Once the indicators are agreed as an institution, we will assess the requirements for data capture, review the feasibility and cost of such data capture and make recommendations on if/how to proceed on a priority basis. As an institution we are at a stage of maturity where we need to move from simply using ‘what is available’ to determining ‘what is required’. Targets and benchmarking Once the framework and indicators are agreed and the relevant SIBI technical development is complete, the key next step is to develop targets for most indicators (there could be some indicators where setting targets is not sensible). This is in itself quite a major undertaking and likely best approached once there is a level of institutional maturity in understanding the data and in conjunction with external benchmarking. S Harrison 15 February 2013


Attachment 1

S Harrison, December 2012 Attachment 1, Page 1

Reporting Performance: An Institutional KPI Framework

Three levels of KPIs are proposed.

CORE KPIs (at two levels) track performance at each of University, faculty and school levels in the following

categories: Brand Mutual accountability Engaged enquiry

– Education – Research – Community and international engagement – Alumni and development

Sustainability – Human resources – Financial – Environmental

Risk factors

LEVEL 1 Core KPIs provide aggregated data as a snapshot of University, faculty and school performance. LEVEL 2 Core KPIs provide granular and complementary information to supplement Level 1 KPIs, also at University, faculty and school levels.

Indicators can be shifted between Level 1 and Level 2 in response to changing University priorities. Both Level 1 and Level 2 KPIs measure University, faculty and school performance. The distinguishing feature between the two is the level of information granularity (for instance student cohort or research grant type).

ENABLING KPIs reflect the strategic goals of each of the PSUs, and track the performance of each unit in

achieving those goals and supporting the University’s core business.

At a glance…

Indicator Type

Characteristic Applies to

Aggregated Granular University Faculty School PSU

Core (Level 1)

Core (Level 2)

Enabling

Targets and Benchmarking Where it is sensible to do so, targets will be considered for each of the KPIs – once the indicators are agreed across this institution. Internal benchmarking and Go8 benchmarking from a discipline perspective will be included where data are readily available.

S Harrison, December 2012 Attachment 1, Page 2

KPI characteristics

CO

RE

KP

Is

Leve

l 1

Core (Level 1) indicators reflect faculty performance, at an aggregated level, and the contribution to institutional performance. These KPIs measure performance across our core business of Learning &Teaching and Research,

and include some sustainability measures (eg those aspects finance and HR for which deans are responsible).

There is drill down from Faculty to School and, conversely, data can be aggregated to University level.

On the whole, they each meet the candidate KPI criteria (see Attachment 3). These KPIs will be the most visible on SIBI.

Leve

l 2

Core (Level 2) indicators are a more granular measure of faculty performance, complement the aggregated KPIs of Level 1 and, again, show the contribution to overall institutional performance. These indicators also measure performance across our core business of Learning &Teaching and

Research, but at a more granular level. They include some sustainability measures (eg those aspects of finance and HR for which deans

are responsible). There is drill down from Faculty to School and, conversely, data can be aggregated to University

level. There is also drill down to more granular cohorts or categories. For example:

Level 1 indicator Level 2 indicator

Total research income Category 1 income Category 2 income and so on

Variance to commencing enrolment target (%) (all students)

Variance to commencing target (%) (CSP) Variance to commencing target (%) (Domestic Fee) Variance to commencing target (%) (International) and so on

Includes any other KPIs that we consider valuable, but not sufficiently critical to be in Level 1. Includes any faculty specific KPIs. There is flexibility between Level 1 and Level 2 – KPIs can be promoted/demoted between the two

levels, depending on the University’s changing strategic priorities.

EN

AB

LIN

G K

PIs

Enabling KPIs reflect the performance of Professional Support Units (PSUs) in supporting achievement of institutional objectives embedded in Core KPIs These indicators are likely to be developed later in 2013 (although some PSUs are well on the way

to developing these). SIBI will concentrate on reporting against Core KPIs as priority. HR, Finance and Development/Alumni will have Enabling KPIs that measure their own professional

performance (for example, metrics around recruitment performance, or central financial management). However there will also be KPIs related to these areas that measure faculty performance (included in Core Levels 1 and 2) (for instance revenue measures and staffing measures).

ICT, CIS, Library will have only have Enabling KPIs, as there is no delivery on this by faculties. Some work was commenced on similar KPIs a few years ago, and may still be relevant.


2

Appendix 2

Attachment 4


T +61 2 9036 6490 M +61 434 568 744

ABN 15 211 513 464 CRICOS 00026A

Institutional KPI Framework: Criteria for indicator selection


swFinal September 2012

1

Relevance

Each KPI needs to be aligned with the University’s Strategic Plan (SP), so that it is transparently relevant to institutional goals (explicit and implicit). Explicit goals derive directly from the SP. Implicit goals include those embedded in Work Slate projects, the Risk Framework, TEQSA and other legislative requirements. Key to satisfying this criterion is the need to ensure the KPI is measuring strategic performance rather than operational goals.

2

Depth

KPIs need to drill down through the organisation so that divisions/faculties/schools can view how they contribute to the whole of institution performance.

3

Driver to impact real change

3.1 Controllable The University needs to be able to control the performance being measured and be able to impact performance, ie the KPI measures outcomes that are managed or controllable by the University and are not principally determined by outside factors. 3.2 Targets and benchmarking The target is agreed by all stakeholders and external benchmarking (sector or international) is used to guide target setting. 3.3 Timely Where possible the KPI is a lead indicator (although true lead indicators are rare) and performance measurement is possible in a time frame that will allow changes in process to impact results.

4

Supports legislative requirements

The extent to which the KPI supports TEQSA review/audit requirements and/or legislative requirements.

5

Accountable

Clear accountabilities for each performance KPI outcome can be established – one owner clearly identified for each.

6

Clarity

The KPI does not require specialised knowledge and makes sense without detailed explanation.

key performance indicators (kpis) background documents

Documents