kpi project final report - intosai working group on...

the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)

KPI Project Final Report — Key Performance Indicators Methodologyfor Auditing IT Programs

Amy Young, IT Center

China National Audit OfficeApril, 2013 Lithuania

the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 2

CNAOFramework

1. Background

• 2. Project objectives

• 3. Project progress

• 4. Project products


CNAO1. Background

•the 19th of the INTOSAI Working Group on IT Audit , April 2010

– three proposed projects:

• Development of IT Performance Indicators

• Performance measures of IT Solutions implemented in government organizations

• Index System about IT Performance Audit


CNAO1. Background

• the 19th of the INTOSAI Working Group on IT Audit , April 2010

– The above three projects were put into a new one. “Key Performance Indicator Methodology for Auditing IT Programs”.

– SAI China volunteered to be the team leader.

– Team members : SAI Bhutan, China, Ecuador, Japan, Kuwait, Malaysia, Pakistan, Poland, Russia and USA.


CNAOFramework

1. Background

2. Project objectives

3. Project progress

4. Project products


CNAO2. Project objectives

• two principal products

– a set of specific and measurable IT-related indicators

– guideline

•the ideas of KPI application

•feasible KPI evaluation methodology


CNAO2. Project objectives

• Analysis, conclusions and decision making based on results of performance measurement

•exchange views, know-how, and information


CNAOFramework

• 1. Background





CNAO3. Project progress

• Milestones

– By the end of 2011, the beta database of KPI indicators.

– By the end of 2012, the draft Guideline.

– In 2013, the final Guideline and KPI Database.



• the first team meeting on Apr 16, 2010 in Beijing.

• a new name for the project, “Key Performance Indicators Methodology for Auditing IT Programs”.

• the minutes of the first team meeting.



• SAI Russia, Japan , USA , Kuwait and Ecuador explained their own understanding about the project from April to June, 2010.

• The project initiation document to Chair India in July 2010.

• SAI Kuwait submitted the beta database of KPI indicators on Dec 29, 2010.



• The original framework based on SAI Kuwait’s and China’s PKI indicators in Feb, 2011.

• SAI Kuwait, Bhutan and Japan gave the comments in Feb and Mar, 2011.

• the Second team meeting on Apr 16, 2011 in South Africa.

• the minutes of the second team meeting.



• SAI Kuwait submitted the categorized KPI beta database on August 11, 2011.

• SAI Bhutan gave comments on August 18 ,2011.

• SAI China composed the beta indicators and send it to the team members for comments on November 30, 2011.

• SAI Japan and Kuwait gave the comments in

December,2011.



• In the 21st meeting in Kuala Lumpur, Malaysia in January 2012, the team made the report about the KPI project.

• SAI China sent the content framework of guideline to the team members for comments on April 9, 2012.

• In April and May, SAI Kuwait and Bhutan gave the comments and SAI Kuwait applied to compose Part 1 and Part 2.

• SAI China began to compose the draft guideline in June, 2012.



• SAI China delivered the draft guideline for comments among the team members on December 30, 2012.

• SAI Japan, Kuwait and USA feedback in Jan and Feb 2013.

• In early March 2013, SAI China send the final guideline v1 to Chair India for comments among all the WGITA members.



• SAI Bangladesh, Brazil, Iraq and Russia gave the comments.

• Based on the comments, SAI China made the final guideline v1.1.

• Comments from SAI Malaysia, Lithuanian and Qatar.

• In Apr 2013, the final guideline V1.2 was delivered for approval in the 22nd meeting of WGITA .



• Communication Management Strategy

– mainly based on email through Internet

– email contact list for the team members

– sometimes discuss trough telephone


CNAOFramework

• 1. Background





CNAO4. Project products

• Criteria for the indicator

– The indicators: be widely accepted and measurable

– the data source of the evaluating indicators : be accessible and analyzable

– the calculation methods of the evaluating indicators: be relatively simple


CNAO4.1- KPI database

• 11 parts, 367 indicators with three-level.

• 1. Decision (15 indicators)

– Compliance with the laws

– Feasibility study

– Participation in decision

• 2. Requirement analysis (13 indicators)

– Organization target

– Core business coverage

– Response/change



• 3. Design / Planning (45 indicators)

– Requirement coverage

– Time limit

– Capacity Planning and Resource Provisioning

– Cost estimation

– IT risks



• 4. Procurement/Development (48 indicators)– Selection for partner or supplier

– Cost control

– Process Control

– Code control

– Outsourcing

– Quality Control

– Testing

– Training

– Upgrading



• 5. Product (26 indicators)

– User satisfaction

– Price

– Delivery

– Performance

– Integration

– Technology applicability



• 6. Maintenance (125 indicators)

– Follow the management rules

– Incident management

– System Usability

– Availability

– Maintenance cost

– Website

– Monitoring

– Change Management

– Data Center



• 7. Security (26 indicators)

– Security plan

– Identity management

– User account management

– Security testing monitoring

– Security incident definition

– Malicious software prevention

– Network security



• 8. Backup and Disaster Recovery (14 indicators)

– Backup plan

– Recovery plan

– Backup operation management

– Recovery operation management

• 9. Service (32 indicators)

– Service request

– Service Response

– Service Satisfaction



• 10. Effectiveness (19 indicators)

– Coverage of the core business

– Benefit

– Internal Management optimizing

– Public Service

• 11. Others (4 indicators)


CNAO4.2 - Guideline

• Part 1. Preface

• Part 2. Audit Plan

• Part 3. Audit Implementation

• Part 4. Audit report

• Reference : INTOSAI training materials, ISACA,

COBIT, ITIL and others.

the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)

Thanks

for your attention!

China National Audit OfficeApril, 2013 Lithuania

kpi project final report - intosai working group on...

Documents