ICT622ICT622 11

Chapter 13Chapter 13Measuring, Reporting, and ControllingMeasuring, Reporting, and Controlling

Managing the Information Managing the Information Technology ResourceTechnology Resource

Paula GouldingPaula Goulding

ICT622ICT622 22

Chapter OutlineChapter Outline Importance of measurementImportance of measurement What and how to measureWhat and how to measure Quality and efficacy of measuresQuality and efficacy of measures Reporting to different audiencesReporting to different audiences Controlling IT through effective SLAsControlling IT through effective SLAs Negotiating a service level agreementNegotiating a service level agreement Making SLAs work – metricsMaking SLAs work – metrics Introducing internal SLA to firmIntroducing internal SLA to firm SLA mythsSLA myths Who owns measurements and what/when to do themWho owns measurements and what/when to do them Can enterprise succeed with or without Can enterprise succeed with or without

measurements?measurements?

ICT622ICT622 33

Why Is There IT?Why Is There IT?

Mostly profit, some cost, 24%

Mostly cost, some profit,

41%

Pure cost center, 25%

Pure profit, 10%

ICT622ICT622 44

Productivity Study of IT and Business ExecutivesProductivity Study of IT and Business Executives

Then why don't we get

recognition for our contribution?,

97% Yes

Do productivity gains translate

into a more profitable

business?, 3% No

Source: InformationWeek research productivity study of 300 IT and business executives

ICT622ICT622 55

Measuring ITMeasuring IT

IT expenditure as percent of revenueIT expenditure as percent of revenue IT expenditure per employeeIT expenditure per employee Revenue per IT dollarRevenue per IT dollar Total IT expense budget vs. actualTotal IT expense budget vs. actual Total IT capital budget vs. actualTotal IT capital budget vs. actual Employees supported per IT employeeEmployees supported per IT employee

ICT622ICT622 66

Importance of MeasurementImportance of Measurement

Business wants to know if IT is being run Business wants to know if IT is being run efficiently and effectivelyefficiently and effectively

Identify opportunities for improving Identify opportunities for improving effectiveness and efficiencyeffectiveness and efficiency

EffectivenessEffectiveness– Doing things rightDoing things right

EfficiencyEfficiency– Doing things the right wayDoing things the right way

ICT622ICT622 77

Measuring IT – SamplesMeasuring IT – Samples

Overall – qualitative measuresOverall – qualitative measures Overall – quantitative measuresOverall – quantitative measures NetworkNetwork Data centerData center

ICT622ICT622 88

What and How to MeasureWhat and How to Measure

Need to measureNeed to measure– Business impactBusiness impact– Customer relationshipsCustomer relationships– Internal organization impactInternal organization impact– Investment impactInvestment impact– Value chain impactValue chain impact

Know how processes are changing between any Know how processes are changing between any measurements, and explain any changes in measurements, and explain any changes in resultsresults

No ideal set of metricsNo ideal set of metrics

ICT622ICT622 99

10-Step Approach to 10-Step Approach to Developing MeasurementsDeveloping Measurements

1.1. Commitment of resources, time, and cooperation Commitment of resources, time, and cooperation from senior business and IT managementfrom senior business and IT management

2.2. Have and review formal strategic plan, mission Have and review formal strategic plan, mission statements, and goalsstatements, and goals

3.3. Committees need to be established for each of Committees need to be established for each of 38 IT processes38 IT processes

4.4. Each committee meets and defines goals of each Each committee meets and defines goals of each IT process as first objectiveIT process as first objective

5.5. Identify critical success factorsIdentify critical success factors

ICT622ICT622 1010

10-Step Approach to 10-Step Approach to Developing MeasurementsDeveloping Measurements

6.6. Establish guidelines to measure if, and to what Establish guidelines to measure if, and to what degree, critical success factors have been metdegree, critical success factors have been met

7.7. Complete pilot should be done and members Complete pilot should be done and members jointly participate in the pilotjointly participate in the pilot

8.8. Recommendation to install these measurements Recommendation to install these measurements should be raised to senior management for should be raised to senior management for endorsementendorsement

9.9. Distribution and implementation of new measuresDistribution and implementation of new measures10.10. Analyze results and review with all partnersAnalyze results and review with all partners

ICT622ICT622 1111

5 Stages of Benchmarking5 Stages of Benchmarking

PlanningPlanning AnalysisAnalysis IntegrationIntegration ActionAction MaturityMaturity

Source: Robert C. Camp and http://www.apqc.org

ICT622ICT622 1212

BenchmarksBenchmarks

PlanningPlanning

– Identify what is to be Identify what is to be benchmarkedbenchmarked

– Identify comparative Identify comparative companiescompanies

– Determine data Determine data collection method collection method and collect dataand collect data

Analysis Analysis – Determine current Determine current

performance “gap”performance “gap”– Project future Project future

performance levelsperformance levels

ICT622ICT622 1313

BenchmarksBenchmarks

IntegrationIntegration– Communicate benchmark Communicate benchmark

findings and gain findings and gain acceptanceacceptance

– Establish functional goalsEstablish functional goals

ActionAction– Develop action plansDevelop action plans– Implement specific Implement specific

actions and monitor actions and monitor progressprogress

– Recalibrate benchmarksRecalibrate benchmarks

MaturityMaturity– Leadership position Leadership position

attained and practices attained and practices fully integrated into fully integrated into processprocess

ICT622ICT622 1414

Iterative Process for BenchmarkingIterative Process for Benchmarking

Source: Dataquest, Gartner Group.

ICT622ICT622 1515

Quality of MeasuresQuality of Measures

Often best when forecasting trendsOften best when forecasting trends Can lead to misleading resultsCan lead to misleading results Gain correct insight as to what Gain correct insight as to what

measurement indicatesmeasurement indicates Good quality may be significant indicator of Good quality may be significant indicator of

good performancegood performance

ICT622ICT622 1616

Dimensions of Service QualityDimensions of Service Quality

ReliabilityReliability– Consistency of performance and dependabilityConsistency of performance and dependability

ResponsivenessResponsiveness– Willingness/readiness of employees to provide service in Willingness/readiness of employees to provide service in

timely mannertimely manner CompetenceCompetence

– Possession of required skills to perform servicePossession of required skills to perform service AccessAccess

– Approachability and ease of contactApproachability and ease of contact CourtesyCourtesy

– Politeness, respect, consideration, friendlinessPoliteness, respect, consideration, friendliness

ICT622ICT622 1717

Dimensions of Service QualityDimensions of Service Quality CommunicationsCommunications

– Keeping customers informed in language they Keeping customers informed in language they understandunderstand

CredibilityCredibility– Trustworthiness, believability, honestyTrustworthiness, believability, honesty

SecuritySecurity– Freedom from danger, risk, doubtFreedom from danger, risk, doubt

Understanding/Knowing the CustomerUnderstanding/Knowing the Customer– Making effort to understand customer’s needsMaking effort to understand customer’s needs

TangiblesTangibles– Physical evidence of servicePhysical evidence of service

ICT622ICT622 1818

Dimensions of Successful Dimensions of Successful IT FunctionsIT Functions

Service qualityService quality System qualitySystem quality Information qualityInformation quality UseUse User satisfactionUser satisfaction Individual impactIndividual impact Work group impactWork group impact Organizational impactOrganizational impact

ICT622ICT622 1919

Source: Anthony, R.N. Planning and Control Systems: A Framework for Analysis, Cambridge, MA: Harvard University Press (1965).

ICT622ICT622 2020

Variety in ReportingVariety in Reporting

Historical DataHistorical Data Statistical DataStatistical Data Assist managers in determining repairsAssist managers in determining repairs Each has different characteristicsEach has different characteristics

ICT622ICT622 2121

Characteristics of Effective SLAsCharacteristics of Effective SLAs

Based on current “user” expectationBased on current “user” expectation Defined by location, function, and service typeDefined by location, function, and service type Uses benchmarks and baselines for measuring Uses benchmarks and baselines for measuring

performanceperformance Specifies minimums, penalties, and incentivesSpecifies minimums, penalties, and incentives Specifies reports and tracking tools to be Specifies reports and tracking tools to be

monitoredmonitored Agreement not too few, not too manyAgreement not too few, not too many Gets adjusted periodically to reflect changesGets adjusted periodically to reflect changes

ICT622ICT622 2222

Negotiating a SLANegotiating a SLA

Get a BaselineGet a Baseline Define ResponsibilityDefine Responsibility Allow for ModificationsAllow for Modifications Ensure Reports Are in Terms Anyone Can Ensure Reports Are in Terms Anyone Can

UnderstandUnderstand Don’t Forget the End-User ExperienceDon’t Forget the End-User Experience

ICT622ICT622 2323

Examples of Service Level MetricsExamples of Service Level Metrics

Application availabilityApplication availability Average application response timeAverage application response time Number of application crashes per Number of application crashes per

unit of timeunit of time Average throughoutAverage throughout Network availability and bandwidthNetwork availability and bandwidth Server availabilityServer availability

ICT622ICT622 2424

Examples of Service Level MetricsExamples of Service Level Metrics

Elapse time to repair hardware failureElapse time to repair hardware failure Mean time between server failureMean time between server failure Number of operating system failure per Number of operating system failure per

unit of timeunit of time Number of middleware failure per unit of Number of middleware failure per unit of

timetime Number of production jobs not Number of production jobs not

completed during batch night shiftcompleted during batch night shift

ICT622ICT622 2525

Questions Concerning Questions Concerning Who Owns MeasurementWho Owns Measurement

Who should own measurement?Who should own measurement? Is measure a process that should be owned Is measure a process that should be owned

by the people doing the measurement?by the people doing the measurement? Is it a corporate asset or cost?Is it a corporate asset or cost? Is this question similar to who should own Is this question similar to who should own

payroll?payroll? Should the owners of internal processes Should the owners of internal processes

that are being measured simply own it?that are being measured simply own it? Does it make a difference?Does it make a difference?

ICT622ICT622 2626

What to do with MeasurementsWhat to do with Measurements

Establish report card for performanceEstablish report card for performance Create effective and understandable Create effective and understandable

benchmarksbenchmarks Senior management needs to know current Senior management needs to know current

status and effectiveness of ITstatus and effectiveness of IT

ICT622ICT622 2727

Performance Measures for Performance Measures for Efficient Order ManagementEfficient Order Management

Short order lead timesShort order lead times In-stock availabilityIn-stock availability Order accuracyOrder accuracy Access to order status informationAccess to order status information Response time to customer inquiriesResponse time to customer inquiries

ICT622ICT622 2828

Performance Measures for Performance Measures for Efficient Customer ManagementEfficient Customer Management

Lower inventory carrying costLower inventory carrying cost Reduced human interventionReduced human intervention Greater order accuracyGreater order accuracy Improved production planning/forecastingImproved production planning/forecasting Lower order administrationLower order administration Lower number of complaintsLower number of complaints Improved customer decision making based on Improved customer decision making based on

preferencespreferences Increase number of automatic restocking Increase number of automatic restocking

customerscustomers Lower stock-out occurrencesLower stock-out occurrences

ICT622ICT622 2929

When Measurements Should be DoneWhen Measurements Should be Done

ContinuouslyContinuously DailyDaily WeeklyWeekly MonthlyMonthly SemiannuallySemiannually AnnuallyAnnually

ICT622ICT622 3030

Can Enterprise Succeed Can Enterprise Succeed Without Measurement?Without Measurement?

Only 42% of companies prepare business Only 42% of companies prepare business case for CRM projectcase for CRM project

Only 45% have centralized CRM Only 45% have centralized CRM responsibilityresponsibility

57% can’t justify investments because they 57% can’t justify investments because they don’t know how to measure customer don’t know how to measure customer profitabilityprofitability

Only 10% measure ROIOnly 10% measure ROI

ICT622ICT622 3131

How Does Enterprise How Does Enterprise Benefit from Measurement?Benefit from Measurement?

Performance is related to effectiveness, Performance is related to effectiveness, efficiency, and qualityefficiency, and quality

To understand process it must be measured To understand process it must be measured and comparedand compared

Part of monitoring project performancePart of monitoring project performance Determine if vendors are performing to Determine if vendors are performing to

service level agreementsservice level agreements