Impact of Dominant Metrics

“Simple can be harder to achieve than keeping it

complex. You have to work hard to

make it simple. But it’s worth it in the end because once you get there, you

can move mountains.”

Objectives• Understand what is “Dominant Information”

• Understand who should use or who has the most to gain by using “Dominant Information”

• Identify an example where Dominant Information could have assisted you

3

Metrics vs. Dominant Metrics

Is There A Difference?

4

Case Study – ASU Dining Service• 2007, ASU implements the Best-Value process to procure Dining Services

• ASU Student Population = 72,000

• $1+ Billion Dining Service contract

• 15-Year Contract

• Results:• 62% in Revenue• $33 Million

5

Documenting Success• We will document:

• Commissions paid to University• Monthly sales per location• Conference sales• Equipment depreciation schedule• Capital improvement depreciation schedule• Average check per customer• Sales per labor hour• Annual statement of operations• Number and type of meal plan sold• Non-commissionable revenue detail• Mandatory and Voluntary meal plan rate (number and $)

6

Performance Metrics

8

Is This Dominant?• Increase sale of food by 14%• Increased money to ASU by 23%• Decreased management cost by 80%• Increased customer satisfaction by 37%

9

CRITERIA2006-2007

(Incumbent)2007-2008

(Year 1)DEVIATION

Total Revenue ($) 27,000,000$ 30,830,000$ 14%Total Commissions ($) 2,170,000$ 2,670,000$ 23%Capital Investment ($) 14,750,000$ 30,830,000$ 109%ASU Administration Effort (#) 7.0 1.5 -79%Student Satisfaction (1-10) 5.2 7.1 37%

Is This Dominant?

10

CriteriaYear 1(From

Incumbent)Year 2

(From Year 1)Year 3

(From Year 2)Year 4

(From Year 3)

Sales 14% Increase 11% Increase 24% Increase 13.5% Increase

Commission 23% Increase 6% Increase 20% Increase 22% Increase

ASU Management Requirement Reduced 79% -- -- --

Student Satisfaction 37% Increase 1% Decrease 9% Increase 3% Increase

The difficulty of making things simple can cause

us to ignore creating “dominant metrics”

11

BenchmarkingDoes It Really Make A

Difference?

12

State of Idaho Inmate Health Services• Scope of Work: Provide health services to correctional inmates across

Idaho (13 facilities – approximately 5,000 inmates). This includes Healthcare Services, Mental Health Services, and Pharmaceutical Services.

• Length of Contract: 3-Year Base Contract (opportunity for 5 total years)

• Budget Constraints: $22,900,000 (2010-2011)

• RFP Issued: 12/01/2009

• Politics: IDOC extremely dissatisfied with the incumbent

13

Analysis of Proposals

• Awarded Vendor was:– 2.8% below the IDOC’s budget– 9.4% below average of all proposals– Highest Interview– Highest Work Plan– Second highest RAVA Plan

NO CRITERIA FIRM A FIRM B FIRM C FIRM D FIRM E

1 Total Cost 85,567,680$ 66,786,240$ 83,710,560$ 66,488,400$ 65,840,160$

2 Interview 4.0 6.9 6.9 4.5 2.2

3 Risk & Value Plan 4.8 6.4 6.6 5.4 3.8

4 Work Plan 6.0 5.0 7.4 4.2 4.6

5 PPI Scores (1-10) 9.6 9.8 9.8 9.8 9.2

6 PPI Scores (#) 26 40 28 32 20

Overall Score (1,000): 709 953 951 779 601

Politics• Client was very disappointed in the best-value process since the highest-

prioritized vendor was the incumbent

• Client did NOT want to re-hire incumbent

• Client not happy with the results

• Could not understand why the better vendor did not propose within their budget

15

Moving Into Phase 2• The vendor was unaware of clients concerns with their company

• They assumed that the client was very satisfied with their past performance.

• The client put together a list of their major ‘deal-breakers’ or issues that were causing them significant displeasure in the past. These included: • Incumbents ability to staff the service properly (high turnover issues)• Lack of clear and understandable performance metrics• Incumbents inability to manage the service adequately

16

Vendor Response:• Very traditional (used vague language / didn’t clearly address the issues)

• Individuals preparing the responses did not understand the issues that had occurred in the past with the previous contract.

• Did not make the client feel comfortable they could manage the issues.

• Could not specifically state what would be done differently

• Without understanding the past issues, it is very difficult to explain what they will do differently and how they can improve.

• Vendor transfers risks/decision making back to the client (“what exactly would you like us to do” “how would you like us to handle this”)

17

Client Reaction• The vendor does not understand the new paradigm. Worried about saying

the right things to get the contract, instead of figuring out how to succeed.

• The client identifies that they have issues about trust, support, and retention

• The client identifies that the Regional Manager is a concern. • Staff members will leave if current RM is not replaced• Staff told not to talk to client or they would be fired• RM is creating an adversarial environment

18

Vendor Reaction• The vendors management dismiss these comments as typical statements

that owners make.

• Vendor management does not believe that there have been problems in the past

• If there were problems, why weren’t they informed previously?

19

How Do We Address These Issues?

20

Quick Survey• Perform a third party survey with the client staff

• Identify their overall satisfaction with• The vendor• The vendor’s management• The client

• Primary Goal: Determine if there is a problem or not!

• Within a couple of days, a survey was creased and sent out to all employees.

• No names, setup online for a quick response.

21

Survey Results• Within a week, we received 83 surveys (out of 128) from 9 different facilities

• Results confirmed that there was a significant weakness with staff satisfaction towards corporate support and regional manager

22

NO CRITERIA RESULTS

1 Overall Job Satisfaction (10 max) 6.8

2 Satisfaction of the Client (10 max) 7.3

3 Satisfaction of the Regional Manager (10 max) 3.7

4 Satisfaction with Corporate Support (10 max) 4.5

5 Total Number of Responses 83

Do We Really Have An Issue?• Surveyed 4 other institutions nationwide (similar in size)• Metrics can be used to identify any potential strengths and weaknesses

23

NO CRITERIA UNIT OVERALL CLIENT A

CLIENT B

CLIENT C

CLIENT D

CLIENT E

1 Overall Satisfaction - Job (1-10) 7.0 6.8 7.5 6.9 7.0 6.4

2 Overall Satisfaction - Job Training (1-10) 6.1 6.0 6.8 6.2 4.9 5.7

3 Overall Satisfaction - Pay / Benefits / Comp (1-10) 6.3 6.5 7.1 6.2 5.6 5.6

4 Overall Satisfaction - Site Management Team (1-10) 6.7 7.2 8.2 6.3 6.0 5.6

5 Overall Satisfaction - Regional Manager (1-10) 6.2 3.7 8.1 6.3 5.9 6.1

6 Overall Satisfaction - Corporate Management (1-10) 6.1 4.5 7.6 6.0 5.5 5.5

7 Overall Satisfaction - Client (1-10) 7.4 7.3 7.7 7.3 7.2 7.5

8 Total Number of Responses # 525 83 138 159 58 87

9 Percent of Surveys Returned % 44% 65% 66% 36% 32% 37%

Summary• Vendors don’t track their critical staff performance

• Clients don’t communicate in a ‘dominant’ manner

• Dominant metrics can minimize surprises, communication, effort

• Dominant metrics can make your operation more efficient

24

Dominant MetricsWho Benefits The Most?

25

Student Health Insurance Consortium• Create a statewide Student Health

Insurance Plan (SHIP) consortium – Boise State University (BSU)– Idaho State University (ISU)– Lewis-Clark State College (LCSC)

• 3-Year Contract | $36 Million (for 3-years)

• Measurements of Success1.Reduce internal University program

administration costs2.Maintain or increase Customer

Satisfaction (University & Students)3.Maintain or increase cost-effectiveness of

program to students

Coverage/Plan Characteristics• Consortium goal was to standardize

coverage between all three University's (to maximum extent possible). However, deviations were made as necessary (BSU athletic coverage, ISU RX Coverage, Capitated Fee, etc)

• Consortium goal was to increase plan characteristics (to provide better coverage for students)

NO CRITERIA BSU ISU LCSC CONSORTIUM

1 Deductible Per Academic Year (In-Network) $250 $250 $250 $250

2 Deductible Per Academic Year (Out-Of-Network) $500 $250 $250 $500

3 Maximum Benefit (Standard) $100,000 $50,000 $50,000 $250,000

4 In-Network Coinsurance 80% 80% 80% 80%

5 In-Network Max out of Pocket $4,000 No MOP No MOP $4,000

6 Out-Of-Network Coinsurance 50% 60% 80% 60%

7 Out-Of-Network Max out of Pocket $6,000 No MOP No MOP $6,000

8 RX Drug Coverage (Max) $400 None $500 $500*

Analysis of Proposals

Total Score: 923 916 886 831 840

NO CRITERIA FIRM A FIRM C FIRM D FIRM E FIRM F

1 Cost - Average Student Premium $1,422 $1,327 $1,365 $1,561 $1,596

2 Cost - Average Spouse & Dependent Premium $1,698 $2,668 $2,343 $2,559 $2,762

3 Average Interview Rating 6.4 6.6 5.2 6.3 6.9

4 RAVA Plan Rating 7.4 6.3 7.4 5.6 5.2

5 Work Plan Rating 6.7 7.2 6.3 5.5 5.6

6 PPI - 1-10 Rating 9.9 9.7 9.9 10.0 10.0

7 PPI - Number of projects and clients 10 17 9 10 10

Results

• Best-Value Results:– Student Premium has decreased by

2% (-$26)– Spouse & Dependent Premium has decreased by 19% (-$519)

– In general, Benefits/Coverage have been increased

– The Best-Value Process was able to stabilize premium rates for the first time in 4 years (for both Student and Spouse/Dependents)

School Premiums 2006-2007 2007-2008 2008-2009 2009-2010Average

Increase Per Year ($)

Average Increase Per

Year (%)Student $1,012 $1,182 $1,263 $1,385 $124 11%Spouse & Dependent $1,843 $2,022 $2,104 $2,220 $126 6%

• Previous Program:• Student Premiums increased $124/year (past 4 years)• Spouse & Dependent Premiums increased $126/year

Post Award Metrics• Client is extremely satisfied with award (increased benefits for decreased

costs to students)

• ASU encourages the vendor to partner:• Perform continuous education with vendor key staff• Collect and analyze student surveys (from SHIP) to document performance• Collect and analyze student surveys (from other programs)• Assist vendor in documenting performance (dominant information)• Assist vendor in documenting the project for long-term success

• Vendor states that they are comfortable and does not partner with ASU

Waiver Process• During the first year of the program, the Vendor runs into issues with the

waiver process at one of the institutions.

• Some students did not meet the minimum requirements (but were allowed to waive out), some were incorrectly denied coverage, and some students have not been notified that they do not meet the standards.

• Vendor does not know how to document the issue. ASU proposes the vendor:• Identifies what the issues are and the magnitude of the issue• Identifies why did the issues occur• What is being done to address the issues• What is being done to prevent the issues from reoccurring

• Vendor takes several months putting together a summary of the issues.

• University is frustrated and request additional audits (added cost to vendor)

Student Surveys• Vendor was required to survey students to identify their satisfaction.

• Results of initial student survey showed:• Overall Satisfaction was 4.6 (out of 10)• Number of Responses was 896

NO CRITERIA RESPONSE1 Frequency of use (how often was student health insurance used this semester) 3.1

2 Satisfaction with the level of benefits offered/provided by SHIP 4.4

3 Helpfulness of the marketing materials 4.4

4 Overall claims experience 4.7

5 Would students rather have greater benefits (10) or lower premiums (1) 5.9

6 Overall satisfaction with SHIP? 4.6

7 Percent of students that feel current healthcare needs are being met by SHIP 51%

8 Number of student responses 896

Student Surveys• ASU proposes to collect similar survey information from other

consortiums to determine if the results are about average with the industry (or above/below).

• ASU does a quick analysis of the student surveys and discovers that the dissatisfied students identified issues with claims from the previous vendor. ASU identifies that it would be beneficial to market and educate the students on the differences of this new SHIP program to their previous programs. ASU envisions a simple website that illustrates the differences (prior to completing the survey).

• Vendor does not see value in either proposal.

Post Award Metrics• Vendor is unaware of how to provide dominant performance metrics.

The Vendor submits a vast amount of data that is confusing to the client (client does not know how to interpret all of the data).

• ASU quickly reviews all of the data and prepares a summary:

• Based on the loss ratio, student premiums should not increase• University satisfaction increased by 18%• Student satisfaction decreased by 2%

NO CRITERIA ANTICIPATED ACTUAL IMPACT1 Total Enrollment 7,895 11,213 42%2 Total Premiums Collected $11,350,311 $15,926,766 40%3 Total Claims $9,080,249 $8,484,393 -7%4 Total Loss Ratio 80% 53% -34%

Dominant MetricsWhen Should You Start

Collecting?

35

The State of Hawaii• The State of Hawaii-Public Works

Division (PWD) received numerous complaints on the quality of completed construction projects.– Roofs were poorly constructed.– Contractors were slow in correcting punch

list items.– Response time to warranty work was

slow.– Painting work was so bad that the State

assumed all painting work on Oahu with their own staff (painting awards dropped from 101 projects to 5 in 1998).

• In 1998, PWD implemented the BV process on roofing and painting projects.

• Over a 4 year implementation, PWD procured 193 projects using BV (106 roofing, 33 painting, 15 mechanical, 5 electrical, and 34 general renovation projects)

Kapuaiwa Courthouse RenovationCase Study• Historical renovation

project • 60 year old, 3 story,

20,000 SF facility• One year design

• Deteriorated balusters allowed water infiltration

Kapuaiwa Courthouse RenovationCase Study• Nobody bid the job (design wouldn’t

work)

• Project was re-bid using PIPS– No specification– Requirement was to waterproof the

building for the longest possible period of time

– Budget was $800K

• Award was made using the PIPS process

Complaints• A small handful of vendors were complaining.

• Contacted the media and 19 articles were published stating:• Process was not open (awarding to the same contractors)• Process was unfair• Process was not working• Process was wasting taxpayer money• Projects took longer to complete• Process violated State Regulations

45

Audit Report• In response to the negative articles,

the State performed an audit in November 2002 on the BV process.

• The audit analyzed low-bid projects and PIPS projects.

• Results/Findings1. PIPS resulted in 3% savings of project

costs2. PIPS resulted in greater contractor

accountability3. PIPS resulted in fewer Change Orders4. PIPS has given higher quality

construction

PIPS Roofing Conclusions(Taken from Audit Report, 2002)

1. Total number of awarded PIPS roofs: 96

2. 100% would rather use the PIPS process over low-bid process (55 DOE users)

3. 100% would use a PIPS contractor again (55 DOE users)

4. PIPS average performance rating 9.6 (10 max)

5. Projects were 6% under budget (adjusting for insulation)

6. Projects finished approximately 35% faster (than LB)

7. Performance rating of PIPS vs. LB: 8.1 vs. 5.6

8. 98% were completed on time9. Contractors were almost twice as

productive ($4.5K/day vs. $2.5K/day)10. In last 4 years, there has been no

roof leaks

Outstanding Results….

48

…But PWD is Directed to go back to Low-Bid

Dominant Metrics• By the time the State had compiled and presented the information, the

decision was already made to go back to low-bid.

• Although the performance metrics were significantly higher, the Dominant Metrics were not available fast enough.

• Decision was based on politics, not performance.

49

Can ‘Domina

nt Metrics’ Assist You?

50

Objectives - RECAP• Understand what is “Dominant Information”

• Understand who should use or who has the most to gain by using “Dominant Information”

• Identify an example where Dominant Information could have assisted you

51