comparing state judicial performance - iaiabc.org · comparing state judicial performance 2016 csa...

KDOL serves Kansas workers and businesses by providing fair and efficient administration of state labor laws.

Comparing State Judicial Performance

2016 CSA Spring Seminar David Sprick, KDOL

Greg Krohm, WorkComp Strategies


Outline

• KDOL DigiComp project • Overview of performance

measurement • Principles of good metrics


Outline • Nuts and bolts of good work comp

metrics • Kansas dispute model & performance

metrics • Current projects to expand work

comp metrics


Kansas’ DigiComp Project • Two stage comprehensive digitization

project • DigiComp is planning phase or phase 1

just finishing!


Kansas’ Digicomp Project • Phase 2 we are building the actual

information system to achieve these organizational goals: – Improve customer service – Reduce administrative costs – Increase operational efficiency &

effectiveness – Improve data quality and integrity


Kansas’ DigiComp Project • Think and plan how to use of technology

to automate functions- yes, absolutely! • Digitize information for easier transport

and retrieval- yes, absolutely!


Kansas’ DigiComp Project • But also use technology to complement

or augment analytic & non-routine tasks!

• Will be awash in digital data


Kansas’ DigiComp Project • Example of an augmentation for

analysts and managers • Metrics for performance measurement!


Kansas’ DigiComp Project • Performance Measurement is not some

abstract (i.e., “useless”) academic exercise


Metrics • A system or standard of measurement • Core component of performance

measurement of an organization – “Accumulated end results of all the

organization’s work processes & activities” - Robbins and Coulter (2005)


Performance Measures • Performance measures support a wide

variety of management functions – Robert Behn’s (2003) Eight purposes – evaluate, control, budget, motivate, promote, celebrate, and learn & improve


Performance Measures • Performance measures support another

function in this context, to understand the law – “The prophecies of what the courts will do in fact, and nothing more pretentious, are what I mean by the law.”

- Oliver Wendell Holmes, Jr. (1987) -WCD measures


Performance Measures • Less rudimentary, more sophisticated • Outputs

– Activity counts • vs. Outcomes

– How efficiently a given service rendered


Performance Measures • At what quality service was delivered – What effect the service is having on

service recipients or community as a whole


Performance Measures • “Proactive understanding of a state

Workers’ Compensation system.” (Greg Krohm, 2015)

• Disputes source of friction costs in system • Can add unnecessary complexity

• “You cannot measure some of the really important things.” (Greg Krohm 2015)


Principles of Good Metrics

Principles generally applicable to any organization

Followed by some WC specific ideas

Greg Krohm


Some Principles of Metrics for Work Comp

• Must be relevant • Must be accepted as valid and accurate • Must address either

– functional efficiency or workloads – outcomes of activities in meeting objectives


Common Mistakes in Metrics • Not linking metric with real mission or

strategic objective • Instead too many metrics just measure

activities or subsystems of the agency

When a leader understands the strategy and turns the strategy into a reality through linking mission to business plan objectives via communication, performance metrics and organizational alignment, then he or she is on the road to achieve predictable success. --SixSigma


Common Mistakes: Not Relevant to Strategic Objectives

• Metrics should be logically linked to overall objectives of the agency

• There can be a hierarchy of metrics building from functional units of an agency to overall system


If you don’t have a clear goal, metrics are a waste of time


Common Mistakes: Lack of Causal Links

• Causal relation of metric to goal not thought out – Why would doing X change Y? – How do circumstances beyond our control

affect Y? • Should be tested

– Survey/focus groups – Detailed measurements of process flows – Measuring external changes affecting system


Common Mistakes: Bloat • Metrics are added like ornaments on a

tree • Becomes distracting and/or misleading

“When all you want is data because having it makes you feel good, then metrics are more like a drug than a reliable decision driver.” -- IndustryWeek.com


Common Mistakes – Not Seeing Patterns/ Trends

• Change over time often critical to understanding

• Related to this is charting performance against targets over time


“Nuts and Bolts” of Workers’ Comp Metrics


Overriding Question: What is a WC Agency’s Objective?

• Keep insurance costs lower than neighboring states?

• Balance fair benefits with reasonable cost to employers

• Administer the law fairly and efficiently • Make the Director’s life easy


Example of Mission Statement “The mission of the [WC agency] is to effectively and efficiently serve the public by responding fairly and timely to the needs of the injured worker… The goals of the reform were to provide for injured workers, while reducing costs in the overall system thereby encouraging job growth on [state].” How would you go about measuring success?


Rogues Gallery • Following are some examples of the many

meaningless/useless measurements reported by state agencies

• Their faults: – Ambiguous, ill defined variables – Lack of causal connection to system

performance – No agency or system changes indicated – Misplaced attention


Ambiguous Data, No Causal Relations, No System Actions

Suggested

Claim Received Year

SOC 2000 Major Group DescriptionNumber of

ClaimsIncurred Costs

TotalIncurred Costs

AverageNot Coded 442 1,346,354$ 3,046$ Accountants and Auditors 40 265,537$ 6,638$ Actors 32 47,462$ 1,483$ Administrative Services Ma 61 690,262$ 11,316$ Advertising and Promotions 12 70,538$ 5,878$ Advertising Sales Agents 6 62,296$ 10,383$ Aerospace Engineering and 1 1,363$ 1,363$ Aerospace Engineers 7 12,016$ 1,717$ Agents and Business Manage 4 24,939$ 6,235$ Agricultural and Food Scie 23 127,745$ 5,554$ Agricultural Equipment Ope 195 1,161,994$ 5,959$ Agricultural Inspectors 29 148,717$ 5,128$

ALLOWED STATE FUND CLAIMS ONLYCLAIM COUNTS AND COSTS BY STANDARD OCCUPATIONAL CODE 2000 MAJOR GROUP DESCRIPTION

Fiscal Year 2015


Technically Sound But Grossly Misused


Absolutely Meaningless But Often Replicated

REGION 2014 TOTAL 2000 - 2014

LOS ANGELES 26.6% 25.9%

BAY AREA 19.8% 21.3%

INLAND EMPIRE 18.7% 17.6%

CENTRAL VALLEY 10.9% 10.8%

SAN DIEGO 8.2% 7.7%

CENTRAL COAST 6.5% 6.1%

SACRAMENTO VALLEY 5.0% 5.5%

EASTERN SIERRA FOOTHILLS 1.8% 1.9%

NORTH STATE-SHASTA 1.4% 1.9%

NORTH SACRAMENTO VALLEY 1.1% 1.3%

Percent of FROIs Reported by Region of Accident


Also Meaningless

Notice n=1 or 2=data disclosure based on small cell size


Analyzing a Common Metric: Dispute Data

• Suppose you had a flawless data capture system for disputes, i.e., tightly defined variables, accurately recorded

• How might you use the data? – Identify patterns and trends – Look for causes of change

• Determine if there is an actionable response to a trend


Using Dispute Data • Suppose you see disputes that were very easily

resolved by clearing up misunderstandings – Better outreach to injured workers with questions

• Suppose the number of causation questions spikes – Who is denying claims and why?

• Suppose there are increased disputes on RTW – Standardize “duty restriction” form for MD and

educate claimant


Rate This Metric • Counting investigation letters sent to

suspected uninsured employers – Very weak link to compliance and universal

coverage – Can be gamed by staff


Rate This Metric • Counting inquiries to agency from

claimants – Unclear whether this is good or bad – No causal linkage to system performance – Definitional problems – Consistent data capture a problem – Can be gamed by staff


Rate This Metric • Average time to claim closure

– Very misleading unless you control for accident date

– Even if controlled, a clear bias toward shorter closures for more recent accident years

– How does this relate to system performance? – What’s driving changes in closures, e.g.

• More settlements • Faster MMI • Less complex/serious injuries


Example of a Powerful & Effective Metric

Washington Ability to Work Assessments


Early AWA Causally Linked to Agency Objective


Showcase Use of Metrics: Speeding AWA


The Kansas Dispute Model & Performance Metrics

David Sprick


Kansas WC Disputes • Confusion over names or “things don’t

mean what they seem” • Example: Pleadings

– A request that an ALJ be assigned is called an “Application for Hearing”

– A reply that Judge Smith has been assigned is called a “Notice of Hearing”


Kansas WC Disputes • Confusion over names • Example: Disputes

– An “Application for Hearing” is not a dispute until a hearing is scheduled

– A “denial” of payment is not a “dispute” until an Application for Hearing is filed and a hearing is scheduled


Kansas WC Disputes • Confusion over names • Example: Docket

– A sequential number created after filing an Application for Hearing

– A listing of a day’s hearings at a court location – A summary of dispute-related activities

occurring after filing an Application for Hearing


The Kansas Dispute Data Model

• Standard definitions • Model dispute related events

– 5 levels of hearing types – Hearing pre- and post-conditions (e.g., terminal dates,

settlement conferences) or status conditions • Model decision types • Model issues • Assumes EDI Claims reporting (e.g., denial

reason codes, injury type)


ALJ PRELIM request date set date ready date hearing

date decision

date

ALJ FINAL request date set date ready date hearing

date decision

date

APL BD P request date set date ready date hearing

date decision

date

APL BD F request date set date ready date hearing

date notify date

CRT APL request date set date ready date hearing

date decision

date

Dispute Events


Model Decision Types • Work causes injury (Was it covered?) • Injury causes impairment (Did a doctor make a

diagnosis?) • Impairment causes disability (Were work

restrictions given?) • Disability causes earnings loss (What is the

resulting earning capacity?) • Injury requires treatment (Medical) • Impairment warrants award (PPD)


The Kansas Dispute Data Model


Model Issue Types • 25 types • Examples: prevailing causal factor,

psychological, misconduct, coming and going

• Relies on “primary” or “dominant” issue per decision type

• Used for analytics, not case management!!


Tested in Kansas • Tested in approximately 50 full hearing

cases – Random sample of the archive

• Data gathered from ALJ decisions, database


Model Dispute Measures • Data on modeled definitions, events, decision

types, and issues support standard performance measures

• 5 measure types – Counts – Program effectiveness – Efficiency – Fairness – Management

• 6th type built on customer survey


Examples • Efficiency

– How long does it take to get a hearing set, by event type?

• By decision type, injury type?

• Fairness – Distribution of efficiency metrics (item 3) by filing party

• Effectiveness – What is total number of hearings, by event type, as a

percentage of lost time claims?


Simple Dashboard Example Law is complied

with

Payments on time and accurate

Disputes/LT claim/payer

Non substantive disputes vetted

Disputes/LT

Settlements before hearing

Hearings are thorough, objective

and legally sound

Appeals/Hearings

ALJ reversed/Appeals


Current Projects to Expand Workers’ Comp Metrics

David Sprick


IAIABC • Claims Measure Survey

– Research and Standards Committee – Current participants (20): including CSA

members SD, WI, IN, ND, NB, MN • 5 measures

– # FROIs, # work fatalities, #compensable LT, average claim duration for LT, average time to make first payment (LT), # LT claims denied


Dispute Comparisons • This is what we mean by “Comparing

State Judicial Performance” • Securing grant funding to expand test of

the KS dispute model to 5 states, Canada jurisdictions – Organized with IAIABC – Linked to EDI standards – Helps with system requirements for new

systems


Cross-Jurisdictional Comparisons

• Test involves terms/event/decision/issue modeling; data review

• Test state benefits – Workflow diagrams, process descriptions – Recommendations on updating data capture,

streamlining forms – Standardized performance metrics – Comparison with other test states


Questions


[email protected]

[email protected]

mailto:[email protected]




References • Behn, Robert. 2003. “Why Measure Performance?

Different Purposes Require Different Measures.” Public Administration Review 63, no. 5 (September/October): 586-606.

• Holmes, Jr., Oliver Wendell. 1997. “The Path of the Law.” Harvard Law Review no. 10, (March) 991-1009.

• Krohm, Greg. 2016. Greg said it to me, again. • Robbins, Stephen and Mary Coulter. 2005. Management

8th Edition. Upper Saddle River, NJ: Pearson Prentice Hall.


Further Reading 1) National Assoc of State Budget Officers, on performance

measures: https://www.nasbo.org/publications-data/reports/performance-management-framework-state-and-local-government-measurement-an

2) State of IN Office of Management and Budget, listing of state performance measures: http://www.in.gov/omb/2342.htm

3) State Finance Officers, link to Nat. Performance Measure Advisory Commission publication (for purchase): http://www.gfoa.org/performance-management-framework-state-and-local-governments-measurement-and-reporting-management

4) MRSC Local Government Success, good resource on state and local performance indicators in use in Washington State, and elsewhere: http://mrsc.org/Home/Explore-Topics/Finance/Budgets/Performance-Measurement.aspx

https://www.nasbo.org/publications-data/reports/performance-management-framework-state-and-local-government-measurement-an



http://www.in.gov/omb/2342.htm

http://www.gfoa.org/performance-management-framework-state-and-local-governments-measurement-and-reporting-management

http://www.gfoa.org/performance-management-framework-state-and-local-governments-measurement-and-reporting-management

http://mrsc.org/Home/Explore-Topics/Finance/Budgets/Performance-Measurement.aspx

http://mrsc.org/Home/Explore-Topics/Finance/Budgets/Performance-Measurement.aspx

comparing state judicial performance - iaiabc.org · comparing state judicial performance 2016 csa...

Documents