1 chapter 6. making success measurable through quality control
TRANSCRIPT
1
Chapter 6.Making Success Measurable
Through Quality Control
2
Chapter 6. Section 1. Introduction to Quality Control
3
Introduction• How do you know the quality and accuracy
of your rent calculations? – Wait for HUD to tell you?– Close your eyes and hope for the best?– Assume that good people lead to good
outcomes?– Wait for your independent audit results?
• Or …..
4
Introduction
• Are you interested in a better way that could:– Enable you to know when to take action on
performance issues– Help document the improvements in error
reduction you are making– Enhance the quality of the services you
provide?
5
Introduction
• A good quality control program can be that better way that helps you:– Prevent errors– Document compliance with program
requirements– Demonstrate performance improvement over
time
6
Introduction
• An effective quality control program:
– Facilitates conversation about expectations and performance for individuals and the organization as a whole.
– Enables program managers to be proactive, rather than reactive
7
Introduction
In this session, we will talk about:
• Three types of quality control
• Illustrate the thought process needed to develop a QC program
• Provide sample QC tools
• Provide guidance on communicating with your staff about quality control issues.
8
Introduction
• We will focus on HUD’s expectations related to income and rent determination
• BUT, the QC concepts we discuss apply equally well to other parts of a PHA operation.
9
Why Quality Control?
• Quality control can help to confirm what you already know. It can:– Provide documentation that supports
recommendations you want to make– Generate statistics that document staff
performance– Provide valuable information on a systematic
basis to share within and outside your organization.
10
Why Quality Control?
• QC can be even more helpful if the results are a surprise. Then the results may highlight:– Problems of which you were not aware– Good news about improved performance or
excess capacity.
• Were there aspects of the RIM review for your agency that were a surprise?
11
Chapter 6. Section 2.Types of Quality Control
12
Types of Quality Control
Preventive
Continuous
Periodic
Types of Quality Control
13
Preventive QC
• Preventive QC is checking on the quality before the transaction is completed. Use Preventive QC when:– The consequences of errors are high, or– The error is not recoverable.
• Examples:– Supervisors sign termination letters even though
specialists sign other correspondence– Occupancy supervisor reviews 100% of rent
calculations for new staff (for some period of time).
14
Preventive QC
• More examples:– Intake specialist presents a “dry run” of the briefing to
staff before doing the real thing in front of participants.
• What examples of preventive QC occur at your PHA?
• Try to identify at least one additional area (related to income and rent) where preventive QC might be helpful.
15
Limits of Preventive QC
• Preventive QC can be an important tool but what are its limitations?
– PHA’s don’t have time or staffing to double and triple check every transaction –you need to prioritize.
– Use common sense: Not everything requires this level of effort.
16
Periodic Quality Control
• In contrast, periodic quality control happens after the fact. Work has already been done and reported.
• It’s purpose is to take a “picture-in-time” assessment of PHA operations that will help improve the program and prevent future errors.
• Sound familiar? Yes – HUD’s RIM reviews and PHAS and SEMAP audits are kinds of periodic QC.
17
Periodic Quality Control
• During a periodic review HUD systematically looks at:– Policies, procedures, and practices and– Individual files.
• But you don’t have to wait for HUD. You can also do your own. You could:– Look at the same things HUD will be
checking, or– Focus your periodic review differently to learn
even more.
18
Limits of Periodic QC
• Periodic QC can provide an accurate point-in-time assessment, but:– It doesn’t help managers see trends– Results come infrequently so it doesn’t help
managers take corrective action during the year.
– The level of effort required may seem overwhelming (but HUD’s RIM Review guide provides a good model for the review)
19
Continuous Quality Control
• Continuous QC is used to help with day-to-day management.
• It involves analyzing a few (but very important) factors on a regular, continuing basis. For example:– Tracking on-time reexaminations each month– Looking at accuracy rates by individual or
transaction
20
Continuous QC
• Continuous QC helps managers:– Focus more quickly on vulnerable areas.– Look at trends, rather than point-in-time
information.
• HUD does some continuous QC of 50058 submissions and uses the data to:– Send error reports to PHAs, and– Assess in general terms how PHAs are doing.
• But only the PHA can implement an effective continuous QC program.
21
Limits of Continuous QC
• Continuous QC is a powerful tool.• But, remember, the scope of the
Continuous QC is limited.• This means:
– Continuous QC provides only a limited picture (hopefully of the key areas)
– Continuous QC should be supplemented by Periodic Reviews that take a broader view.
22
Continuous QC Example
Page 6-11 and 6-12
• Look at the “Error Analysis” chart– QC results are shown by quarter & individual– Caseload = annual caseload– Errors = things HUD would call findings– Comments = errors that would not be findings– Types of errors are defined on page 6-12
23
Continuous QC Example
• What issues does this report raise for you?
• What are your concerns about:– Productivity?– Types of errors?– Error Rates?
• What are the implications of the information you see for this PHA?
24
Chapter 6. Section 3.Designing a QC Program
25
Designing a QC Program
• Now that we have discussed each type, it’s time to think about how to design your program.
• Before you begin it is very important to think through: – What you want your QC program to look at. – What you will do with the information.
26
Designing a QC ProgramWhere to Start
• To focus your program ask three questions:– What are the outcomes you want to achieve?– What would success look like for each
outcome?– What is the best way to confirm (track,
measure) performance in each area? • Then decide:
– Which measures are useful and feasible.– How you are going to record the information.– Who needs to know the results.
27
A Simple Example- Sleep
• What is my desired outcome?– A good night’s sleep
• What would success look like?– Eight hours of uninterrupted sleep
• What is the best way to measure?– An independent 3rd party could observe my
sleep.– I could track myself the time I go to bed, time I
wake up, and number of interruptions.
28
Simple Example (Cont)
• Decision:– Having someone watch me sleep could be
expensive and disruptive– Instead, I will track sleep patterns myself– I’ll write this down in a notebook– No one needs to see it but me (unless I think
maybe my doctor should know)
29
A More Serious Example – Utility Allowances
• What is my desired outcome?– Current and correct utility allowances
will always be used in rent calculations.
• What would success look like?– Utility allowances would be used
correctly in 100% of the files.
• Are there other ways you would describe success re: utility allowances?
30
A More Serious Example – Utility Allowances
• What are possible ways to measure the outcome?– 100% file review in advance (preventive)– 100% file review after the fact– Sample file reviews after the fact
• What else could we do to see if the correct utility allowances are being used consistently?
31
A More Serious Example
• Decision:– 100% review would be too time consuming
(before or after)– Therefore, review a sample of files (after the
fact) to see how often staff have used the correct utility allowance
– Generate a report that summarizes errors by employee, by program type, by transaction type.
32
Thinking About Outcomes
• Choosing the outcomes you want to use is an important part of the design process.
• Outcomes can be expressed in very general or very specific terms.
• Some “big picture” outcomes might include:– Consistency in data collection, documentation – Complete and accurate verifications– Accuracy in rent calculations– Timeliness – Complete, accurate reporting
33
Thinking About Outcomes
• Even broader outcomes might include:– No findings on the next RIM review– Informed decision making by applicants and
residents– Fair and equitable treatment– Equitable workload– Being a PHAS high performer– Positive perceptions – we’re a good agency.
34
Thinking About Outcomes
• Caution: Generally more specific outcomes are better understood and easier to measure. For example:– Reduce departmental error rate to less than
one percent– Complete 100% of annual reexaminations
accurately and on-time
• The more specific the outcome, the easier it is to track performance.
35
Designing a Program to Support Outcomes
• Page 6-20 through 6-21 show examples of turning desired outcomes into a QC plan.
• Follow along on the chart
• We will walk through Row 1 and Row 2 together
36
The End Result
• The end result of this thinking will be a Quality Control Plan. (see page 6-25).
• The planning itself is more important that the format for the plan.– Think through the process before you begin
data collection– Collect information about the items that have
the most effect on error reduction– Be realistic about what data you can collect.
37
What’s Next?
• The balance of this chapter discusses 5 important “how to” aspects of QC:– Conducting file reviews– Conducting a “general” review of operations– Using reports to track quality– Addressing staff skills and skill gaps– Communicating about error reduction and
quality control
38
Chapter 6. Section 4.How to Aspects of Quality
Control
39
Conducting File Reviews• Files reviews are used:
– To determine accuracy and completeness– To identify potential policy and management
issues
• “Files” for this purpose include the 50058 record as well as paper files.
• Caution: A file review can tell you what has happened, but not necessarily why.
40
Conducting Tenant File Reviews
• Sample size and characteristics of files selected depend upon the purpose.– Generally PHAs use a “directed” sample. – Directed means selecting the files that contain
the characteristics you want to monitor.– Page 6-28 shows characteristics that may be
important to consider for the sample.
• Not every characteristic must be represented each time you review files.
41
Conducting File Reviews
• For Continuous QC – Purpose is timely feedback on few issues. – File Selection Priorities would be:
• Each specialist• Basic transaction types• Files completed over time
42
Conducting File Reviews
• For Periodic QC– Purpose is a more comprehensive review.– File Section Priorities:
• Each specialist (but perhaps more files)• More transactions types• Larger cross section of income sources and
household types
43
Conducting File Reviews
• Tools for Selecting Files:– PIC reports can generate lists for some
transactions. (See Appendix A, A-25)– PHA vendor software may be able to generate
reports that list files in categories.
• How many files do I need to review?– QC sample should be sufficient to meet
SEMAP requirements
44
Example File Sampling
• Useful guideline for determining number of files to review
Universe Minimum Sample Size
50 or less 5
51-6005 plus 1 for each 50 (or part of 50) over 50
601-200016 plus 1 for each 100 (or part of 100) over 600
Over 200030 plus 1 for each 200 (or part of 200) over 2000
45
Conducting File Reviews
• Logistics – Use a file checklist to document your review.
(RIM checklists are a good start.)
– Complete 50058 calculations without looking at the completed calculations in the files.
– Look for consistency between the paper and electronic files.
46
What The Reviews May Show
• Key RIM Review programmatic findings:– Earned income/pension/public assistance
calculations– Incomplete/outdated PHA policies– Data not being transmitted to PIC– Lack of verification of income and deductions– File data doesn’t match data used to calculate
rent– Improper calculation of utility allowances
47
What the Reviews May Show
• RIM reviews also found errors related to automation.
• In general, these errors include:– Programming problems (software vendor) – Errors related to the way PHAs use
automated systems such as:• Not attending to error reports and• Inadequate training for staff on the system
• See page 6-32 for details
48
Conducting the “General” Portion of a Periodic Review
• Periodic Reviews enable PHAs to take the same broad look that HUD takes during a RIM Review.
• The agenda:– Look at policy and procedural documents– Conduct an expanded file review
• HUD’s RIM checklists are good models.
B. ACOP G. Rent
E. Applications I. Reexaminations
F. Income J. HUD 50058
49
Conducting the “General” Portion of the Review
• You may need to add to HUD’s checklists:– Questions covering PHA-specific policies or
performance standards– Protocols for checking how PHA-specific
software handles income, rent and subsidy determinations.
50
Using Reports to Track Quality
• Perspective is important.– Identifying individual errors is important but
it’s only one aspect of QC.– Looking at trends (performance over time)
provides important insights that lead to error reduction.
– Looking at trends can also help set performance standards.
51
Using QC Reports to Track Quality
• What should be tracked regularly?– How often things happen– How long things take– Whether improvement is being made– How individuals are doing.
• Caution: Tracking reports can raise flags but may not answer the question: “Why?”
52
Analyzing Data
• Pages 6-36 – 6-38.• Study the three reports
– On-time reexaminations– Transactions by type– Error rate trends
• What flags does each report raise?• What additional information would you like
to have?
53
Addressing Staff Capacity Issues
• Error reduction will not happen without staff capacity.
• No one is born knowing the rules of assisted housing! Most PHAs must grow their own “experts.”
• Staff development is a joint responsibility of the organization and the individual.
• Capacity building begins with a mutual understanding of both the job requirements and current performance.
54
Addressing Staff Capacity
• A good QC program can support staff development efforts if it includes an analysis of: – The knowledge and abilities that each job
requires – The actual knowledge and abilities of the staff.
• And, if the PHA uses this information to create:– A strategy for addressing any gaps in capacity– A plan for developing staff and monitoring staff
progress in meeting performance goals.
55
Addressing Staff Capacity
• Skills Set is the term used to describe what people need to know, understand, and be able to do.
• See pages 6-40 & 6-41 for a list of the skill sets required by staff who perform income and rent determinations.
• Review the list briefly. Do you think these items represent the skills needed by staff at your PHA?
56
Addressing Staff Capacity
• How to use your analysis of skills sets:– Your analysis can improve your recruiting
efforts and hiring decisions– Be aware of skill gaps when considering
changes in staff roles or position – Use your understanding of required skills vs.
available staff skills to help diagnose performance problems
– Use your understanding of skill sets to select the right employees and to develop employee training plans.
57
Addressing Staff Capacity
• Consider using Individual Development Plans to:– Help set expectations– Demonstrate a commitment to staff
development• See page 6-49 for a sample IDP. Note how it
sets expectations and specifies development activities that will help the employee succeed.– Goal #2: Employee will become proficient….– Goal #2, Activity 3. Attend a 3-day occupancy
training program.
58
Communicating About Error Reduction and QC
• The Challenge:– Reducing errors requires staff
commitment.– A QC program must be seen as a
positive activity to staff as well as management.
• To succeed, staff need to know:– What good performance is– How it will be measured– How they are doing as performers.
59
Communicating About Error Reduction and QC
• Performance expectations help staff understand what is required. They generally relate to:– Quality and Quantity of Work– Timeliness– Customer Service
• Quality control program helps managers to:– Set expectations – Share information about actual performance in
comparison to goals– Recognize and reward good and improving
performance
60
Communicating About Error Reduction and QC
• Sharing information about performance helps staff keep focused on:– What is important– What needs to be done– What they have accomplished
• Most organizations have too many reports and not enough information.
• Finding effective ways to share this information is challenging.
61
Communicating About Error Reduction and QC
• Consider using “Scorecards” to help staff maintain perspective
• Keep them simple (see page 6-45) and focus on:– Goals and Progress– Information that makes paperwork real
(household demographics, neighborhoods)– Quality and quantity of work– Progress over time– Accomplishments – things to brag about
62
Communicating About Error Reduction and QC
• Scorecards for individuals also can help reinforce expectations and give feedback to staff.
63
QC Recap
• A good quality control system should:– Be comprehensive but not burdensome– Cover key performance areas– Fairly and accurately measure performance– Be perceived as fair by staff and managers
• Few organizations are perfect. QC will find problems.
• But a good QC program will also show the pathway to improvements.
64
QC Recap
• The next session talks about how QC data can help you to develop strategies for:– Responding to RIM findings– Solving performance problems, and– Reducing errors.