wap monitoring & quality control

Weatherization Assistance Program eere.energy.gov

WAP Monitoring & Quality Control

Thursday, September 23, 2010


Agenda

• DOE Monitoring Approach

• IG/GAO Trends

• DOE Field Implementation

• Quality Assurance

• Grantee Monitoring


DOE Monitoring Approach

• Monthly

– Timeliness

– Accuracy

– Production

• Planned vs. Actual

• Previous Month vs. Current Month

– Expenditures

• Planned vs. Actual

• Previous Month vs. Current Month



• Quarterly

– Completeness

• OMB 1512

• DOE 504.3 Program Report

• SF-425 Federal Financial Report

– Timeliness

– Accuracy of Data

– Performance Targets

• Production

• Expenditures

• Monitoring

• Training



• On-Site Visits (Minimum frequency/Year)

– Grants over $100 Million – Quarterly

– Grants between $100 Million - $40 Million – Three visits

– Grants under $40 Million – (Semi-Annual)

• On-Site Visit Content

– Program Administrative Management

– Program Technical Management

– Program Financial Management


IG/GAO Trends

• Quality of Oversight of Programs, Contractors, and Program Staff

• Substandard installation of energy saving materials

• Davis-Bacon non-compliance

• Low Spending Rates/Grant recipients fell short of goals to weatherize homes


IG/GAO Trends

• Inadequate systems capable of tracking overall performance by individual contractors and local agencies

• Insufficient Internal Controls to Prevent Fraud, Waste, and/or Abuse of Funds:

– Poor workmanship

– Incomplete client files

– Lack of qualified staff

– Incomplete vehicle and equipment inventories

– Onsite monitoring and/or inspection


A Snapshot of IG Conclusions and Observations

The State X CAA had not implemented financial and reporting controls needed to ensure Weatherization Program funds are spent effectively and efficiently.

Specifically, State X CAA had not:

• Performed on-site financial monitoring of any of its sub-grantees under the Recovery Act;

• Reviewed documentation supporting sub-grantee requests for reimbursements to verify the accuracy of amounts charged;

• Periodically reconciled amounts paid to sub-grantees to the actual cost to weatherize units;

• Maintained vehicle and equipment inventories as required by Federal regulations and state and Federal program directives; and,

• Accurately reported Weatherization Program results to the Department.


DOE Field Implementation

• Grantee Level State Plan Implementation

Grantee Policy and Procedures Manual and Field Standards

Reporting Systems (Financial, Administrative, and Programmatic)

Training and Technical Assistance

• Subgrantee Level Program and Financial Management

Production Planning and Quality Control Process

Procurement

File Management


DOE Field Implementation

• Dwellings

Training Requirements

Application Process

Initial Inspection/Audit

Health and Safety

Technical Standards and Installation

Quality Control

Job Costs and Funding Sources


Federal – QA Reviews

• All agencies to be visited by contractors

• Contractors may NOT be “weatherizers”

• Home Inspectors that are BPI credentialed; RESNET/HERS; or Weatherization Trained

• Captures information for PO to determine where attention needs to be focused

• Strategy

• “SMS version” -- Hit the state with 3-6 staff – disrupt operations for a week in lots of agencies – then get out

• “QA Contractor” – PO schedules reviews, depending on need, could be continuously (xx% a week)

• Results

• Aggregated results/discussion will lead to focused attention by the Project Officer


The Process is a Cycle


Quality Assurance Reviews

• The unit review form has 3 major sections:

1.Agency/unit identification

2.File review

3.On-site work assessment

1.Attic

2.Sidewalls/kneewalls

3.HVAC

4.Subspace

5.Windows/doors

6.Other measures

7.Additional attention needed?


Unit identification and file review

• The unit identification and desk file review sections should be completed at the State or sub-grantee office prior to the site visit.

– We usually spend half a day reviewing 8-12 files then a day and a half in the field

• 45 minutes for the first one; 35 minutes for the next; 15-20 minutes for the rest… (we are trainable!)

• 45 minutes (on average) going through a unit


Example:

FILE REVIEW

Unit Assessed Using: (select one) Energy Audit Priority List None Evidenced

Energy Audit or Priority List is included and aligns with expected audit/list (approved by DOE)

Energy Audit or Priority List is included, but does NOT align with approved audit/list

No clear indicator of use of site specific audit or approved priority list in assessment

Work Order/Building Weatherization Report Yes No

DOE Investment: $_________________________ Total Job Investment: $_____________________

Yes, work order is included in file and appears to follow the audit/list

Yes, work order is included but does NOT follow approved audit/list

No, work order is not in file

Quality Assurance


Trends in Quality within State

• With reviews of 4-6 different agencies in a state

– We look for isolated issues for specific agencies

• Agency may be mis-entering information in the audit tool

(e.g., door = infiltration measure reduction)

• Missed opportunities for savings -- sometimes the “low hanging fruit” (e.g., no refrigerator replacements allowed)

– We look at what may be an area for the state to affect change (peer-to-peer, training needs)


Electric Base Load Measures


Regional Trends and Opportunities

• With reviews of 4-6 different states in a region

– We look for isolated issues for specific states

• Statewide concerns (e.g., windows sans justification)

– We look for trends that might be housing stock or climate based

• Air sealing challenges based on a specific housing stock

• How solar screens are being implemented in the program in some of the hot climates

– We think about how best to pass information from one state to another


Window Sunscreens


National Trends and Opportunities

• After seeing about 20 states, you start to think about national opportunities/conversations

– Audits and Priority Lists

• Clarifying when states with a priority list need to run a site specific audit

• Multi-family misunderstandings – what states need to do if they do not have an approved multi protocol in place

– Health and Safety

• What does that mean, how does it get implemented


Passing along “Best Practices”

• We see opportunities to share with others

– Colorado – Passport for crew advancement

– Iowa – Agency photo files were superb

– Idaho – Best LSW documentation we’ve seen

– North Carolina – Agency has amazing file documentation, should be replicated

– Vermont – Great use of two-part foam

– Mississippi, Alabama, Illinois, Georgia, New (all)…


And, building up our “can you believe” library

Cardboard box used for fan enclosure


Can you figure this one out??


Personal Favorite

Agency assured us, this is not their typical method


What do we know

• We know:

– 30,000 houses are going to be reviewed

• To date 2000 have been done

– Every local agency is going to be visited

• Maybe multiple times, depending on production levels

– Local agencies are going to be asked to go with the contractor on reviews (Grantee is welcome)

• At their discretion, agency can send the contractor out without local agency staff; that is the choice of the agency


Grantee/Subgrantee Scheduling is Not Random

Factors• Levels of Grantee and mandatory

visits/year requirement and % requirements

• Factors in last (most recent) review dates

• Current state of Actual production, units weatherized vs. Planned

• Number of units to be inspected per Subgrantee based on the “inherited” schedule from the Grantee – even considers some Subgrantees with disproportionately higher number of units than others in state

• Factors in the net number of reviews done already

• Need to balance to @ 2000-2200 units/month over several years

Scheduling Approach• Schedule 3 months in advance• Advise PO’s and Contractor

– PO’s can confirm schedules with Grantees/sub-grantees• Problems on timing or availability

can be accommodated with schedule adjustment (which, when)

• Vacations can be worked around– Contractor can identify resources

as required• Finalize schedule @2 weeks prior to

start of next month


What we DON’T Know

• We don’t know:

– Who – contract hasn’t been awarded

– When – when is predicated on determining who…

– What – (I just threw this in since I was on a roll)

– Where – DOE hasn’t determined priority points of what states will go first

• Will it be states that haven’t been visited by SMS?

• Will it be large states with high production?

– How – training of contractors STILL must occur, even after award

• Let’s make sure they know what they are looking at


We’re all in this together

• Weatherization Plus goal

– The quality of work would be the same regardless of locale

• Evaluation… Quality Assurance Reviews… PO increased monitoring efforts…

– These equal lots of opportunities to get it right and affect change… together!


• Questions/Comments?

wap monitoring & quality control

Documents

financial monitoring

weatherization program

weatherization program

federal program directives

program reportsf

actualprevious month

federal regulations

observationsthe state