prepared by government consulting services for audit and evaluation branch, environment canada

Clean Air Agenda Horizontal Evaluation Roll-up: Synopsis of Evaluation Results Available Up to September 2010

Prepared by Government Consulting Services

for Audit and Evaluation Branch, Environment Canada

November 8, 2010

Final

Acronyms

ADAPT Adaptation Theme

CARA Clean Air Regulatory Agenda

CE Clean Energy Theme

CT Clean Transportation Theme

DFAIT Department of Foreign Affairs and International Trade

DG Director General

DGTLCC Director General Theme Leads Coordinating Committee

DM Deputy Minister

EC Environment Canada

GHG Greenhouse gas

HC Health Canada

HRSDC Human Resources and Skills Development Canada

IAQ Indoor Air Quality Theme

IC Industry Canada

ICC Interdepartmental Coordinating Committee

INAC Indian and Northern Affairs Canada

INTL International Actions Theme

M&A Management & Accountability

NRC National Research Council

NRCan Natural Resources Canada

PHAC Public Health Agency of Canada

TB Treasury Board

TBS Treasury Board Secretariat

TC Transport Canada

UNFCCC United Nations Framework Convention on Climate Change

Introduction

• This report has been prepared in fulfillment of a commitment to produce a horizontal roll-up of the results of the evaluations done for the Clean Air Agenda (CAA) themes.

• The roll-up is intended to help participating departments and agencies report on their CAA results to Parliamentarians and Canadians, and to inform decisions regarding CAA funding and activities beyond 2011.

• The report comprises four sections: CAA Profile; Methodology; Findings; and Observations.

Acknowledgements:

• The project team was led by Martine Perrault under the direction of the Environment Canada Evaluation Director, Shelley Borys, and included Linda Lee, Gavin Lemieux and Katheryne O’Connor. The team would also like to thank all the individuals from the following organizations who participated in the working group for this project and well as contributed to the development of the evaluation reports that form the basis for this horizontal roll-up: Environment Canada (EC) (including the Clean Air Agenda - Results Management Secretariat), Department of Foreign Affairs and International Trade (DFAIT), Health Canada (HC), Indian and Northern Affairs Canada (INAC), National Research Council (NRC), National Resources Canada (NRCan), Public Health Agency of Canada (PHAC), and Transport Canada (TC). This report was prepared for Environment Canada by James Coflin from Government Consulting Services (PWGSC).

CAA Profile

This section provides an overview of the CAA under these headings:

• Background;

• Overview of Themes;

• CAA Program Components;

• Planned and Actual Expenditures by Theme;

• Agenda Logic Model; and

• Governance.

Background

• In the 2006 Speech from the Throne, the Government committed to taking measures to achieve tangible improvements in our environment, including reductions in air pollutant and greenhouse gas (GHG) emissions.

• Budget 2006 committed $1.7 billion, later increased to $2.2 billion, to the Clean Air Agenda (CAA) for the four-year period from 2007-08 to 2010-11.

• The CAA comprises: the Clean Air Regulatory Agenda (CARA), an integrated regulatory approach to reducing air pollutant

(nitrogen oxides, sulphur oxides, volatile organic compounds, and particulate matter) and GHG emissions, and

a series of program measures to: i. support action by governments, individuals, organizations and businesses in reducing emissions of

GHGs and air pollutants, and ii. to provide effective responses to climate change.

• The CAA is a horizontal initiative comprising eight themes and 45 programs. (Note: The Agenda was expanded to include the Clean Energy Dialogue as a component of the International Actions theme in 2009-2010).

• A tabular overview of the eight themes is presented on the next page, followed by lists of the program components within each theme, identifying the department(s) that is delivering the program.

• Detailed information on the Agenda and the program components are available in the 2008-09 Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives: http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481.

Overview of Themes

Theme Lead department2007-2011 Allocation

Clean Air Regulatory Agenda (CARA)a Environment Canada $347.3 M

Clean Energy Natural Resources Canada $1177.0 M

Clean Transportation Transport Canada $461.7 M

Indoor Air Quality Health Canada/National Research Council $23.0 M

Adaptation Environment Canada $115.9 M

International Actions Environment Canada $50.0 M

Partnershipsb Environment Canada $12.0 M

Management and Accountability Environment Canada $5.0 M

a This total does not include additional funds under a reprofiling of CARA that was approved in 2009. b Note: The Partnerships Theme was not launched.

Source: Environment Canada, 2008-2009 Departmental Performance Report, Supplementary Information, Table : Horizontal Initiatives. http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481. Unless otherwise stated, all budget allocation numbers reported in this document are from this source.

CAA Program Components 1 Clean Air Regulatory Agenda (EC)

1. Industrial Sector Regulatory Actions (EC)

2. Transportation Sector Regulatory Actions (EC, TC & NRCan)

3. Consumer and Commercial Products Regulatory Actions (EC, NRCan)

4. Indoor Air Quality Management Actions (HC)

5. Science in Support of Regulatory Activities & Accountability (EC, HC)

6. Emissions Reporting (EC)

7. Emissions Trading System (EC)

8. Enforcement and Reporting on Progress (EC)

9. Policy Analysis (EC)

Clean Energy (NRCan)

1. ecoENERGY for Buildings and Housing (NRCan)

2. ecoENERGY Retrofit (NRCan)

3. ecoENERGY for Industry (NRCan)

4. ecoENERGY for Renewable Power (NRCan)

5. ecoENERGY Renewable Heat (NRCan)

6. ecoENERGY for Technology (NRCan)

7. ecoENERGY for Aboriginal and Northern Communities (INAC)

8. Policy, Communications, Monitoring and Reporting (NRCan)

International Actions (EC)

1. International Obligations (EC, DFAIT)

2. International Negotiations (EC, NRCan, DFAIT)

3. Asia-Pacific Partnership (NRCan, EC, IC)

4. PM Annex (EC)

5. Clean Energy Dialogue (EC, NRCan, DFAIT)

Partnerships (EC)

1. Community Partnerships (EC)

Clean Transportation (TC)

1. ecoMobility (TC)

2. ecoTECHNOLOGY for Vehicles (TC)

3. ecoENERGY for Personal Vehicles (NRCan)

4. ecoENERGY for Fleets (NRCan)

5. ecoFREIGHT Partnerships (TC)

6. National Harmonization Initiative for the Trucking Industry (TC)

7. Freight Technology Demonstration Fund (TC)

8. Freight Technology Incentives (TC)

9. Marine Shore Power Program (TC)

10. ecoAUTO Rebate Program (TC, HRSDC)

11. Vehicle Scrappage Program (EC)

12. Analytical & Policy Support (TC)

Indoor Air Quality (HC/NRC)

1. Indoor Air R&D Initiative (NRC)

2. Radon Strategy (HC)

Adaptation (EC)

1. Assist Northerners in Assessing Key Vulnerabilities and Opportunities (INAC)

2. National Air Quality Health Index and Forecast Program (HC, EC)

3. Climate Change and Health Adaptation in Northern / Inuit Communities (HC)

4. Improved Climate Change Scenarios (EC)

5. Innovative Risk Management Tools

6. Regional Adaptation Action Partnerships (NRCan)

7. Climate and Infectious Disease Alert and Response System to Protect the Health of Canadians (HC, PHAC)

Management and Accountability (EC)

1. Management & Accountability (EC)

Planned and Actual Expenditures by Theme

($ million) 2007-08a 2008-09a

Theme Approved Actual Approved Actual

Clean Air Regulatory Agenda 63.8 42.2 76.0 62.6

Clean Energy 101.5 81.6 206.9 202.5

Clean Transportation 98.8 92.6 219.4 160.4

Indoor Air Quality 5.5 4.6 6.4 6.4

Adaptation 7.3 5.7 26.0 20.3

International Actions 13.2 7.8 10.7 9.7

Partnerships b 3.0 1.0 3.0 0.3

Management and Accountability 1.3 1.2 1.3 1.0

Total 294.4 236.7 549.7 463.4Source: Environment Canada, 2007-08 and 2008-2009 Departmental Performance Report, Supplementary Information, Table: Horizontal Initiatives.

a Expenditures reported here may differ from those reported by individual departments or CAA themes due to differences in reporting periods.

b Although the Partnerships Theme program activities were not launched, some expenditures were incurred during the program’s planning phase.

Agenda Logic Model

Canada’s emission of air pollutants has been

reduced

Health, economic and environmental benefits for

Canadians have been realized

Risks to communities, infrastructure and to the health

& safety of Canadians resulting from Climate Change

have been reduced

Risks to the health of Canadians and the

environment resulting from exposure to air pollution

have been reduced

Innovations related to reducing air pollution and addressing climate

change have maintained Canadian competitiveness and provided

economic benefits.

Canada’s emission of GHGs have been

reduced

Indoor air quality has been improved

Canadians and communities have taken actions to reduce their vulnerabilities from and

have adapted to predicted impacts of Climate Change

Trans-boundary movement of air

pollutants has been reduced

International trends on climate change are

consistent with Canada’s interests.

1

2 3

4 5 6 7 8 9 10

Long-Term Outcomes(Government of Canada)

Canadians take actions in

response to forecasted

levels of air quality

International Air quality agreements

are consistent with Canada’s

interest

Reduction measures for

indoor air pollutants and

radon exposure have been

implemented

Targeted industrial sectors have

reduced emissions of GHGs and air

pollutants and improved energy

efficiency

. Emissions of GHGs and air

pollutants have been reduced, and energy efficiency has improved in Canadian homes

and buildings

Emissions of GHGs, air

pollutants and energy consumption have been reduced

from modes of transportation

Emissions of GHGs and air pollutants have been reduced

and energy efficiency has improved from the use of efficient

and environmentally safe practices and

products

Emission of GHGs and air pollutants have been reduced due to the use of

clean and renewable energy

sources.

Canadian communities and

user groups use tools and information to

assess climate change risks and plan adaptation

strategies

International climate change agreements are consistent with

Canada’s interests

20191817161514131211

Intermediate Outcomes(Clean Air Agenda)

Regulations, Standards, Guidelines, Agreements

Financial incentives

Clean Transportation Clean Energy Partnerships Adaptation Indoor Air Quality International

Theme Outcomes

Technology Development & industrial R&D

Research studiesInformation & Decision

SupportPolicy advice

Partnerships & networks

Tools, training, processes and systems

CAA Outputs

Management & Accountability

Clean Air Regulatory Agenda

V1.0

Governance

• EC is the lead department on the overall CAA and is responsible for the Clean Air Agenda-Results Management Secretariat (CAA-RMS). In all, eight departments (EC; NRCan; TC; NRC; INAC; HC; PHAC; and DFAIT) are responsible for themes and/or individual programs. Note: Two additional departments (Industry Canada and Human Resources and Skills Development Canada) are partners in specific programs.

• The CAA accountability structure is composed of these interdepartmental committees: DM Committee. ADM Steering Committee DG Theme Management Committees DG Theme Leads Coordinating Committee (DGTLCC)

• The DGTLCC is the primary operational committee responsible for implementing the elements of the CAA, comprising the leads from each of the eight DG Theme Management Committees.

• The CAA-RMS is the main support structure for the CAA management and accountability theme. Its key functions include:

Providing support and coordination for the horizontal management of the CAA across departments; Maintaining a database of financial resource information; Analyzing financial and non-financial information to support an assessment of performance across the

CAA, including achievement of results, use, allocation and sufficiency of resources, and risks; and Providing relevant reports, as required, to the governance structure and TBS.

Methodology

This section discusses the plan for the CAA evaluation under these headings:

• Scope and Coverage Limitations;

• Approach and Methodology;

• Theme and Program Evaluations Used for the Horizontal Roll-Up;

• Evaluation Issues and Questions; and

• Methodological Limitations.

Scope and Coverage Limitations

Scope • The roll-up addresses the evaluation questions (page 15) set out in the 2008 horizontal evaluation plan.

• It focuses on assessing theme and program level outputs and immediate outcomes (as reported in CAA program and theme evaluations) that are contributing to the achievement of the CAA intermediate outcomes (see Logic Model, page 9). The CAA long term outcomes were not assessed as part of this horizontal roll-up because they are expected to occur in a longer time frame.

Coverage Limitations

• The roll-up covers 39 (87%) of the 45 CAA program components (see page 14).

• The combined four-year funding for the programs covered in this roll-up represents about 95% ($2.1 billion) of the $2.2 billion CAA allocation (see page 14).

• Most of the contributing evaluations were conducted in 2009-10, thereby only covering 2 to 2½ years of implementation of the CAA activities.

• Evaluations not included in the current roll-up will be reported separately when available. The current roll-up was finalized without the benefit of those missing components due to a commitment to provide this horizontal report in October 2010 to inform discussions on future CAA program activities.

Approach and Methodology

Purpose • This report, which examines the relevance and performance of the Clean Air Agenda, is intended to meet

reporting commitments and to contribute to ongoing clean air and climate change policy development.

Audience • The CAA evaluation roll-up is intended to help meet the information needs of senior management at the

departments and agencies that have developed and delivered CAA themes and program components, as well as at the Treasury Board Secretariat.

Methodology • The sole line of evidence for this report is the content of the two thematic roll-up reports and 14 theme and

program level evaluations available for review by September 23, 2010.2 This excludes evaluations that are still pending, as specified on page 14.

• The following methods were used in the 14 theme and program evaluations that contributed information to this roll-up:

key informant interviews (14), literature review (6), document review (14), analysis of performance data (5), analysis of secondary data (1), case study (5), survey (2), and experts group (1).

• The evidence was extracted from the reports to produce crosswalk matrixes that mapped equivalent data from multiple theme and program evaluations to fields defined by the CAA evaluation questions.

Theme and Program Evaluations Used for Horizontal Roll-UpLegend: ✓- Evaluation of this theme/program was included in this roll-up; X – Evaluation of this theme/program is in progress or planned.

a - This total does not include additional funds under a reprofiling of CARA that was approved in 2009. b - This program was not subject to a discrete evaluation as per NRCan’s evaluation plan.c - These two programs extended their Terms and Conditions for one year and will thus expire in 2011-2012. As a result, the evaluation was pushed back until winter 2011.

Themes/Programs Number of Programs Lead Type Included

4-year Allocationa

1. Clean Air Regulatory Agenda (CARA) 9 EC Thematic $347.3M

2. Clean Energy 8 NRCan Thematic ---

2.1 ecoENERGY for Buildings and Housing NRCan Program $61.0M

2.2 ecoENERGY Retrofit NRCan Program $520.0M

2.3 ecoENERGY for Industry NRCan Program $18.0M

2.4 ecoENERGY for Renew able Pow er NRCan Program $276.0M

2.5 ecoENERGY Renew able Heat NRCan Program $36.0M

2.6 ecoENERGY for Technology NRCan Program $226.4M

2.7 ecoENERGY for Aboriginal & Northern Communities INAC Program X $15.0M

2.8 Policy, Communications, Monitoring and Reporting NRCan Program b $24.6M

3. Clean Transportation 12 TC Thematic $461.7M

4. Indoor Air Quality 2 HC Thematic $23.0M

5. Adaptation 7 EC Roll-up ---

5.1 Assist Northerners in Assessing Key Vulnerabilities and Opportunities INAC Program $14.0M

5.2 National Air Quality Health Index and Forecast Program EC / HC Program $30.0M

5.3 Climate Change and Health Adaptation in Northern / Inuit Communities HC Program $7.0M

5.4 Improve Climate Change Scenarios EC Program $15.0M

5.5 Innovative Risk Management Tools

5.6 Regional Adaptation Action Partnershipsc

5.7 Climate and Infectious Disease Alert and Response System HC / PHAC Program X $14.9M

6. International Actions 5 EC Thematic $50.0M

7. Partnerships (Not launched) 1 EC - - -- $12.0M

8. Management and Accountability (HMARF) 1 EC Thematic $5.0M

Total 45 $2,191.9M

$35.0MX Program NRCan

Evaluation Issues & Questions

Relevance – Are CAA activities consistent with and do they contribute to addressing federal government clean air priorities and the key environmental needs of Canadians?

1. Are CAA activities aligned with federal government priorities?

2. Are CAA activities connected with key environmental needs?

Success – Has the CAA achieved its intended outcomes?

3. To what extent has the CAA achieved its intended outcomes?

Cost-Effectiveness – Are the most cost-effective and efficient means being used to achieve outcomes?

4. Are there more cost-effective and efficient means of achieving CAA objectives?

5. How could the efficiency of CAA activities be improved?

Design & Delivery – Is the CAA designed and delivered in the best possible way?

6. Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time?

7. Is the CAA management and accountability structure in place and functioning as anticipated?

8. Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers?

9. What are the best practices and lessons learned from CAA activities?

Methodological Limitations

Limitations affecting the horizontal roll-up

• Roll-up vs. horizontal evaluation: The large scale of CAA activities (i.e., $2.2B, 45 programs and 8 departments) dictated the conduct of several theme and program level evaluations, and a subsequent horizontal roll-up of these evaluation findings, rather than an umbrella evaluation of the overall CAA.

• Program maturity: Most evaluation evidence was collected during early program implementation and therefore relies on limited performance data and/or data focusing on outputs rather than outcomes. Annual performance reports were however developed for the first two years of the CAA.

• Lines of evidence: Many of the evaluations contributing to this report were heavily reliant on information drawn from key informant interviews. The roll-up of the interview evidence cannot be considered highly reliable given the open-ended nature of the interviews, the variations in questions, and the self-interest of a majority of the respondents (about 46% of the 547 interviews conducted were with federal government employees managing and/or delivering the programs).

• Level of Focus: It is difficult to draw conclusions about some of the CAA evaluation questions because of the level of detail and focus of the contributing evaluations. Specifically, theme-level evaluations and theme-level roll-ups tend to produce generalized or heavily qualified findings that cannot be quantified or consolidated.

Underlying Limitations

The following methodological and analytical limitations were reported by more than one of the theme evaluations. (The number in parentheses indicates the number of evaluations identifying the limitation) :

• Program maturity (4): The programs were not completely implemented and it was too early to assess outcomes.

• Lines of evidence (3): Only two or three lines of evidence were used.

• Limited performance data (6): The available data were limited to output/activity level indicators.

• Policy processes (5): It was difficult to access/collect relevant output and outcome data on such processes.

• Cost data (4): Current and/or sufficiently detailed program and activity level financial data was not available.

Findings

This section rolls-up the findings and conclusions from the program and theme-level evaluations that are relevant to each horizontal-level evaluation question.

Relevance

1. Are CAA activities aligned with federal government priorities?

Evidence

• Overall CAA activities and outcomes are seen to align with federal priorities as expressed in documents and statements such as:

Budgets and speeches from the Throne; Turning the Corner, the April 2007 Government of Canada plan to reduce greenhouse gases and air

pollution; UNFCCC commitments, including the commitment under the Copenhagen Accord to a 17% reduction in

GHG emissions from 2005 levels; Canada’s Performance 2008–09 that identifies “a clean and healthy environment” and “healthy

Canadians” as two of the government’s thirteen outcome areas; A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act , May 2010.

• The evaluations confirmed that the program components they reported on are aligned with departmental priorities and program activity architectures.

Finding

• CAA activities are aligned with federal priorities.

Relevance

2. Are CAA activities connected with key environmental needs?

Evidence

• The CAA evaluation plan listed three indicators for this question: demonstration of the utility/rationale for CAA activities; CAA programs remain relevant as new scientific evidence on air quality and climate change emerges; and, evidence that CAA program architecture has adapted to new evidence and changes of government policy and priorities.

• The theme and program evaluations, with the exception of the M&A theme evaluation, considered “needs” in the course of assessing the relevance of programs and activities. However, the interpretation of the concept and the kinds of evidence gathered varied. Some looked at key informant opinions about workloads or the responses of beneficiaries and stakeholders as evidence of need, while others referred to research results and/or secondary data sources to document air pollutant and GHG emission levels. The Clean Energy theme report demonstrated a continued need for several of the theme’s program activities.

• Three of the thematic evaluations examined whether new scientific evidence had emerged that would raise questions about the relevance of CAA activities – none was found.

• There were no discussions or reports of modified program architecture.

Finding

• Several of the theme and program evaluations provided evidence to demonstrate that CAA activities are connected to key environmental needs.

Success


Approach

• The roll-up assessed the extent to which each CAA intermediate outcome (see Logic Model, page 9) has been achieved by examining the “success” evidence from the evaluations for the themes that are expected to be the major contributors to each outcome, as identified below:

CAA Intermediate Outcomes CARA CT CE ADAPT IAQ INTL

Canadians take actions in response to forecasted levels of air quality. International air quality agreements are consistent with Canada’s interests. Reduction measures for indoor air pollutants and radon exposure have been implemented. Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency. Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings. Emissions of GHGs, air pollutants energy consumption have been reduced from modes of transportation. Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products.

Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources. Canadian communities and user groups use tools and information to assess climate change risks and plan adoption strategies. International climate change agreements are consistent with Canada’s interests.

Success


Approach (continued)

• For each of the CAA intermediate outcomes, success information is presented under three headings: The plans made to achieve the outcome, including which themes/programs were involved and the nature

of interventions -- incentives; regulations and agreements; or capacity building -- that are being used.

The term “capacity building” is used here to refer to activities that involve obtaining or distributing information with the goal of changing individual and organizational behavior including research, public information products and services, social marketing campaigns, training and educational activities, and demonstration projects.

The evidence from the theme and program evaluations about the extent to which planned outputs have been produced and intended outcomes achieved.

Findings that can be drawn from the available evidence. It is important to note here that outcome data were not available for most of theme and program level evaluations because programs had not been fully implemented when evaluated, or their outputs were only expected to have the impacts that will contribute to key outcomes some months or years later. The findings sections that follow do not repeat this qualification, but rather highlight where outcomes data are available.

continued......

Success


Estimate of Progress

• An indicator (see example below) was developed to provide a high level estimation of progress made toward the achievement of each CAA intermediate outcome based on the evidence reported by the theme and program evaluations reviewed:

No outputs – no outputs because program implementation was delayed or terminated. Partial outputs – outputs are being produced but not to planned levels, or work has been significantly

delayed. Outputs delivered – most planned outputs are being delivered, but insufficient time has elapsed for their

contribution to CAA intermediate outcomes to have been experienced. Outcomes likely achieved – given the extent of outputs delivered and program maturity (i.e., sufficient

time has elapsed since program implementation), and assuming that the CAA results logic is accurate, it is likely that the intended intermediate outcome is being achieved. There was however no data collected in the program or theme evaluation to assess whether the outcome was being achieved.

Outcomes achieved – there is reliable evidence that the intermediate CAA outcome has been achieved.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved

CAA Intermediate Outcome:

Canadians take actions in response to forecasted levels of air quality.

The Plan

• The seven Adaptation theme programs were designed to improve Canada’s capacity to reduce health risks from and adapt to a changing climate and air pollution. The program designers expected increased capacity to be achieved through research and the development of tools, partnerships and standards. Research and tool development is being done directly by the participating departments and by other organizations supported through contribution funding.

• The theme targets: the North; human health; and infrastructure.

Evidence

• The Adaptation programs, with one exception, were launched in 2008-2009, not in 2007-2008, the first year of the CAA. The exception is the AQHI which is the extension of a predecessor program.

• The theme evaluation, drawing on four program evaluations, reported that the programs were being implemented as planned, although two of them were at risk of partial or delayed implementation.

• The available evidence showed that where implementation has begun, the programs are beginning to achieve immediate theme-level outcomes: increased capacity to conduct science; greater collaboration; and increased availability of and access to products.

Findings

• There is some preliminary, although weak, evidence that the Adaptation theme is making progress toward achieving the theme’s immediate outcomes that are expected to contribute directly to this CAA intermediate outcome.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


International air quality agreements are consistent with Canada’s interests

The Plan

• The PM Annex component of the International Actions theme is working toward the negotiation of an annex to the 1991 Air Quality Agreement. Under such an agreement, the U.S. and Canada would commit to reducing emissions of PM (particulate matter) and its precursors. PM comprises air-bound, microscopic pollutants that have negative impacts on air quality, the environment and human health.

Evidence • The PM Annex initiative has two major outputs: i) partnerships and processes; and ii) formal and informal

consultations with domestic stakeholders.

• The International Actions theme evaluation reports evidence that some progress was made toward developing partnerships and processes with the U.S. to establish the PM Annex.

• The evaluation found no evidence that stakeholders were consulted directly and it was not clear how Canada’s position was developed or how the views of domestic partners and stakeholders have had an influence or been meaningfully reflected.

• Changes in policy direction and positions on key issues in both Canada and the US have resulted in delays in achieving the PM Annex.

Findings • PM Annex activities have produced outputs related to partnerships and processes but there is little evidence to

show that consultations with domestic stakeholders have helped to inform the development of Canada’s position in PM Annex negotiations. Such consultations are necessary to ensure the PM Annex is consistent with Canada’s interests.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


Reduction measures for indoor air pollutants and radon exposure have been implemented

The Plan• The CAA expected to achieve this intermediate outcome through:

the Indoor Air Quality theme, which is focusing on capacity building activities including research on improved ventilation, distribution and other indoor air quality solutions, construction of a indoor air research laboratory, creation of a national committee on indoor air quality; the mapping of radon zones; testing radon levels at federal sites and a radon public awareness campaign; and

The CARA theme that planned new federal guidelines (regulations) for indoor air contaminants, and a national radon risk management strategy.

Evidence• Because it was done during the third year of a four-year program and many of the outcomes are not anticipated

until the end of the four-year period, the IAQ evaluation examined the progress that had been made under the theme’s two program components. It found that they were generally on track to deliver their planned outputs. The notable exceptions are:

the programs have not delivered the planned radon public awareness campaign or the assessments of the indoor air quality public awareness effort due to restrictions on the conduct of public opinion research; and,

radon testing of federal buildings is behind schedule. • The CARA evaluation reported that federal guidelines for indoor radon, carbon monoxide and ozone levels have

been published within planned time frames and that related work is generally on track.

Findings • The Indoor Air Quality Theme and CARA have produced, or are on track to produce, the outputs that are

expected to contribute to the achievement of theme and CAA intermediate outcomes, but it is too early to expect that the outputs have had their intended impact.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency

The Plan

• The Industrial Regulations component of the CARA theme was designed to develop and implement sector-specific GHG emissions regulations.

Evidence

• The CARA evaluation reports that the development of regulations for industrial greenhouse gas emissions, as well as the development of emissions trading and offset systems, have been affected by “directional change in policy.” The program, however, has undertaken many of the planned activities (consultations, research, drafting instructions), but further work has been deferred.

Findings • CARA has not yet had the opportunity to produce the outputs needed to achieve the GHG and air pollutant

emission reductions required for the achievement of this CAA outcome.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings

The Plan• The Clean Energy theme offered homeowners, as well as small and medium –size organizations, incentives to

retrofit or construct energy efficient homes and buildings through two programs: ecoENERGY Retrofit ($805 M) and ecoENERGY for Buildings and Houses ($61 M).

Evidence• At the end of its first two years (2007-2008 and 2008-2009), the ecoENERGY Retrofit for Homes program had

already achieved 64-80% of its targeted range for GHG emission reductions and energy savings. Evidence suggests that 84% of energy savings are directly attributable to the Program. Moreover, homeowners are undertaking double the number of retrofits they had originally planned and 60% are motivated to undertake further measures outside of the Program.

• At the end of its first two years (2007-2008 and 2008-2009), the Buildings and Houses program had achieved 63-71% of its targeted range for energy savings and 80% of its GHG emission reductions and had effectively extended the reach of NRCan’s energy efficiency programs in the residential sector. The program has also exceeded its target of 5,000 trained energy advisor professionals in the construction of energy efficient new homes and buildings.

• The ecoENERGY Retrofit for Small and Medium Organizations program had achieved GHG emission reductions and energy savings but not to the extent anticipated when targets for the programs were set. Furthermore, attribution of this progress to the program’s financial incentive activities is only partial as some firms would have implemented these changes on their own. For the most part, capacity building activities (e.g., training and design validation), are increasing the capacity of commercial / institutional energy users to adopt energy management best practices and implement energy efficiency measures. In turn, this contributes to the outcome of energy savings and GHG emission reductions

Findings• Evidence shows that, although they have not yet fully reached all of their intended targets, Clean Energy programs

are achieving GHG reductions and energy efficiency improvements in the Canadian homes and buildings sector.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


Emissions of GHGs, air pollutants, energy consumption have been reduced from modes of transportationThe Plan• Two themes are contributing to the goals of increasing fuel efficiency and reducing GHG and air pollutant

emissions within the personal and commercial transportation sectors: The CARA theme is focused on developing and implementing a range of regulations and related

products; and the Clean Transportation theme comprises capacity building and incentive programs intended to

increase the adoption of fuel efficient transportation technologies and behaviours, as well as plans to establish voluntary emission reduction agreements with the air, railway and marine sectors of the transportation industry.

Evidence• The CARA theme evaluation found that the development of transportation-related regulations was

happening as planned and within scheduled time frames. • The Clean Transportation theme evaluation found that, with small exceptions and some delays, the

initiative was delivering its planned activities and outputs. With respect to outcomes, it found evidence that the ecoAUTO Rebate program , an incentive program, had contributed to an increased uptake of fuel-efficient technologies in the personal vehicle market. There was also evidence of some increased awareness of energy efficient transportation technologies among target audiences.

• With respect to the capacity building programs, both the theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found the required outputs had been delivered but that outcomes could not be assessed due to a lack of baseline and outcomes data and the challenges of attributing changes in consumer and business behaviours to specific interventions.

• The Clean Transportation theme evaluation reports that the voluntary agreements with the air and railway sectors have been renewed or maintained, but a similar agreement for the marine sector has not been developed.

Findings• By and large, the CAA transportation-related activities have delivered the regulatory, incentive and

capacity building outputs that are expected to contribute to the achievement of this CAA outcome.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products

The Plan

• It was expected that this outcome would be realized as a result of capacity building outputs such as public information services, activities promoting smarter energy use, support for the development and deployment of clean energy technologies, and fuel efficient driver training delivered by the Clean Energy and Clean Transportation themes, as well as regulatory actions developed through the CARA theme.

Evidence

• The Clean Energy evaluation reports that awareness building activities and adoption of technologies under the ecoEnergy for Industry program contributed to reduced emissions. At the time of the evaluation, the program was within its target range for energy savings and GHG reductions. It however reports that, while the activities and programs funded within the ecoEnergy for Technology Initiative lay the groundwork for increased deployment of clean power generation, reduced emissions from this program will depend on the degree to which these technologies are deployed on a large scale.

• The Clean Transportation theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found that the planned capacity building outputs were being produced. The theme evaluation also highlighted the challenges of attributing changes in consumer and business behaviour to specific capacity building interventions.

• With respect to regulatory interventions, the CARA theme evaluation reported that the planned development of energy -efficiency standards for consumer products and regulations to reduce the emissions of volatile organic compounds (VOC) in the manufacture and use of consumer and commercial products were on track, but that the impact of the regulations would not be measurable until some years after they had come into effect. NRCan’s evaluation of Equipment programming indicated that its Energy Efficiency Standards and Labelling Program contributes to increasing the minimum energy efficiency of equipment manufactured by suppliers, and therefore purchased by consumers, thereby contributing to reductions in energy use and associated GHG emissions.

Findings

• There is evidence that the Clean Energy, Clean Transportation and CARA themes have produced the regulatory and capacity building outputs designed to increase the use of efficient and safe practices and products and some evidence that this outcome is being achieved as a result. It is however too early to conclude on the achievement of this outcome for 2 of the 3 themes involved.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources

The Plan

• The CAA expects to achieve this outcome through the incentives delivered by two Clean Energy theme programs:

ecoENERGY for Renewable Power ($276.0 M) provides incentives for the producers of renewable power (e.g., from wind, low impact hydro, biomass and solar photovoltaic); and

ecoENERGY for Renewable Heat ($36.0 M) uses contributions to commercial, industrial and institutional users to encourage the installation of solar water and air heating systems. As more are installed, prices are expected to fall and the consumption of conventional energy should decline.

Evidence

• By April 2009, Renewable Power projects with a total capacity of 3,100 megawatts had been commissioned or were under construction – the program’s target is approximately 4,000 megawatts by March 31, 2011.

• During its first two years, the Renewable Heat Program supported 334 solar thermal projects – its target is 700 systems by March 31, 2011.

Findings

• NRCan’s Renewable Energy Evaluation concluded that, after two years, the renewable energy programs have delivered planned outputs and are on track to reach most of their intended outcomes. In addition, it expressed the opinion that many of the programs’ long-term outcomes will be achieved, although it is not possible to directly attribute the achievement of these long-term outcomes to the programs due to interacting factors and considerations.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved

CAA Intermediate Outcome:Canadian communities and user groups use tools and information to assess climate change risks and plan adaption strategies.

The Plan

• Several of the Adaptation Theme programs included capacity development activities (i.e., the development and distribution of tools and information such as guidelines, methodologies, data tables, models, and web pages) to help researchers, government departments and agencies, businesses, community groups and others assess risks and plan adaptive measures.

Evidence • The Adaptation Theme evaluation reported that the evaluations completed for four of the seven programs

indicate that the programs were generally being implemented as planned and that Improved Climate Change Scenarios may not be fully implemented within the program timeframe.

• Available information indicates that the use of adaptation information and products is most likely to be used by groups who have an immediate interest in climate change impacts (e.g., scientists or individuals who are at-risk from adverse health impacts).

Findings • The evidence points to increased use of tools and information by groups with a clear interest in, or an incentive to

use, adaptation products and information.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved


International climate change agreements are consistent with Canada’s interests

The Plan• The International Partnerships and Negotiations component of the International Actions Theme is the most

important contributor to the achievement of this CAA outcome. Its activities are intended to ensure that Canadian environmental and economic interests are advanced through international negotiations. Note: For the purposes of evaluation, “Canada’s interests” are those goals and interests defined by the Government of Canada in federal negotiating instructions and formal policy statements.

Evidence • The International Actions evaluation reported that the International Partnerships and Negotiations program had

produced planned outputs.• The evaluation concluded that Canada’s interests are reflected in international climate change agreements,

although it is difficult to attribute this directly to the activities under the International Actions theme. As well, there is clear evidence of enhanced consultation activities but they have not necessarily resulted in increased consideration of stakeholder interests in Canada’s positions

Findings • There is evidence that the International Actions theme has, at least partially, achieved this CAA intermediate

outcome.

Outcomes likely

Outputs delivered

No outputs

Partial outputs

Outcomes achieved

Success

Progress toward Outcomes: Summary

CAA Intermediate OutcomesNo

outputsPartial

outputsOutputs delivered

Outcomes likely

Outcomes achieved

Canadians take actions in response to forecasted levels of air quality.

International air quality agreements are consistent with Canada’s interests.

Reduction measures for indoor air pollutants and radon exposure have been implemented

Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency

Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings

Emissions of GHGs, air pollutants and energy consumption have been reduced from modes of transportation.

Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products.

Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources.

Canadian communities and user groups use tools and information to assess climate change risks and plan adoption strategies

International climate change agreements are consistent with Canada’s interests.

Cost-Effectiveness/ Alternatives 4. Are there more cost-effective and efficient means of achieving CAA objectives?

Evidence

• Opinions about alternative more efficient approaches: Key informants were either satisfied that the current program models were as or more efficient than alternatives, or they did not know of any realistic alternatives.

• Comparison of CAA activities to other comparable programs: Only the Clean Transportation Theme evaluation reported the results of a comparative analysis. It found that comparable programs exist for most Clean Transportation programs, but the availability of cost information for the comparable programs limited the analysis to the ecoAUTO Rebate and the Vehicle Scrappage programs. The Vehicle Scrappage program seems to be more efficient than United States and United Kingdom initiatives based on calculations of incentive cost per vehicle scrapped and total cost per vehicle scrapped. A comparison of the ecoAUTO Rebate program and the U.S. Cash for Clunkers program shows similar unit costs for administration. 3

• Opinions about whether CAA investments are a good use of public funds: While the evaluations do include key informant suggestions for program refinements and adjustments, the overall tenor of the evidence is that program staff and other stakeholders are generally satisfied that the programs are doing their job at reasonable cost and that no realistic alternatives are available.

• Costing emission reductions: The Clean Transportation evaluation calculated a cost per tonne of GHG emissions reduced based on forecasted GHG emissions reductions and original program budgets. The results suggest that the Vehicle Scrappage and ecoAUTO Rebate programs are not cost-effective.

Finding

• While there is key informant and comparative evidence that CAA activities are efficient, calculations of the cost per tonne of GHG emission reductions suggest that some programs, particularly incentive programs, are not cost-effective.

Cost-Effectiveness/ Alternatives 5. How could the efficiency of CAA activities be improved?

Evidence

• The evaluations sought key informant opinions about ways to improve the efficiency of the programs they were associated with.

• The suggestions that were offered and reported tend to be specific to particular programs, even to particular activities.

• A comparison of the evaluations failed to identify any suggestions that were repeated across themes and programs.

• During the first two years of operation CAA theme actual expenditures were under budget (see page 8).

Finding

• There was no evidence of efficiency improvements that would be relevant to CAA activities generally.

Design and Delivery 6. Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time?

Evidence

• Information about the implementation status of 40 of the 45 CAA programs is available.

• The majority of the programs have been implemented as planned, that is they are producing the expected outputs and, generally, within the expected timeframes:

One program, Clean Transportation’s ecoAUTO Rebate program (TC) has been completed; and Twenty-nine programs were identified as on track – delivering their outputs as planned.

• One (1) CAA program was not implemented as planned i.e. Community Partnerships (EC).

• Work on two CARA initiatives – the Emissions Trading System (EC) and the Industrial Sector Regulatory Actions (EC) – has been deferred.

• Seven programs have experienced challenges resulting in delayed or partial implementation: The Clean Transportation programs ecoMobility (TC); Marine Shore Power Program (TC); National

Harmonization Initiative for Trucking Industry (TC), and ecoFREIGHT Partnerships on Freight (TC); The International Actions initiatives PM Annex (EC) and Asia-Pacific Partnership (NRCan); and The Radon Strategy (HC).

Finding

• The majority (30 of 40) of the CAA programs evaluated have been implemented as planned, six have been implemented with significant schedule and/ or output changes, two were not implemented and two are described as deferred.

Design and Delivery 7. Is the CAA management and accountability structure in place and functioning as anticipated?

Evidence

• The CAA management and accountability structure is the subject of a separate evaluation that concluded, in part, that:

The M&A Theme has brought coherence to the horizontal reporting of CAA activities and results and has emerged as a strong success story, in large part because of the CAA–RMS. Central agency representatives said that it was a key model to follow for reporting on horizontal initiatives.

Roles and responsibilities for CAA management have been clearly defined, processes to support decision making and coordination have been established, and results are being reported to Parliament and Canadians in a transparent manner.

There was limited evidence, however, of policy and program adjustments being made within the CAA horizontal decision-making structures. When needed, horizontal collaboration has occurred through other more flexible channels.

• The general tenor of the management and accountability evaluation findings at the theme level tend to validate the M&A theme evaluation conclusion: horizontal management structures have been implemented and play an important role in reporting, but have limited roles in program and policy decision-making, in part due to limited interdependencies between themes (see page 20). Instead, decision making operates on vertical lines of accountability and, where there are interdependencies between programs, bilateral planning and coordination solutions are used.

Finding

• The agenda and theme level horizontal management structures have been implemented as planned. Their functioning in terms of CAA reporting responsibilities exceeds expectations, but they have not played any significant role in decision-making.

Design and Delivery

8. Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers?

Evidence

• The CAA evaluation plan listed three indicators for this question:

• Populated performance data systems: The key informant and document review evidence reported by the theme and program level evaluations suggests that the programs are meeting basic departmental and horizontal reporting requirements.

The evaluations do not contain enough specific information to do a program-by-program analysis, but taken together, their observations suggest that a minority of programs have fully functioning systems.

The most commonly identified issues were: the lack of an adequately structured and maintained system; processes or systems that focus on activities and outputs, but lack key outcome data; and indicators that are not relevant or appropriate.

Six (6) of the nine (9) that examined performance data questions made recommendations for improvements to performance data.

• Decisions based on performance information: Only two evaluations reported on the use of performance data in decision making: the International Actions evaluation observed that it was used for departmental reporting, but little else, while the Clean Transportation document review and key informant interviews found examples of the use of such data in program management decision making, as well as reporting.

• Baseline data: Only a few programs have collected or will be able to collect baseline data. As a result it will be difficult, if not impossible, to assess the achievement of intermediate and longer-term outcomes whether the outputs involved incentives, capacity building activities or regulations.

Finding

• It appears that CAA programs have established systems or processes to gather and maintain activity and output data that supports program management, but some are not collecting the baseline and outcomes data needed to measure and evaluate performance.

Design and Delivery 9. What are the best practices and lessons learned from CAA activities?

Evidence• The following best practices and lessons learned are those at the Agenda level that cut across several CAA

themes or programs. Other best practices and lessons learned can be found in the CAA theme and program evaluation reports.

• Best Practices: A best practice is a methodology or work process that has been demonstrated to achieve superior results. The evaluations reported a number of best practices, but often fail to provided evidence to demonstrate that the activity described was a superior solution.

• The best practices at the Agenda level reported by the Management and Accountability evaluation were: Requiring approval of CAA performance reports by DMs of partner departments and agencies before the

reports are submitted to TBS. The detailed charter, and the CAA–RMS implementation of the terms of the charter. The approach of the CAA–RMS to ensure effective financial and performance data collection, compilation

and reporting (acknowledged by TBS to be a model of reporting on a horizontal initiative). • Lessons Learned: A lesson learned describes an aspect of a project or program that did or did not work well, and

identifies the risk of ignoring the lesson. In all, six of the evaluations reported a total 19 “lessons learned.” • The evaluations tended to provide only a brief description of the lesson without context or identification of risk. • While the lessons tended to be specific to a particular program or activity, it is worth noting that four of the

evaluations (CARA, Adaptation, Renewable Energy and IAQ) listed nine lessons learned that called for proactive communication to and engagement of external stakeholders, including communities and individuals.

Finding• The CAA–RMS led processes for financial and performance data compilation and reporting represents a best

practice for horizontal initiatives.

Overall Observations

This section presents observations in two groups:

• Program observations: Observations related to overall CAA relevance and performance

• Evaluation observations: Observations related to technical aspects of horizontal evaluations

Program Observations

Relevance

• The CAA programs are aligned with government priorities and are connected to key environmental needs.

Achievement of Outcomes

• Early evidence shows that a large majority of the planned activities are being implemented and delivered.

• As acknowledged in the 2008 CAA horizontal evaluation plan and the theme and program evaluations, the impacts of most of the CAA programs will not be felt or measureable for some time, even if the program has been fully implemented.

• Some outcome data for incentives programs (ecoEnergy for Retrofit, Renewable Energy, eco AUTO, and Vehicle Scrappage) suggests that there is increasing uptake of energy efficiency products and services, fuel-efficient technologies and clean energy sources, but there are also outstanding attribution challenges.

• With the exception of Clean Energy programs, there is mostly anecdotal evidence about the likely impacts of CAA capacity building activities and, without baseline data and given restrictions on survey research, it will be difficult to reliably assess those impacts in the future.

• Literature reviews have found international evidence about the effectiveness of regulatory interventions, but evaluations of the Clean Air regulations have not yet been done.

Cost Effectiveness

• There is preliminary evidence that incentive programs, like the ecoAUTO Rebate program and the Vehicle Scrappage program, may not be cost-effective when measured in terms of estimated emission reduction.

Design and Delivery

• The CAA horizontal management structures have enabled very successful horizontal reporting but have had limited usefulness for program and policy decision-making in light of the limited interdependencies between CAA themes and the maintenance of vertical lines of departmental accountability.

Evaluation Observations

Horizontal Evaluation Frameworks

• The development of the horizontal evaluation plan provided some degree of consistency across the theme and program level evaluations, but not enough to facilitate a strong roll-up of results. Future horizontal evaluation planning might benefit from establishing requirements for all theme and program evaluations to address a small number of core questions, with well defined indicators and comparable/equivalent data sources.

Performance Measurement

• The majority of the theme and program evaluations reported that they encountered problems because there was limited outcome data. In many cases this was because the program had not reached the stage of its implementation where outcomes could be measured, or because of methodological issues such as limitations on public opinion research. In other cases, however, it may be that program managers did not collect and maintain outcome data.

• Evaluation planning would benefit from scheduling early reviews of program performance measurement frameworks and their implementation.

Common Terminology

• The roll-up of the theme and program evaluation results was complicated by different interpretations of some concepts and terms such as needs, utility/rationale, unintended outcome, best practice, and lessons learned. While variation in interpretation and understanding of these terms is common and not usually an issue with a single evaluation, a horizontal evaluation plan would benefit from including a terminology lexicon to obtain consistent evidence across several evaluation units and numerous evaluators.

Endnotes

1. Descriptions of each of the CAA program components are included in the Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives. The 2008-09 supplementary table is published at http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481.

2. The primary information sources for this evaluation roll-up were the following program and theme-level evaluation reports:CARA: Evaluation of the Clean Air Regulatory Agenda -DRAFT (June 14, 2010), Audit and Evaluation Branch, Environment Canada.

CARA: Evaluation of the Equipment Sub-Sub Activity - Draft Final Report (June 28, 2010), NRCan Strategic Evaluation Division. (Note: This evaluation covers NRCan’s Energy Efficiency and Standards Labelling Program, which is a sub-component of CARA under the Consumer and Commercial Products Regulatory Actions program).

Clean Energy: NRCan’s Clean Energy Thematic Report - Draft (September 2010), NRCan, Strategic Evaluation Division.

Clean Energy: Renewable Energy Evaluation – Final Report (July 2010), NRCan, Strategic Evaluation Division.

Clean Energy: Evaluation of the Clean Electrical Power Generation (CEPG) S&T Sub-sub Activity (PAA 2.1.4.5) - Final Evaluation Report. (September 17, 2010), NRCan, Strategic Evaluation Division.

Clean Energy: Evaluation of the Oil and Gas Sub-sub Activity. Draft Report (September 22, 2010), NRCan, Strategic Evaluation Division.

Clean Energy: Evaluation of Energy Efficiency for Industry, Housing and Buildings – DRAFT Report (September 15, 2010), NRCan, Strategic Evaluation Division.

Clean Transportation: Government Consulting Services, Report on the Evaluation of the Clean Transportation Theme of the Clean Air Agenda (July 2010), Transport Canada. (Note: This report covers the Clean Transportation theme programs delivered by Transport Canada and Environment Canada).

Clean Transportation: Evaluation of the Transportation Sub-Sub Activity - Final Report (July 6, 2010), NRCan, Strategic Evaluation Division. (Note: This evaluation covers NRCan’s participation the transportation sector regulatory actions and six programs, two of which are part of the Clean Transportation theme).

Indoor Air Quality: Government Consulting Services, Evaluation of the Indoor Air Quality Theme of the Clean Air Agenda (June 2010), Health Canada and National Research Council.

Adaptation: Evaluation of the Clean Air Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings -DRAFT (July, 2010). Audit and Evaluation Branch, Environment Canada.

Adaptation: Evaluation of the National Air Quality Health Index Program (February 2010), Audit and Evaluation Branch, Environment Canada.

Adaptation: Evaluation of the Improved Climate Change Scenarios Program - Final Draft Report (June 2010), Audit and Evaluation Branch, Environment Canada.

International Actions: Evaluation of the International Actions Theme of the Clean Air Agenda - Draft Final Report (June 2010), Audit and Evaluation Branch, Environment Canada.

International Actions: Marbek | Stratos. Evaluation of the Canada-U.S. Clean Energy Dialogue - Draft Report, Audit and Evaluation Branch, Environment Canada.

Management and Accountability: Evaluation of the Clean Air Agenda Management and Accountability Theme - Draft (Revised July 13, 2010), Audit and Evaluation Branch, Environment Canada.

Endnotes

4. It should be noted, though, that the United Kingdom and United States programs were adopted for economic stimulus reasons rather than to reduce transportation emissions.

5. The NRCan Evaluation of the Equipment Sub-Sub Activity (contributing to the CARA Theme) made similar calculations for the two non-CAA programs it reported on (equivalent data for the CAA programs was not available): (page 42)

The Accelerated Standards Action Program was expected to reduce GHG emissions by 2.8 Mt/year in 2010. In total, $28.4 million was spent under the Accelerated Standards Action Program. Therefore, if the Accelerated Standards Action Program achieved its GHG emissions reductions estimate, it will result in $10.14 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.

EnerGuide for Industry was expected to reduce GHG emissions by 1.4 Mt/year in 2010. In total, $2 million was spent under EnerGuide for Industry. Therefore, if EnerGuide for Industry achieved its GHG emissions reductions estimate, it will result in $1.43 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.

prepared by government consulting services for audit and evaluation branch, environment canada

Documents

clean air agenda caa

caa results

environment canada ec

caa funding

health canada hc

northern affairs canada

transport canada tc

caa profilethis section