(titre-36 – arial black – centré) - environment and … · web viewjuly 2008. report clearance...

Evaluation of theMeteorological Service of Canada (MSC) Transition

Project

July 2008

Audit and Evaluation Branch Evaluation of the MSC Transition

Environment Canada

Audit and Evaluation Branch Evaluation of the MSC Transition Project

Report Clearance Steps

Planning phase completed September 2007Report sent for management response April 2008Management response received June 2008Report completed June 2008Report approved by Departmental Evaluation Committee (DEC)

July 2008

Acronyms used in the report

AES Atmospheric Environment ServicesADMAMDAR

Assistant Deputy MinisterAircraft Meteorological Data Relay

CIOBDMFDMS

Chief Information Officer BranchData Management FrameworkDepartmental Management Services

DSO Departmental Service OrganizationEC Environment Canada EQ Evaluation QuestionITMSC

Information TechnologyMeteorological Service of Canada

MNL

MillionNational (Research) Laboratory

NSO National Service OfficeNSU National Service UnitOPGPAAQMSRWIS

Outcome Project GroupProgram Activity StructureQuality Management SystemRoad Weather Information System

S&TSPC

Science and TechnologyStorm Prediction Centre

TBS Treasury Board SecretariatWEP Weather and Environmental Predictions WES Weather and Environmental Services

Acknowledgments

The Evaluation Project Team including Robert Tkaczyk, Lindsay Fitzpatrick, Karine Kisilenko and Martine Perrault and led by Janet King and Shelley Tice under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights, and comments crucial to this evaluation.

Prepared by the Evaluation Division, Audit and Evaluation Branch

Environment Canada


Table of Contents

EXECUTIVE SUMMARY.................................................................................................... i

1.0 INTRODUCTION...................................................................................................1

2.0 BACKGROUND AND CONTEXT..........................................................................22.1 Profile................................................................................................................22.2 Program Logic Model.........................................................................................52.3 Roles and Responsibilities.................................................................................72.4 Resources.........................................................................................................7

3.0 EVALUATION DESIGN.........................................................................................93.1 Purpose and Scope...........................................................................................93.2 Evaluation Methodology and Approach.............................................................9

4.0 FINDINGS...........................................................................................................144.1 Relevance........................................................................................................144.2 Success...........................................................................................................164.3 Cost-Effectiveness/Alternatives.......................................................................284.4 Design and Delivery.........................................................................................31

5.0 CONCLUSIONS..................................................................................................38

6.0 RECOMMENDATIONS.......................................................................................40

Annex A Planned Resource Allocations by Transition Component................................49Annex B Evaluation Issues and Questions.....................................................................50Annex C Relationship between Expenditure Review Questions and the MSCTransition Project Evaluation Questions.........................................................................53Annex D Key Documents................................................................................................54Annex E Master Interview Guide.....................................................................................58Annex F Summary of Findings........................................................................................63Annex G Budgeted and Actual Expenditures versus Treasury Board Funding for theFirst Four Years of the Transition Project (2003–2004 to 2006–2007)...........................64

Environment Canada


EXECUTIVE SUMMARY

In 2003, the Government of Canada announced increased funding to revitalize and transform the operations of the Meteorological Service of Canada (MSC). The objectives of this project, known as the Transition Project, were to help the MSC become a more sustainable organization, to advance science- and weather-related services, and to help Canadians adapt in ways that safeguard security, health and safety, economic prosperity and environmental quality. The government allocated approximately $74 million (M) in new funds over a five-year period and $5M per year thereafter. This funding was to be combined with $11M from internal allocations over the same five-year period and approximately $2M annually thereafter.

An evaluation of the progress made to modernize Environment Canada’s weather services over the 2003–2004 to 2007–2008 period was required by Treasury Board. The importance of evaluating this significant and transformative measure was also endorsed by Environment Canada’s Audit and Evaluation Committee1 and the conduct of an evaluation was included in the department’s 2007–2008 Audit and Evaluation Plan.

The purpose of the evaluation is to assess

relevance: whether the Transition Project goals were consistent with federal priorities and addressed actual needs;

success: the extent to which the Transition Project has met its intended outcomes; cost-effectiveness / alternatives: whether the most appropriate and cost-effective

means were used to achieve those outcomes; and design and delivery: whether the Transition Project was delivered in the best

possible way.

The evaluation addresses activities funded under the Transition Project over the five-year project period from fiscal year 2003–2004 through 2007–2008. Multiple methodologies were used to collect information. These methods included

document and file reviews; secondary data analyses and reports; a media scan; key informant interviews (58 people in total: 36 internal staff and 22 external

stakeholders); and three facilitated workshops with Environment Canada employees (47 people in total).

The major conclusions of the evaluation are presented by evaluation issue.

1 In 2007, the Departmental Audit and Evaluation Committee was split into two components. The component that deals with evaluation is now known as the Departmental Evaluation Committee.

Environment Canada i


Relevance

The objectives and intended outcomes of the MSC Transition Project a) were aligned with Environment Canada’s mandate for meteorological services and departmental strategic outcomes as set out in corporate documents; b) were related to priorities identified in the Government of Canada’s speeches from the Throne and budgets; and c) addressed actual needs to modernize Environment Canada’s weather services.

Success

The Transition Project has made progress on most of the activities undertaken. These activities have resulted in the consolidation of weather offices, a focus on high-impact weather, increased training and recruitment of staff, the introduction of new products and services and the life-cycle management and modernization of some equipment. Activities that have not been completely implemented and require further attention include the decommissioning and cleanup of some climate and hydrometric sites and the development and implementation of quality assurance processes and data management frameworks.

Despite the achievement of these activities, the intended outcomes at the immediate and ultimate outcome levels will only be partially achieved by project end (2007–2008). For example, while there is evidence of ongoing recruitment in the Meteorology Group and Engineering and Support Staff categories, there is a widespread perception, both internal and external, that current succession planning efforts are inadequate to address the impact of an aging workforce. Notwithstanding the progress made to modernize monitoring networks through network rationalization, considerable work remains to be done. Steps have been taken to enhance the quality assurance of and access to key data, but the data management framework is still under development and quality assurance processes are considered to be under-resourced. Further, even though progress has been made in decommissioning sites and stations and installing new equipment at reference climate stations, the decommissioning of sites has not occurred at the rate expected. These partial achievements weaken the ability of the Transition Project to attain its overarching outcome, which is to make MSC a more sustainable organization.

The progress achieved under the Transition Project was affected by factors internal to the project itself:

1) The costs of some activities (e.g. consolidation and modernization of forecast offices) were underestimated and resources had to be redirected from other activities (e.g. from restoring and developing key skill sets).

2) Staffing, training and consequently relocation processes took longer to get into place than expected (e.g. National Service Offices).

The ability to implement activities and achieve the intended outcomes was also affected by factors external to the project:

Environment Canada ii


1) The ability to achieve outcomes was affected by department-wide changes to governance and resource allocation processes. Budget decisions became the responsibility of boards, not line-based organizations, and resources were centralized and allocated on a priority basis.

2) Government decisions were not matched with funds (e.g. the cost of the offices in Gander, Newfoundland and Winnipeg, Manitoba), meaning resources had to be redirected from Environment Canada’s A-Base programs.

3) The implementation of activities was delayed by policy requirements (e.g. land claims negotiations) and new procedures for the management of property.

Cost-effectiveness / alternatives

Environment Canada interviewees widely agreed that the Transition Project used the most appropriate and effective means of achieving the intended outcomes and that there were no obvious alternatives. Although there is compelling qualitative evidence that the project provided value for the money received, a complete assessment could not be done due to a lack of financial information.

Design and delivery

While the concept and design of the project was considered to be the best way of achieving the intended outcomes, the project was not delivered entirely as planned. Some activities were dropped and others have not been completed to date, notably in the monitoring component of the Transition Project.

As previously indicated, MSC underestimated some of the funding requirements and resources were reallocated within the project.

Pockets of dissatisfaction with the project exist among Environment Canada interviewees and some stakeholders. Environment Canada employees are concerned about the potential to achieve the outcomes sought and some stakeholders express concern over the delivery and quality of services and the adequacy of Environment Canada’s efforts to address the impact of an aging workforce.

As a result of the inability to access all financial information, the evaluation was not able to confirm whether or not the project had remained within budget.

Environment Canada iii


Recommendations

The following recommendations are based on the findings and conclusions of the evaluation. The first three recommendations are directed to the Weather and Environmental Services (WES) Board and the fourth recommendation is addressed both to the WES Board and the Departmental Management Services (DMS) Board.

1. Take stock and communicate which activities need to be continued in support of the modernization and sustainability of Environment Canada’s weather services.

Given the ongoing importance of modernizing Environment Canada’s weather services, attention is needed to maintain the progress and momentum achieved to date. Priority attention, in particular, is required to ensure the sustainability of Environment Canada’s weather services. In deciding which of the activities initiated under the Transition Project need to continue, the WES Board needs to

a) Identify the status of the activities and how the activities relate to the ongoing modernization and sustainability of MSC

The overarching objective of the Transition Project was to help MSC become a more sustainable organization and infrastructure. The achievement of the other ultimate outcomes — strengthened linkages between production, science and service; improved services for Canadians and key stakeholders; and adaptation of Canadians to the environment in ways that safeguard security, health, the economy and quality of the environment — depend on the existence of a sustainable organization and infrastructure. Thus, in considering which activities should be continued, the WES Board needs to assess how these activities relate to the sustainability sought under the project and, further, how these activities contribute to the overall sustainability of Environment Canada’s weather services funded by existing A-Base resources.

In this context, the WES Board also needs to consider the risk to the mandates and priorities of internal stakeholders (e.g. the Environmental Protection Board) and external stakeholders, including other federal departments and agencies, other levels of government, the private sector and Canadians at large of not continuing the activities launched under the Transition Project” to align with what was sent for response.

b) Determine the resource requirements of ongoing activities

The modernization efforts of the Transition Project were to be complemented by other A-Base programs; hence resource allocation should be considered within the broader context of A-Base funding for weather services. Seven million dollars annually has been allocated for the implementation of activities initiated under the Transition Project. In considering the budget allocations, the WES Board needs to consider how to best use and leverage the existing resources (financial and human) provided under the project with MSC’s other A-Base programs.

Environment Canada iv


c) Determine the adequacy of current succession planning efforts

External stakeholders in particular voiced concern over the impact of an aging workforce on the capacity of Environment Canada to deliver high-quality weather services. The WES Board needs to assess the capacity of Environment Canada and the adequacy of current succession planning efforts to support sustainable and high-quality weather services to stakeholders.

d) Articulate a strategic direction and implement an action plan

Based on the information collected above, the WES Board needs to decide which activities will continue to be funded, how the resources will be allocated and linked to existing A-Base programs, and what needs to be done to ensure the capacity needed to deliver quality weather services. The strategic direction should be accompanied by an action plan which would set out specific measurable deliverables, timelines and accountabilities.

2. Communicate the strategic direction and action plan

The strategic direction and the action plan articulated by the WES Board need to be communicated to

a) MSC staff and other internal stakeholders

The vision for the Transition Project was clear and, overall, MSC staff were motivated to achieve the intended outcomes. Over time, Environment Canada interviewees2 became discouraged by their perceived inability to achieve the intended outcomes. Modernization of Environment Canada’s weather services is an ongoing requirement of the department, and employees need to be clear about the direction of programming. The strategic direction and the action plan formulated by the WES Board need to be communicated to the MSC, to the Science and Technology (S&T) Branch and to the staff of the Chief Information Officer (CIOB) Branch.

b) External stakeholders outside of Environment Canada

Many external stakeholders are of the view that the 2004–2005 changes in the organizational structure of MSC and the turnover in personnel have moved Environment Canada away from being a sustainable organization. External stakeholders are concerned that Environment Canada does not have the capacity or succession planning efforts in place needed to create a stable organization. The changes in governance at Environment Canada, along with MSC’s efforts to address succession planning issues, need to be addressed with external stakeholders to ensure their continued support and the continued credibility of Environment Canada’s weather services.

2 The term “Environment Canada interviewees” refers to MSC staff and former MSC staff (now in the Science and Technology Branch).

Environment Canada v


3. Strengthen performance measurement, monitoring and reporting at the outcome level

While information on the progress of the project was collected at the activity level, information on how these activities contributed to the achievement of immediate and ultimate outcomes was not consistently collected and aggregated. Despite the existence of a performance measurement strategy, that strategy was not implemented and information at the outcome level was not readily available. The absence of performance information weakened the ability of the department to tell its full performance story as to how weather services have been modernized over the past five years and to identify ongoing requirements for programming and resources. To demonstrate good stewardship and accountability for the use of public funds and informed decision making, the WES Board needs to strengthen performance measurement, monitoring and reporting at the outcome level.

4. Ensure that mechanisms to track and report financial information are in place

The financial information system did not specifically track the use of Treasury Board Secretariat funds from the use of other A-Base funds or indicate the impact of government and department-wide budget reductions on the Transition Project. Financial information also needs to be linked to results to enable Environment Canada to determine and report on the cost-effectiveness or value for money of programming. To ensure accountability of funds and to demonstrate good stewardship over the use of funds, the WES Board, in conjunction with Departmental Management Services (DMS) Board, needs to clarify who is responsible for tracking the allocations, budget cuts and expenditures of new and A-Base funds (e.g. program managers or Corporate Finance) and what mechanisms will be used.

Management Response

It must be noted that the Meteorological Service of Canada is a much different organization now than it was in 2003. Specifically, atmospheric and climate research is now conducted under the auspices of the Science and Technology Branch. Similarly, information technology activities are carried out under the Chief Information Officer Branch (CIOB). While Weather and Environmental Services Board strategically manages the work of MSC and Science and Technology (S&T) related to weather and environmental services, the Information Technology (IT) infrastructure critical to everything WES Board does is under the purview of the Departmental Management Services Board. Therefore it will be critical to ensure that all recommendations are implemented in an integrated and horizontal fashion across organizations and Boards to ensure results.

1. Take stock and communicate which activities need to be continued in support of the modernization and sustainability of Environment Canada’s weather services

WES Board agrees with the recommendation. The ultimate outcome of a more sustainable organization and infrastructure with strengthened linkages between science, production and service remains a key goal. Significant work was done in first considering

Environment Canada vi


alternate service delivery models and ultimately in determining resource requirements to deliver the mandated responsibilities of a sustainable weather service. This was the basis for the MSC Transition. As the A-Base on which Transition was predicated has steadily eroded and as other internal and external drivers have evolved, it is timely to re-evaluate the path forward. There are a number of “stock-taking” initiatives that are already underway that are larger in scope than just the Transition activities and that will necessarily include them. Given that CIOB is critical to the delivery of Weather and Environmental Services (WES), it will be imperative that DMS Board is engaged in these WES Board activities:

WES Board will contribute to an Environment Canada Strategic Review by validating its mandate, conducting a full review of the costs associated with its programs, identifying lowest priorities and associated risks by Sept 2008;

Concurrently, WES Board will define a long-term strategic way forward that is aligned with the directions of society and government;

Both of these “review initiatives” will guide recommendations for adjustments to core programs and services including the Transition activities in the form of a business case by fall/winter 2008/2009 designed to ensure continued relevance in a sustainable fashion; and

MSC People-planning will underpin everything to ensure the right level of skill-sets are available to the MSC/WES Board to be a continuous learning organization into the future and to put in place the succession plans to get us there.

The following sections respond to the more detailed sub-recommendations by expanding on these themes and identifying timelines.


Management Response to Recommendation 1a)

WES Board agrees with the recommendation. The present Transition evaluation report goes a long way to identifying the status of the Transition activities. An Office of the Auditor General (OAG) evaluation of the MSC’s warning program is also expected to provide valuable insight into the continued relevance of those activities in contributing to ongoing sustainability (the first draft report is expected in summer 2008). These reports will inform the review initiatives and will result in recommendations as to which Transition activities need to continue. This will lead to the development of a business case in fall/winter 2008/2009.


Management response to Recommendation 1b)

The review initiatives will result in an understanding of the relevance and resource needs down to the program level and position us to recommend the path forward to meet our stakeholders’ and partners’ ongoing and increasing demands for environmental information and services. As above, completion is anticipated by fall/winter 2008/2009.

Environment Canada vii



Management Response to Recommendation 1c)

As a knowledge-based organization, MSC is critically dependent on its people. The MSC has developed a People Plan to address succession planning and much more. The plan comprehensively addresses the needs of the meteorological enterprise of Environment Canada to effectively integrate business and human resource planning to facilitate the attraction, development and retention of a highly effective and innovative workforce. Again, integration with similar planning at Science and Technology Branch (S&TB) and CIOB will be critical in ensuring that the WES Board is fully equipped with the skill-sets required to move forward on its strategic direction and achieve the desired results.

The MSC People Plan is based on a departmental framework built around the People Component of the Management Accountability Framework (PCMAF) developed for the public service at large. The plan is built around three key elements: Inclusive Culture, Building People Capacity and the Healthy & Respectful Workplace. Under these elements, there were 8 key areas identified and MSC Management Champions have been assigned to each one. The Management Champions have established working groups and have begun working on implementation of the tasks identified in the plan.

For succession planning in particular the MSC has already identified critical positions, instituted a coaching and mentoring program for employees interested in moving into management and instituted learning Plans for 95% of its employees. While the MSC is making progress in succession planning, there are challenges with a potential bow wave of retirements as well as the need to maintain unique, national and international subject matter experts. It is recognized that mentoring, coaching, job shadowing and developmental assignments are excellent mechanisms to mitigate this challenge, but scarce resources are impeding the MSC’s ability to move ahead as fast as it would like.

Financial resources needed to implement this people plan will be a part of the business case being developed. MSC will also review the succession action plan to establish a mechanism to better communicate our approach to stakeholders.


Management response to Recommendation 1d)

The current review initiatives as described in this response so far are focused on delivering just this: a recommendation on a strategic direction and implementation plan by fall/winter 2008/2009.

2. Communicate the strategic direction and action plan to:

a) MSC Staff and other internal stakeholders

Environment Canada viii


Management response to Recommendation 2a)

WES Board agrees with Recommendation 2a).

Communications has always been a priority for WES/MSC. Recently the MSC Forum was created as a mechanism for managers to exchange information, and communicate the regional perspective. The MSC Forum is chaired by the Assistant Deputy Minister (ADM) and includes Director Generals, Directors, Regional Directors and Planners. The forum meets every six weeks and ensures that all aspects of MSC/WES Board operations are considered at the management table. This has helped to ensure consistent messaging from management to staff throughout the MSC.

The ADM MSC (in some sites with the ADM S&T Branch) is conducting cross-country Town Hall meetings with staff in the spring 2008. The purpose of these meetings is to engage MSC, as well as S&T and CIOB staff critical to WES activities in a dialogue for the future. These meetings will focus on successes, on WES strategic directions and on people planning issues and will be conducted on an annual basis going forward.

Throughout the MSC Transition, employee surveys were conducted to assess the staff’s perceptions and identify concerns. We have recently completed the 2007 MSC Employee Survey and will be reviewing the results in the next few months as well as the next steps. In addition, we will be launching an Intranet site as approved by MSC Management to provide enhanced information such as bi-weekly ADM news items, progress on our strategic exercise, updates on our people plan and a connection with WES activities. A generic email has also been created for all staff to provide their comments and views on the future directions of WES. ([email protected])

The ADM MSC will continue to consult and communicate horizontally within Environment Canada through his Priority Management Board membership and participation.



WES Board agrees with Recommendation 2b). WES Board recognizes that the stakeholders’ concerns raised in this evaluation will need to be further explored and addressed.

We will continue to build on long-standing relationships with Other Government Departments, non-federal partners in academe, Emergency Measures Organizations and the provinces in communicating our strategic directions and action plans. The MSC National Advisory Board – key to the development of the previous MSC Transition strategy – will be re-invigorated as a forum to seek input. Board membership is being updated to include a broad range of key stakeholders. The first meeting of the new Advisory Board will be held in the fall 2008 and the focus will be the strategic review initiatives.

A proactive strategy is also being developed for the ADM MSC (WES Board delegate) to connect with other key stakeholders both internal and external to government on the

Environment Canada ix


evolution and direction-setting of WES programs. WES Board commits to the development and initiation of this strategy this spring.

3. Strengthen performance measurement monitoring and reporting at the outcome level

Management response to Recommendation 3

WES Board and DMS Board agree with the recommendation. Significant work has been carried out over the past several years in identifying a variety of performance indicators for the WES program. More recently, the WES Board Quality Management System (QMS) has been working towards registration to the ISO 9001:2000 International Standard. This standard focuses on using performance information to continuously improve the quality of our information and services to better meet the needs of clients.

We have completed Phase I of the QMS project which saw sixteen activities (Outcome Projects), including four CIOB activities, successfully ISO registered in 2007 and we are moving forward to expanding the registration to include an additional twenty one activities by the fall of 2008, essentially registering all WES Board operations. ISO registration will certainly contribute to meeting this recommendation. The QMS work will be entirely in step with the performance framework being developed in support of the Program Activity Architecture (PAA). This performance framework will identify metrics relative to all PAA levels right up to the ultimate Outcomes. The QMS will stay linked to this effort to ensure consistency with the PAA objectives and metrics. Together, these initiatives will produce an upward cascade of metrics showing the contribution of activities to intermediate results and ultimate outcomes.

WES Board commits to implementing phase two of the ISO 9001:2000 registration by fall 2008. WES Board commits to implementing a new PAA with performance measures up to the ultimate Outcome level by fall 2009. Through these activities and the lessons learned from the MSC Transition evaluation, MSC will be well-poised to propose an effective performance measurement framework for future endeavours.



WES Board and DMS Board agree with the recommendation. For the first three years of the project the project funds including carry-forward were properly tracked. A disconnect happened in the 4th year where the carry-forward was not tracked and budgeting was based on previous year’s expenditures.

Tracking funds and expenditures is a shared accountability among all management structures at Environment Canada. DMS Board in consultation with the Chief Financial Officer (CFO) will work with the WES Board to determine how the mechanism(s)

Environment Canada x


currently available within the departmental financial systems are best used to enable the priority management boards and organizations to effectively exercise their financial accountability.

Proposed Implementation Plan

Following is an outline of the implementation plan for the Management Response to the Recommendations contained in the Evaluation of the MSC Transition Project. Financial resources required for this implementation are not addressed herein. In many cases, activities will fall under the responsibility of WES Outcome Project Group (OPG) 2B3 and as such will depend on adequate resource availability.

Recommendation Task Lead Target Date1. Articulate a strategy and a path forward to sustainability

- Conduct a sustainability review

- Strategic review of programs and resource requirements

- WES Board strategic directions

- Succession planning as part of the implementation of a comprehensive People Plan

- Develop business case

ADM, MSC Sept . 2008

Early winter 2008Ongoing with regular updatesFall/winter 2008-2009

2. Communication of strategic direction to staff and external stakeholders

MSC staff and internal stakeholders(e.g., S&T Branch and CIOB)

- ADM Town Halls to engage staff in the vision exercise

- Email for staff feedback - Establish MSC Intranet site

External stakeholders- ADM Engagement Strategy - MSC Advisory Board

Meeting and regular schedule established

ADM, MSC

April - June 2008

AnnuallyOngoing with regular updates April-June 2008

Fall 20083. Strengthen performance measurement, monitoring and reporting at outcome level

QMS and phase two ISO Certification

PAA review and identification of performance measures at Outcome levels

ADM, MSC

Fall 2008

Fall 2009


Departmental Management Services Board to clarify process for tracking allocations, budget cuts and expenditures of new and A-based funds and mechanisms.

ADM, Finance and Corporate Branch

Schedule to be determined by DMS Board by Fall 2008

Environment Canada xi


1.0 INTRODUCTION

In 2003, the Government of Canada announced increased funding to revitalize and transform the operations of the Meteorological Service of Canada (MSC). The objectives of this project, known as the Transition Project, were to help the MSC become a more sustainable organization, to advance science- and weather-related services, and to help Canadians adapt in ways that safeguard security, health and safety, economic prosperity and environmental quality. An evaluation of the progress made to modernize Environment Canada’s weather services over the 2003–2004 to 2007–2008 period was required by Treasury Board. The importance of evaluating this significant and transformative measure was also endorsed by Environment Canada’s Audit and Evaluation Committee.3 The conduct of an evaluation was included in the department’s 2007–2008 Audit and Evaluation Plan, which was approved by Environment Canada’s Departmental Audit and Evaluation Committee on April 18, 2007.

An evaluation committee was created to facilitate and guide the evaluation process from start to finish. Members included representatives appointed by the Assistant Deputy Minister of the MSC and the Audit and Evaluation Branch. This evaluation followed the terms of reference set out in the evaluation plan approved in September 2007 by the evaluation committee4 and this document is the result.

This document presents the findings and recommendations of the evaluation of the MSC Transition Project and is organized in the following way: section 2 presents the background and context for the evaluation; section 3 presents the purpose and scope of the evaluation and the approach and methodology used to conduct the evaluation; section 4 presents the evaluation’s findings; and sections 5 and 6 lay out, respectively, the conclusions and recommendations. Supporting information is presented in the annexes to the report.

3 In 2007, the Departmental Audit and Evaluation Committee was split into two components. The component that deals with evaluation is now known as the Departmental Evaluation Committee. 4 Evaluation Plan for the Evaluation of the Meteorological Service of Canada (MSC) Transition, Environment Canada, September 2007.

Environment Canada 1


2.0 BACKGROUND AND CONTEXT

2.1 Profile5

Modernization Announcement

In March 2003, the Government of Canada announced increased funding to revitalize and transform the MSC. The funding received — approximately $74 million (M) over five years and $5M per year thereafter — was intended to help the MSC to be able to continually upgrade its infrastructure, advance its science and improve its services. This modernization process was directed toward ensuring that the MSC provides the best and most useful weather information and services possible to safeguard the health and safety, security, prosperity and quality of life of Canadians. The funding was intended to allow the MSC, which has been providing weather services to Canadians since 1871, to transition to a modern, sustainable organization that continues to provide quality services to Canadians. This investment was to be combined with approximately $11M in internal reallocation over the same five-year period and approximately $2M annually thereafter to drive the Transition Project effort.

Essentially, the Transition Project funding was intended to a) modernize Environment Canada’s weather forecast operations; b) increase its capacity for outreach to citizens and clients; c) recruit and train staff; d) improve targeted products and services; and e) strengthen its capacity to monitor the environment.

Impetus for Transition

The MSC has been Canada’s weather service for over 130 years. The mandate of the MSC is to enhance public safety and informed decision making by issuing weather warnings; forecasting weather, ice and wave conditions; supporting critical weather-sensitive government services; monitoring atmospheric conditions and predicting the state of the climate; monitoring water levels; and providing scientific research for service improvement and policy advice. To meet this mandate, the MSC provides weather and hydrometric services, with a $336.7M technological infrastructure that operates 24 hours a day, 365 days a year.6

MSC, previously known as the Atmospheric Environment Service, or AES, faced resource challenges over the 12-year period leading up to the Transition Project. The

5 This section is based on the following Environment Canada sources: - The Meteorological Service of Canada’s annual reports for 2002–2003 (financial information),

2003–2004 and 2004–2005 MSC Case Study Report: Final, Delta Partners (March 31, 2004) Improving Canada’s Weather Service – Regional Considerations

(http://www.ec.gc.ca/media_archive/press/2003/030313-2_b_e.htm) MSC Transformation: Priority Area: Transformation of the MSC

(http://www.ec.gc.ca/rpp/2004/en/msc.cfm).6 The Meteorological Service of Canada Annual Report 2003–2004.



government-wide program review in the mid 1990s resulted in, for the Atmospheric Environment Service, a 30–40% cut to the budget, closure of offices and a reduction in staff complement. In addition, a 1997 Alternative Service Delivery Study identified an ongoing annual shortfall in capital and operating expenses. In December 1999, the MSC was created as a departmental service organization (DSO) within Environment Canada, replacing the Atmospheric Environment Service; however, only limited one-time funding for Program Integrity priorities was allocated. For the next several years, while the MSC made plans to “live within its means” (an exercise cast as Focusing for the Future), project and funding approval for the modernization of MSC continued to be sought. These efforts resulted in a Ministerial announcement of the Transition Project in March 2003.

Transition Purpose, Strategy and Benefits

The MSC Transition Project has five components. All five were intended to contribute to one of the Transition Project’s overall goals: a more sustainable organization and infrastructure. Many of these elements complemented, reinforced or accelerated existing A-Base activities.

1. Consolidation and Modernization of the MSC’s Forecast Operations

A mix of new investment and internal reallocation was intended to be used to consolidate the MSC’s existing 14 regional locations into five Storm Prediction Centres — Dartmouth (Nova Scotia), Montreal (Quebec), Toronto (Ontario), Edmonton (Alberta) and Vancouver (British Columbia), with a component of the Edmonton Storm Prediction Centre located in Winnipeg (Manitoba). Aviation weather services were to be consolidated into the Edmonton Storm Prediction Centre and the Montreal Storm Prediction Centre, to ensure service levels were maintained at a lower cost and, eventually, to allow for service improvements.

The establishment of focused National Research Laboratories collocated with each of the five Storm Prediction Centres was intended to enhance the transfer of scientific knowledge to forecast production operations and thus strengthen linkages between science, forecast production and service to the public. The scientific research of the laboratories was intended to benefit not only forecasters across MSC, but also research organizations within and outside of Canada.

The intent of the increased use of automation and establishment of national standards was to increase the efficiency of routine weather forecast production and to help maintain overall production effectiveness. The restructuring of forecast offices and the automation of routine tasks were intended to allow more concerted attention to the forecasting of severe weather, thus improving warnings of high-impact events that affect all Canadians, levels of government and emergency measures organizations.

2. Creation of National Service Offices and Outreach Capacity

A mix of new investment and internal reallocation was to be used to establish three National Service Offices (NSO) in Gander (Newfoundland and Labrador), Rimouski (Quebec) and Kelowna (British Columbia), plus a smaller National Services Unit in



Regina (Saskatchewan). Staff from these offices was to work with other national and regional MSC staff to better understand Canadians’ needs, to help in meeting those needs, and to enhance public understanding and use of MSC’s services by working with regional and national stakeholders and with the five National Research Laboratories and Storm Prediction Centres. The offices were also to support the MSC’s outreach activities.

The rationale for the National Service Offices was to help MSC to understand and target the specialized needs and relationships of economic sectors. The outreach network was also intended to result in better understanding and use of MSC’s products and services by all users — the general public, the media, emergency measures organizations, and so on — and thus help improve Canadians’ ability to make informed decisions to protect themselves and their property and to maximize the economic benefits of weather and climate information.

3. Restoring and Developing Key Skill Sets

A mix of new investment and internal reallocation was intended to restore and develop key skill sets across the country to address pending demographic gaps in technical and scientific areas of expertise. One of the fundamental objectives of the restructuring of operations was to ensure that sufficient resources continued to be available for the recruitment and training of both scientific and technical staff, as well as for the ongoing training and development of existing staff. This investment was intended to ensure that the MSC had the people with the necessary skills for the subsequent years, that Canadians would continue to be provided with high-quality products and services, and that the MSC was able to maintain effective staff succession as the demographic profile of the MSC’s human resource base underwent a fundamental changeover.

4. Introducing Product and Service Enhancements and Innovation

The new investment funds were intended to allow the MSC to respond to some of the key needs expressed by Canadians through MSC’s 2002 national survey, including safer winter roads, improved seasonal and extended-range forecasts, and development and provision of information needed to improve the understanding of societal vulnerabilities to severe weather and associated community responses. These product and service enhancements were to benefit a wide range of users in both the private and public sectors. The rationale was that this component would assist the public, governments and emergency measure organizations to better prepare for and adapt to high-impact weather events.

5. Invigorating the MSC’s Monitoring Capacity

The new investment was also intended to be used to invigorate the MSC’s environmental monitoring capacity through the rationalization and modernization of the MSC’s monitoring systems; the establishment of life-cycle management approaches; enhanced quality assurance and data access systems; and the enhancement and continued operation of the Aircraft Meteorological Data Relay (AMDAR) system (a commercial aircraft-borne weather sensor system).



The monitoring system was considered to be the foundation of all MSC’s products and services, and critical to the quality and improvement of all operations. Data quality control and access improvements were expected to ensure reliable, high-quality weather, climate and water quantity data, which would contribute to provincial, national and global data-sharing networks and feed scientific research in Canada and internationally.

2.2 Program Logic Model

Figure 1 is a visual representation of the MSC Transition Project that identifies the linkages between activities and intended outcomes.



Figure 1: Logic Model for the MSC Transition7

7 This logic model was developed for the Evaluation Framework for the Meteorological Service of Canada Transition, April 2005. The dollar amounts listed in the logic model under inputs refer to Treasury Board funding.



2.3 Roles and ResponsibilitiesWhen the Transition Project started in 2003, it was managed through the line structure of the MSC and was part of the Weather and Environmental Prediction Business Line. In 2004, Environment Canada began a transformation from a line-based organization to a results-based management structure. This transformation integrated common departmental functions under a new board structure and pooled and allocated departmental resources on a priority basis. The MSC Transition Project was kept as a separate Outcome Project which reported to the Weather and Environmental Services (WES) Board.8

2.4 Resources

Table 1 contains planned resource allocations for the Transition Project.9 Treasury Board Secretariat (TBS) funds and funds that were to be reallocated from existing A-Base funds are included in this table.

Table 1: Planned Resource Allocations by Transition Project Component10

SUMMARY OF INITIATIVES(Million dollars)

5-YEAR INVESTMENT ON-GOING (POST 5-YEAR) INVESTMENT

TBS

Funding

Internal Re-

allocation TotalTBS

Funding

Internal Re-

allocation Total

A. CONSOLIDATION AND MODERNIZATION OF THE MSC’S FORECAST OPERATIONS

13.068 2.500 15.568 0.000 0.500 0.500

B. CREATION OF NATIONAL SERVICE OFFICES AND OUTREACH CAPACITY

12.000 8.750 20.750 2.400 1.750 4.150

C. RESTORING AND DEVELOPING KEY SKILL SETS

11.660 0.000 11.660 0.000 0.000 0.000

D. INTRODUCING PRODUCT AND SERVICE ENHANCEMENTS AND INNOVATION

11.600 0.000 11.600 2.100 0.000 2.100

8 As of November 7, 2007, the Outcome Project number changed to 2B3G – MSC transition. Previously it was 2B2e – Focusing for the Future (MSC Transition).9 Program staff indicated that in FY 2005–2006 a decision was made to re-profile $500,000 within MSC to FY 2008–2009. 10 Evaluation Framework for the Meteorological Service of Canada Transition, April 2005.



E. INVIGORATING THE MSC’S MONITORING CAPACITY

25.472 0.000 25.472 0.500 0.000 0.500

TOTAL INVESTMENT STRATEGY 73.800 11.250 85.050 5.000 2.250 7.250

A further breakdown of resource allocations by components is provided in Annex A.



3.0 EVALUATION DESIGN

3.1 Purpose and Scope

The purpose of the evaluation was to assess the success, performance and relevance of the MSC modernization efforts (MSC Transition Project). The evaluation will be used to help the WES Board and the department’s Executive Management Committee decide on next steps and the future direction of modernization efforts.

More specifically, the objectives of the evaluation were to examine and make recommendations with respect to

1. relevance: whether the Transition Project goals were consistent with federal priorities and addressed actual needs;

2. success: the extent to which the Transition Project has met its intended outcomes;3. cost-effectiveness / alternatives: whether the most appropriate and cost-effective

means were used to achieve those outcomes; and4. design and delivery: whether the Transition Project was delivered in the best

possible way.

The evaluation covered the five-year Transition Project period from fiscal year 2003–2004 through 2007–2008.11 Prior developments, contextual factors and other issues arising in the period directly preceding the Transition Project were also considered and documented. The evaluation does not focus on initiatives not funded under the Transition Project.

3.2 Evaluation Methodology and Approach

In 2005, an evaluation framework for the MSC Transition Project was approved by the Assistant Deputy Minister, MSC. That framework included a logic model, performance measures and an evaluation strategy.

This evaluation uses the 2005 logic model and performance measures and builds upon the evaluation issues and sources of information presented in the evaluation framework. The evaluation issues, evaluation questions, statements of what should be observed, indicators, sources of information and methods of collection are presented in Annex B. A table demonstrating how the seven expenditure review questions are addressed through the evaluation questions can be found in Annex C.

The evaluation involved the use of multiple lines of evidence. These methodological approaches are described in Table 2.

11 Unless otherwise noted, data available up until February 22, 2008 were used to address coverage for the 2007–2008 fiscal year.



Table 2: Methodological Approaches

Methodology Description

Document and File Review

An in-house review and analysis of documents related to the Transition Project were conducted. Examples of documents reviewed included project files, progress and final reports, Board documents, key correspondence records, relevant data systems and analyses, and financial information. Where applicable, certain topics were identified as research pieces and were researched in more depth.

A list of key documents is presented in Annex D.

Secondary Data and Reports

Although not previously incorporated into the plan for this evaluation, this methodology was used to collect performance and financial data due to difficulties experienced in finding evidence of reporting against some activities, outputs and outcomes and resources (budgeted and expended) under the project.

The supplemental information was provided by Environment Canada interviewees either through email or through reports prepared by Environment Canada interviewees for this purpose. This information is referred to as secondary data and reports in this evaluation report.

Media Scan A media scan was conducted in order to provide further information regarding stakeholder perspectives on initiative impacts/effects. The media scan used Green InSight as a primary search tool for articles about the MSC, which were further sifted to capture only articles that pertain to Transition Project issues. As the Green InSight archives are limited, this search only included articles from January 2005 to September 2007. Articles from 2003 and 2004 were found from a Globe and Mail and Google News search. A total of 29 articles were found and analyzed.

Survey Review Four surveys completed for the MSC by independent consultants were reviewed as part of this evaluation. The surveys included two national public opinion surveys (referred to as Survey I in question matrix in Annex B)



Methodology Descriptionand two MSC Transition employee surveys (referred to as Survey II in the question matrix in Annex B). 12

The following surveys were reviewed. National Survey on Meteorological Products and

Services - 2002 13

National WES Products and Services Survey 2007 14 (draft)

Meteorological Service of Canada Transition Employee Survey 2003 15

Meteorological Service of Canada Transition Employee Survey 2004 16

Interviews In order to receive feedback from key partners17 and stakeholders, a total of 58 interviews were conducted in-house with key informants from the following categories:

Environment Canada program staff (20) Environment Canada senior management (11) Environment Canada internal partners (5) Federal partners and stakeholders (5) Non-federal partners and stakeholders (10) Provincial and municipal stakeholders (6) International partners and stakeholders (1)

The master list of interview questions is presented in Annex E. Interview guides containing questions appropriate for each stakeholder group were formed from the master list of interview questions. Some questions were only addressed to small sub-groups of specialists who, for the most part, were involved in the design and or implementation of the project throughout much of the duration of the project.

12 Note: Although an MSC Transition employee survey was also being completed for 2007, results from this survey were not available in time to be incorporated into this evaluation.13 Completed by Decima Research Inc.14 Completed by EKOS Research Associates Inc.15 Completed by Environics Research Group.16 Completed by Environics Research Group.17 In this report, the word “partner” is used in a general sense rather than in its legal sense.



Facilitated Workshops with Environment Canada Employees

Three workshops with 47 participants, representing staff and managers, were conducted to assess and contrast the perspectives of program managers and staff from various parts of the organization on topics related to strengthening linkages between production, science and service.18

The three workshops were conducted with participants in Dartmouth, Montreal and Edmonton. Seven participants in the workshop in Edmonton attended via teleconference from Winnipeg.19

Ratings for the Evaluation Issues and Questions

A summary of ratings for the evaluation issues and questions is presented in Annex F. The ratings are based on a judgement of whether the findings indicate that

The intended outcomes have been achieved;20 The intended outcomes have been achieved but based solely on subjective

evidence;21 Considerable progress has been made to meet the intended outcome, but

attention is still needed;22 or Little progress has been made to meet the intended outcome and attention is

needed on a priority basis.23

The meaning of the “too early to say” column is that while immediate outcomes may have been achieved, it is too early to see the long-term impacts resulting from the outcomes. The N/A column identifies items where a rating is not applicable.

Limitations of the Evaluation

There are four specific limitations associated with this evaluation. The evaluation addresses these limitations through reliance on the qualitative views24 of stakeholders.

18 Strengthened linkages between production, science and service is one of the ultimate outcomes identified on the logic model for the MSC Transition (See Figure 1, Section 2.2).19 Given the locations of the workshops, it was noted in the final report from the workshops that “views of program managers and staff from Storm Prediction Centres in Vancouver, Toronto and Gander are not included in this research.” (Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, p. 3–4.)20 Labelled as Ö in summary table in Annex F and as Achieved in the Findings section.21 Labelled as ~Ö in summary table in Annex F and as ~Achieved in the Findings section.22 Labelled as Progress made, attention needed in the Findings section.23 Labelled as Little progress, attention needed in the Findings section.24 “Qualitative views” refers to opinions.



1. While the project collected information at the activity level, information on how these activities/outputs contributed to the achievement of immediate and ultimate outcomes was not consistently tracked and therefore, the full performance story of the achievements of the project could not be reported.

2. Many of the activities and outputs are just moving towards the stage of showing results (e.g. the operational results of forecasting can take years to appear); hence the evaluation often had to rely on information provided in interviews and not reports.

3. Funding provided under MSC’s Transition Project complemented A-Base funding and other targeted ongoing meteorological programs. Therefore, it was difficult at times for EC employees to distinguish the impacts of the Transition Project from the impacts of other programs.

4. The financial information system does not specifically identify the amount of Treasury Board Secretariat funds received and after 2005–2006 does not distinguish between the use of Treasury Board Secretariat funds and the use of other A-Base funds. Thus, it was not possible to give a complete picture of the allocation and use of TBS and departmental funds.



4.0 FINDINGS

The findings presented in this section address the four evaluation issues and the evaluation questions corresponding with each issue (see Annex B for the list of evaluation questions grouped by issue). The overall finding for each evaluation issue is presented first, followed by the findings for the corresponding evaluation questions. Findings for evaluation questions three and five, which address the impact of changes in governance and external factors, are incorporated and discussed throughout the findings for the other evaluation questions.

4.1 Relevance

Evaluation Issue: RelevanceWas the initiative consistent with federal priorities and requirements?

Overall Finding

Yes, the objectives and intended outcomes of the MSC Transition Project were relevant to the mandate and priorities of the department and the Government of Canada.

Evaluation Issue: Relevance Indicator(s) Rating25

EQ 1: Was the Transition Project aligned with the WES Board, Environment Canada, and government priorities?

Extent to which the strategic directions documented in the Transition Project match that of the WES Board, Environment Canada, and government priorities

Achieved

EQ 1 Findings

Yes, the objectives and intended outcomes were aligned with Environment Canada’s mandate for meteorological services and departmental priorities.

o The document review demonstrated alignment with the Weather and Environmental Prediction Business Line 2004–2005 strategic outcome Help

25 As discussed in section 3.2, ratings have been assigned to many of the evaluation questions. Possible ratings include Achieved; ~Achieved; Progress made, attention needed; and Little progress made, attention needed. When it has been determined that it is too early to assign a rating, Too early to say is indicated. Not applicable has been indicated for items when a rating has been determined not to apply. See section 3.2 for further detail.



Canadians adapt to their environment in ways that safeguard their health, safety and security, optimize economic activity, and enhance environmental quality and, now under the WES Board, with the 2007–2008 strategic outcome Weather and environmental predictions and services reduce risks and contribute to the well-being of Canadians.

Several topics in the Federal Budgets reviewed relate to topics associated with the MSC Transition Project. 26 In addition, topics from recent Speech from the Throne documents also related to topics associated with the MSC Transition Project. 27

Evaluation Issue: Relevance Indicator(s) Rating

EQ 2: Did the Transition Project address the needs identified by MSC to realign their function?

Demonstration that the Transition Project did/did not serve the needs of MSC

Achieved

EQ 2 Findings

EC interviewees widely agree that the objective and five components of the Transition Project were needed and the right areas to focus on

o Component 1: Consolidation and Modernization of the MSC’s Forecast Operations: Interviewees note that the reduction of forecast centres was relevant and that it was necessary to demonstrate the efficient use of funds.

o Component 2: Creation of National Service Offices and Outreach Capacity: Interviewees are of the view that it was important for MSC to move forward with a client/service focus and that the creation of National Service Offices and outreach capacity was an appropriate step. Interviewees also note that in some cases the geographic location of National Service Offices was influenced by the government.

o Component 3: Restoration and Development of Key Skill Sets: It was noted during the interviews that recruitment of new employees was important as there was a period in the 1990s when there was virtually no recruitment and also that training and recruitment was particularly important within the context of the retirement of meteorologists and workforce demographics. Human resources documentation reviewed containing retirement eligibility and demographic-related

26 Budgets reviewed included Budget 2003, Budget 2004, Budget 2005, Budget 2006 and Budget 2007. Examples of topics include infrastructure, emergency preparedness and response, and skills and learning.27 Speeches reviewed included Speeches from the Throne 37th Parliament, 2nd session to 39th Parliament, 2nd session. Examples of topics include infrastructure, emergency management, training and skill development.



data from the early part of the Transition Project period also reinforced the need for the restoration and development of key skills.

o Component 4: Product and Service Enhancements and Innovation: The importance of this in light of changing technology was noted during interviews. An interviewee noted that in the past a lot of product development was done “off the edge of somebody’s desk.”

o Component 5: Invigorating Monitoring Capacity: Environment Canada interviewees indicated that tackling the issue of rust-out was one of the biggest problems and part of the rationale for the project. Interviewees also noted the importance of closing some sites and of moving to a life-cycle management approach to maintain the integrity of equipment.

In the final report from the workshops conducted as part of the evaluation, it is stated that “[g]enerally, participants agreed with the goals of the Transition project and were supportive of the expected outcomes. They appeared to value particularly the fact that the Transition project was an initiative, with funding, designed to change the MSC into a sustainable organization, within the constraints that it was facing in 2003.”28

While Environment Canada interviewees are of the view that the concepts and theory of the project were good, the project was not necessarily implemented as intended. Environment Canada interviewees widely agree that the change of governance and budget allocation processes in 2005 ”derailed” the Transition Project from completing the implementation of these components and the achievement of its overarching objective.

4.2 Success

This section examines both intended outcomes (at the immediate and ultimate levels) and unintended outcomes of the project. Unintended outcomes are defined as outcomes that occurred but were not anticipated at the outset of the project.

Evaluation Issue: SuccessTo what extent has the initiative met its intended outcomes?

Overall Findings

The Transition Project has made progress towards but has not completely achieved all of its intended outcomes at the immediate and long-term (ultimate) level.

28 Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation, December 6, 2007, p. 14.



Pockets of dissatisfaction are noted among Environment Canada interviewees and external stakeholders.

The ability to achieve intended outcomes is affected by organizational and departmental changes in governance and resource allocation processes, unfunded government commitments, budget reductions, new policy requirements related to the management of property, unforeseen land claims issues, and an under-estimation of the time to set up the new National Service Offices and laboratories.

Changes in departmental governance and resource allocation processes were considered by many Environment Canada interviewees to delay or weaken the implementation of the Transition Project and to fragment the services provided to outside stakeholders, contributing to the perception that MSC was becoming an unstable rather than a more sustainable organization.

Evaluation Issue: SuccessIntended Immediate Outcomes

Indicator(s) Overall Rating

EQ 4A: To what extent has the Transition Project achieved its intended immediate outcomes?

Documentation of achievement of activities/outcomes identified in the logic model

Progress made, attention needed

EQ 4A Findings – Immediate Outcomes

Component 1: Consolidation and Modernization of MSC’s Forecast Operations

Rating: Progress made, attention needed

1 a) Intended Immediate Outcome: Increased Efficiency of Forecast Operations

Consolidation of Forecast Operations

Document review and interviews show that weather forecast operations were consolidated. Seven weather centres were closed and five Storm Prediction Centres located in Dartmouth, Montreal, Toronto, Edmonton and Vancouver were established. One additional Storm Prediction Centre was introduced in Winnipeg as a satellite office of the Edmonton centre. Two aviation centres were consolidated into the Edmonton and the Montreal Storm Prediction Centres. Five national laboratories with specialized focus points were collocated with the Storm Prediction Centres. The focuses of the laboratories were marine and coastal weather (Dartmouth), severe weather forecasting (Montreal), Nowcasting (Toronto), arctic weather and hydrometeorology (Edmonton), and coastal and mountain meteorology (Vancouver).

Some internal interviewees are of the view that the consolidation of forecast operations improved efficiency by reducing the number and thus operating costs of forecast offices. Secondary data shows a reduction of over $8M in the budget



allocations for weather forecast offices. 29 While this data shows reductions in budget allocations to the production component of MSC, this reduction needs to be considered within the broader requirements and expenditures of consolidating and modernizing the offices of the Transition Project itself.

Modernization of MSC’s Forecast Operations

Interviews, secondary data and document reviews indicate that MSC’s Transition Project provided additional resources needed to implement an automated public weather forecast tool called SCRIBE. Secondary data and reports indicate a steady improvement in the accuracy of public forecasts for the five-day period.

o Over the past two years, the SCRIBE team has also worked on developing a marine SCRIBE.

Interviewees and the document review indicate that resources provided under the Transition Project enabled the implementation of SCRIBE in all five Storm Prediction Centres. Interviewees explained that automation ensured the consistency of forecast format across the regions and freed up the time of meteorologists needed to focus on high-impact weather events.

o Environment Canada program staff are of the view that continued improvements to SCRIBE (e.g. to integrate other forecasting tools such as the Air Quality Health Index and to continue work on a marine SCRIBE) are not supported by a sufficient level of resources. An Environment Canada staff member expressed concern that the level of software maintenance for basic public forecast needs has been reduced.

o While some Environment Canada interviewees point out that SCRIBE is built on a rapidly aging technology, additional secondary information indicates that the next generation of SCRIBE will build on an investment of the Transition Project in the Ninjo workstation project.30

Communications Program

Challenges were experienced in obtaining a comprehensive picture of the extent of communication relating to the Transition Project because of, for example, difficulties in locating documentation and staff turnover. Findings and examples provided are therefore based upon information found.

Document reviews, website searches and secondary data provide examples of material used to inform MSC employees, the media and the general public about the MSC’s Transition Project. Examples include:

29 Data taken from “How MSC’s Financial Situation has Evolved,” September 30, 2004 and other internal documents prepared for the WES Board.30 Ninjo is a workstation software. A workstation is the computer used by the forecaster to access, display and manipulate data.



o Bulletins, communiqués and annual reports were used to communicate the information about the MSC Transition Project to employees; and

o News releases, background information and other products were used to inform the media and the general public about the project.

1 b) Intended Immediate Outcome: Improvements in warnings and forecasts of high-impact events

Environment Canada interviewees are of the view that performance has improved over time and that these improvements were a function of the Transition Project and other A-Base programs.

Some interviewees note the absence of performance data in this area and a need to define what is meant by a “basic level of service” and what is meant by a “high-impact” event, and to ensure a common understanding across regions.

While other interviewees point to recent examples of the department’s success in issuing warnings of high-impact events (e.g. Post-Tropical Storm Noel), it is not clear to what extent improvements can be related to the Transition Project alone.

1 c) Intended Immediate Outcome: Improvements in technology transfer through the creation of five new national research and development labs to support Storm Prediction Centres

Documents and Environment Canada interviewees report that five new national research and development laboratories were created and collocated with the Storm Prediction Centres. Each lab had a different focus: Dartmouth — marine and coastal; Montreal — severe weather; Toronto — nowcasting; Edmonton — arctic weather and hydrometeorology; and Vancouver — coastal mountain meteorology.

A challenge noted was that the announcement of the national labs was not accompanied by additional funding.

With respect to improvements in technology transfer to the Storm Prediction Centres, interviewees provided little detail and explanation. Interviewees seemed to associate improvements in technology transfer to the length of time that the laboratory has been in operation.

Component 2: Creation of National Services Offices and Outreach Capacity


2 a) Intended Immediate Outcome: Improvements in services for and strengthened partnerships with clients



Overall, Environment Canada interviewees are of the view that improvements in service and strengthened partnerships with clients will only become evident in the coming years.

o Environment Canada interviewees note that the work of some National Service Offices has been encumbered by their geographic locations, the delays in start up due to the requirements for language training and budget cuts which impeded hiring.

On the other hand, some Environment Canada interviewees and documentation reviewed indicate that Warning Preparedness Meteorologists improved services and partnerships by working with Emergency Management Organizations and engaging and enhancing relationships with the media.

Interviewees from the department and other government and non-government organizations note that Environment Canada’s website has increased the availability and accessibility of data and forecasts for different government jurisdictions, media and the general public.

EC interviewees indicate that services for clients were strengthened with the establishment of the National Inquiry Response Team, which centralized and directed feedback from the public to the appropriate departmental managers.

2 b) Intended Immediate Outcome: Increased use and effectiveness of atmospheric data and services through a new outreach network

Information regarding the extent of achievement of this outcome was not found. Rather, secondary data and documentation focused on the formation of a new outreach network.

o Secondary data and reports indicate that 75% of the outreach officer positions have been staffed and this has increased client awareness of products. At the same time, Environment Canada interviewees perceive that more could have been done.

Environment Canada interviewees note confusion over the roles of outreach officers.

Component 3: Restoring and Developing Key Skill Sets

Rating: Although there is compelling subjective evidence that the program is doing well, a complete assessment cannot be done due to the lack of human resource, financial and overall performance data.

3 a) Intended Immediate Outcome: More productive workforce with the capacity to meet MSC needs



Environment Canada interviewees note that the automation of forecast predictions increased the efficiency of MSC staff by “freeing up” the time of meteorologists, thereby enabling them to concentrate on high-impact weather.

Documentation (e.g. program descriptions) shows evidence of training programs (e.g. the Meteorologist Operational Internship Program, the Meteorological Technologist Apprenticeship or Professional Training Program, and the Hydrometric Technician Apprenticeship or Professional Training Program) to increase the capacity of MSC staff. In the documentation reviewed

o The Meteorologist Operational Internship Program is noted to be an entry-level training program to teach university graduates skills and knowledge needed to become operational weather forecasters and to be the first step in the Meteorologist Occupational Training Program;

o The Meteorologist Apprenticeship or Professional Training Program is designed to address the gap between qualifications obtained by recruits prior to joining MSC and those that are required by Operational Technologists; and

o It is stated as part of the objective of the Hydrometric Technician Apprenticeship or Professional Training Program that it will ensure availability of well qualified and trained Hydrometric Technologists, and it is noted that recruitment is based on candidates meeting specified criteria, qualifications and competencies.31

These programs were intended to address demographic gaps in technical and scientific areas of expertise.

Environment Canada interviewees note the challenges related to the restoration and development of key skill sets across the country. For example, some staff were placed in bilingual positions that required additional language training.

3 b) Intended Immediate Outcome: Sustainable workforce

Environment Canada interviewees and documentation indicate ongoing recruitment in the Meteorology Group (MT) and Engineering and Support Staff (EG) categories, although it is not evident the extent to which the recruitment is linked to the Transition Project as compared with other MSC programs.

Internal and external interviewees perceive that current succession planning is not enough to address the impacts of an aging workforce. Environment Canada interviewees are of the view that it is hard to get a sustainable workforce with financial pressures.32

Component 4: Introducing Products, Service Enhancements and Innovation31 Details provided regarding the programs are based on information contained within the program description documents obtained from the MSC.32 For more information on financial pressures, see the findings for EQ 7.




4 a) Intended Immediate Outcome: Improved extended range and seasonal forecasts

Three- to Five-Day Forecasts

Interviews, secondary data and reports indicate a steady trend of improvement in the quality of the three- to five-day forecasts over the years.

o It is noted that this improvement cannot be attributed to the Transition Project alone.

Extended Range Forecasts

Environment Canada interviews, secondary data and reports indicate that the extended range forecast, which was to extend the prediction capability to day six and seven, is expected to be achieved in 2008. No verification of the accuracy of this product is available yet.

Seasonal Forecasts

Interviews, secondary data and reports indicate that the Transition Project allowed for the development and implementation of an improved seasonal forecast system and that the new system became operational in December 2007. It is reported that the new system shows an improvement over the previous systems.

North American Ensemble Forecast System

Interviews, secondary data and reports indicate that the Transition Project enabled the North American Ensemble Forecast System to be put into operation. The verification system is under development.

4 b) Intended Immediate Outcome: Improved winter road safety weather services

Documentation reviewed indicates that the Road Weather Information System (RWIS) has been operational since February 2006. An Environment Canada staff member reports that as of the end of FY 2007–2008, Environment Canada has data-sharing agreements with British Columbia, Alberta, Manitoba, New Brunswick, Nova Scotia, Prince Edward Island, Newfoundland and Labrador, and Yukon.

4 c) Intended Immediate Outcome: Enhanced scientific analysis of high-impact vulnerabilities and adaptations

Environment Canada interviewees and documents reviewed33 indicate that progress has been made towards focusing on high-impact weather and towards

33 E.g. MSC Year 3 Transition – Year End Report; www.hazards.ca; www.criacc.qc.ca



making this information available to the public and other governments. The intent was to bring together information related to hazards from different science sources and provide it through a website format. The hazards websites in Ontario and Quebec are cited as “success stories,” however, interviewees noted that sites for other provinces were not yet at the same level.

In discussing the achievement of this outcome for the Ontario site, it was noted in interviews that the Government of Ontario had passed a Bill whereby every municipality had to assess hazards and risk in their community. The website information that was put together was able to be used for this requirement, and Environment Canada interviewees are of the view that this information has been integrated into the province of Ontario’s emergency preparedness program.

Component 5: Invigorating the MSC’s Monitoring Capacity


5 a) Intended Immediate Outcome: Modernized monitoring networks through network rationalization

Environment Canada interviewees are of the view that some progress has been made in modernizing networks. o For example, Environment Canada interviewees indicate that Transition

Project funding resulted in the purchase of new equipment for 75 Reference Climate Stations (100% achieved of intended outcome).

The rationalization (or reduction) of networks, meaning the closure of stations, proved to be a sensitive issue at the community level. Environment Canada interviewees indicate that in the past they have used a huge volunteer network and, even though there are now more efficient uses for the funds, it is hard to close down networks, especially when they are filled with dedicated people who have been keeping records for years. The consequence of not closing networks is that they cost money to keep.

o A review of secondary data and reports34 shows the extent to which the closure of networks has been achieved:

Removal of equipment and remediation of surface weather stations (38% achieved of intended outcome);

Removal, disposal of observing facilities on ships (67% achieved of intended outcome);

Decommission and cleanup of hydrometric sites (86% achieved of intended outcome); and

Reduction of the volunteer network (network reduced from 1600 to 846 stations).35

External factors have delayed the implementation of some activities. For example:

34 Based on information received on January 15, 2008.35 The number of volunteer networks was increased from the initial intended outcomes. Thus, the achievement is not calculated as a percentage.



o land claims negotiations have delayed the clean up and remediation of the Mould Bay and Isachsen sites; and

o new procedural requirements for the management of property resulted in delays to the implementation of monitoring projects.

5 b) Intended Immediate Outcome: Enhanced quality assurance of and access to key atmospheric, water, ice and air quality data

Environment Canada interviewees report that the development of the data management framework has been slower than expected. The data management framework is a project to consolidate a number of data acquisition and processing flows for the MSC's operational network. An Environment Canada interviewee noted that progress relating to the data management framework has been delayed by substantial reductions to A-Base resources and hence the inability to staff some positions. Another Environment Canada staff member noted that an automatic quality control intended to be a component of the data management framework has not yet been developed and is behind schedule. Environment Canada interviewees and federal partners and stakeholders are of the view that this is a serious shortcoming for the department and other stakeholders. As one interviewee explains, “having a sustainable way to manage our data is key to providing warnings to Canadians.”

At the same time, an Environment Canada staff member notes improvements in access to data for the public. Examples of improvements in access to information include the MSC's weather office and climate and water quantity (stream flow and level) websites.

5 c) Intended Immediate Outcome: Improved measurement of the upper atmosphere conditions via Aircraft Meteorological Data relay (AMDAR) to enable improved forecasts

An Environment Canada staff member reports that Aircraft Meteorological Data Relay (AMDAR) systems were implemented on eight aircraft operating in remote locations. The older avionics of these aircraft, however, did not allow the AMDAR systems to generate data meeting international AMDAR data quality standards. It was emphasized by this staff member that work done as part of the Transition Project has advanced the overall file on measuring upper atmosphere conditions, and other avenues to move forward with the file are being examined.

Evaluation Issue: SuccessIntended Ultimate Outcomes

Indicator(s) Rating

EQ 4B: To what extent has the Transition Project achieved its ultimate outcomes?





EQ 4 B Ultimate Outcome Findings: Success

Findings for the achievements of ultimate outcomes are mostly based on interviews with stakeholders and workshops conducted with Environment Canada employees.

Ultimate Outcome 1: A more sustainable organization and infrastructure

Rating: Little progress, priority attention required

Overall Finding

The majority of interviewees, both internal and external to Environment Canada, are of the view that MSC has not become a more sustainable organization.

Detailed Findings

Environment Canada Employees: The majority of Environment Canada interviewees are of the view that funding provided by the Transition Project has not made the meteorological science, production, services and infrastructure more sustainable. Although some elements are in place such as a life-cycle approach to managing equipment, replacement of some equipment in the Reference Climate Stations and the automation of forecasts, interviewees are of the view that budget and human resource cutbacks are the biggest barriers to the stabilization of services.

Federal Partners and Stakeholders: Federal partners and stakeholders have mixed views as to whether MSC has become a more sustainable organization over the last five years.

Provincial and Municipal Stakeholders: Provincial and municipal stakeholders had few direct comments to offer with respect to the sustainability of MSC and infrastructure. Changes in personnel and the perception that the MSC had become more centralized were noted by interviewees.

Non-Federal Partners and Stakeholders: Non-federal partners and stakeholders consider that Environment Canada has moved away from being a sustainable organization and infrastructure in recent years. Some non-federal interviewees perceive that MSC has become fragmented from an organizational perspective and no longer exists as a unified service. Also, non-federal interviewees perceive that succession planning has not been adequately addressed by MSC.

Ultimate Outcome 2: Strengthened linkages between production, science and service


Overall Finding



Stakeholders are of the view that some progress has been made towards strengthening linkages, but more progress is needed to achieve this outcome.

Detailed Findings

Environment Canada Employees: Many Environment Canada interviewees consider that the building blocks (e.g. national laboratories, research support desks) are in place but that the linkages between production, science and service have not been strengthened to the extent needed. Environment Canada interviewees express a variety of views:

o “NSOs and outreach are good ideas, but there is further to go”;o “Labs are good ideas, but they do not function at the same level”; o “Linkages between science and production [are ]uneven”; ando “Linkages between science and service [have ]deteriorated”.

Echoing the concerns, workshop participants are of the view that the department’s Transformation Process fragmented linkages rather than strengthening them and that “the Transformation goals of centralizing authority and decision-making, and standardizing policy implementation had an impact on the whole organization, including the MSC’s organizational structure, resources, supports and program priority-setting…[m]any MSC workshop participants feel powerless, having fewer resources, less direction from senior management and less decision-making authority.”36

Federal Partners and Stakeholders: Federal partners and stakeholders hold mixed views. While several of the interviewees were not able to comment on whether the MSC has shown movement in the last five years towards strengthened linkages, other interviewees pointed to the increase in the level of collaboration between production, science and service, as well as services being more direct and targeted as examples of improvements.

Provincial and Municipal Stakeholders: Limited responses were received from provincial and municipal stakeholders regarding this outcome. An interviewee commented on being impressed by MSC’s ability to connect them with people in the organization and another noted an increase in outreach.

Non-Federal Partners and Stakeholders: Non-federal partners and stakeholders point to the establishment of research labs as being a positive outcome but are of the view that critical linkages between science/production/service have been fragmented or broken. Some of the non-federal interviewees perceive that MSC’s automation of forecasts has reduced services to the public.

International Stakeholder: An international stakeholder interviewed considers that progress has been made in strengthening linkages but identifies concern with the separation of research from MSC.

36 Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, p. 5.



Ultimate Outcome 3: Improved services for Canadians and key stakeholders


Overall Finding

Improvements are noted by all stakeholders, but further improvements required.

Detailed Findings

Environment Canada Employees: Environment Canada interviewees report that improvements (e.g. high-impact weather, creation of the Canadian Atmospheric Hazards Network and information on MSC websites) have increased the availability and accessibility of information. Data indicate that the usage of MSC websites, specifically weatheroffice.ec.gc.ca and climate.weatheroffice.ec.gc.ca, has increased substantially over the Transition Project period.

o The number of sessions recorded on the Weatheroffice website has increased from approximately 12 million in October 2004 to over 33 million in October 2007. With respect to specialized users, media groups have had free access to a dedicated website for media that allows them to tailor information to meet their individual needs, although the initiation of this website predates the Transition Project.

o An Environment Canada interviewee identifies two primary challenges to improving services: 1) new products cannot be developed if resources are insufficient and 2) new products cannot be developed if other products are not dropped.

Federal Partners and Stakeholders: Federal partners and stakeholders consider

that progress has been made and more progress is needed to meet this outcome.

o Examples of improvements cited include the creation of a National Service Unit agricultural office in Regina, the Road Weather Information System, an increased focus on the needs of clients and the standardization of marine weather products.

On the other hand, federal partners and stakeholders are concerned that Environment Canada resources may be insufficient to keep up with science and production needs.

Provincial and Municipal Stakeholders: Provincial and municipal interviewees identify that improvements to service have been made.

o For example, a provincial interviewee considers that in recent years MSC has given a greater priority to dealing with clients, stakeholders and the public. “Before that, EC was more insulated (likely linked to what was happening in



the 90’s with reorganization/downsizing); it was not a very good period in terms of partnering. This is driven by the fact that we are looking at climate change scenarios in the country as a whole and need governments to be working together around those issues.”

Non-Federal Partners and Stakeholders: Non-federal partners and stakeholders note that the increased availability of data has improved services to stakeholders and impacted their research. However, interviewees also note concerns regarding service provision (due to staffing issues) and high prices charged to clients.

Ultimate Outcome 4: Adaptation of Canadians in ways that safeguard security, health and safety, economic prosperity, and environmental quality

Rating: Not applicable

Overall Finding

Information available at the time of the evaluation indicates that, for the most part, it is too early to tell whether the Transition Project has helped Canadians to protect their well-being, economic prosperity and environmental quality. Moreover, a change in the behaviour of Canadians would be difficult to attribute to the Transition Project and Environment Canada alone (e.g. the Road Weather Information System is not solely an MSC Transition Project and Environment Canada responsibility).

Evaluation Issue: SuccessUnintended outcomes

Indicator(s) Rating

EQ 6: Were there any unintended (positive or negative) outcomes? Were any actions taken as a result?

Presence/absence of unintended outcomes

Where appropriate, documented management actions and/or learning from unintended outcomes

Not applicable

EQ 6 Findings

Few unintended (positive or negative) outcomes were identified. A notable example of a positive unintended outcome arising from the project is the creation of the National Inquiry Response Team, which used staff from the National Service Offices. The team tracks telephone and email inquiries from stakeholders and the public and directs them to the responsible manager for action, thus coordinating the response process. An interviewee noted that the creation of this team was not part of an intended outcome of the Transition Project, rather it was a spin-off.



4.3 Cost-Effectiveness/Alternatives

Evaluation Issue: Cost Effectiveness/AlternativesWere the most appropriate and cost-effective means used to achieve outcomes?

Overall Findings

Although there is compelling subjective evidence that the project provided value for the money received, a complete assessment could not be done due to the lack of financial information.

While there are no obvious benchmarks to learn from, Environment Canada employees widely agree that the project used the most appropriate and effective means of achieving intended outcomes.

Evaluation Issue: Cost-Effectiveness/Alternatives

Indicator(s) Rating

EQ 7: Has the Transition Project provided value for federal dollars spent?

Extent to which Transition Project budget/ expenditures were appropriate in consideration of the stated objectives

Identified suggestions for improvement which made the Transition Project more cost-effective

Financial breakdowns for each component of the Transition Project (intended versus actual)

Resources leveraged

~Achieved37

EQ 7 Findings

There is a widespread perception among departmental staff and federal partners that the MSC Transition Project provided good value for money given the circumstances surrounding the project. Overall, Environment Canada employees agreed that the activities were the most appropriate way to achieve the desired outcomes.

37 ~Achieved: Although there is compelling subjective evidence that the program is doing well in the given issue area, a complete assessment cannot be done due to lack of performance and financial data.



Environment Canada interviewees noted several external factors that affected the achievement of outcomes:

o The ability to achieve outcomes was affected by department-wide changes to governance and resource allocation processes. Budget decisions became the responsibility of Boards, not line-based organizations, and resources were centralized and allocated on a priority basis. Environment Canada interviewees note that changes in resource-allocation processes and departmental cutbacks meant that the monitoring component did not always have access to its A-Base and capital resources.

o Government decisions were not matched with funds (e.g. the cost of the offices in Gander, Newfoundland and Labrador, and Winnipeg, Manitoba) and resources had to be re-directed from Environment Canada’s A-Base programs.

o The implementation of activities was delayed by policy requirements (e.g. land claims negotiations) and new procedures for the management of property.

Environment Canada employees and provincial and international stakeholders cite examples as to how the Transition Project leveraged other resources. Internally, Environment Canada employees indicate that many of the initiatives of the Project accelerated work on existing MSC initiatives. Externally, the project leveraged resources provided by Transport Canada in the Road Weather Information System and in the implementation of the North American Ensemble Forecast System.

Financial breakdowns for each component are presented in Annex G. While there is widespread agreement that the department received good value for the money, this perception could not be substantiated with financial information provided to date. Even though Environment Canada interviewees put extensive efforts into accounting for the use of funds, 17% of the Treasury Board Secretariat funding allocation has not been accounted for. Environment Canada interviewees attribute this difficulty to: changes made to departmental resource allocation processes post 200438; the inconsistency in coding expenditures 39 and transfer of Transition Project funds to other Boards.40 As of the writing of this report, the use of around $5.36M of Treasury Board funds cannot be tracked to expenditures made under the Transition Project.41 Further, $7.3M allocated for the decommissioning of Mould Bay and Isachsen could

38 Allocation of resources was based on priority requirements. 39 The fact that managers can code the use of funds differently means that errors in coding may have occurred. As such, researchers were only able to derive a rough approximation of the total value.40 For example, funds for core support services were transferred to the Departmental Management Services Board, but the use of these funds was not necessarily linked back to the Transition Project itself.41 $73.8M was received from Treasury Board. Of this amount: $56,260,300 was spent between FY 2003–2004 and FY 2006–2007 on program activities (includes $1,189,000 allocated for Accommodations expense for five years); ~$5.36M cannot be accounted for.



not be accounted for.42 The total of these two figures amount to 17% of the funds allocated by Treasury Board for the Transition Project.

Evaluation Issue: Cost-Effectiveness/Alternatives

Indicator(s) Rating

EQ 8: Were there alternative ways of achieving the stated outcomes of Transition Project?

Analysis of various delivery options/ approaches

Not applicable

EQ 8 Findings

At a general level, Environment Canada interviewees note that the strategy was appropriate, although some did feel that there should have been more structure than what was undertaken.

While examples from other countries show alternative ways of structuring and governing the delivery of meteorological services, no clear evidence was found that such alternative means would be more effective in achieving the Transition Project’s intended objectives, given Canada’s unique geography, population size, type of governance, and budget.

Although the governance and organizational structure of other countries’ meteorological services were examined, no formal benchmarking study was initially produced to inform and justify the Transition Project’s design and no documentation was found to explain the rationale for the Transition Project design choices.

4.4 Design and Delivery

Evaluation Issue: Design and DeliveryWas the initiative delivered in the best possible way; were these the correct set of activities/outputs to achieve goals?

Overall Findings

For the most part, the concept and design of the project were appropriate to achieving the intended outcomes, although it was not delivered entirely as planned.

o Some activities were dropped and some have not been completed to date, notably in the monitoring component.

o Pockets of dissatisfaction with the project exist among Environment Canada interviewees and some stakeholders. Some Environment Canada

42 The decision was made to delay these two projects (they have subsequently become the responsibility of Assets Management under Contaminated Sites Management).



interviewees are concerned about the potential to achieve the outcomes sought, and some stakeholders express concern over the delivery and quality of services.

o MSC underestimated some of the funding requirements and resources were re-allocated within the project.

o As a result of the inability to access all financial information, the evaluation was not able to confirm whether the project remained within budget.

Evaluation Issue: Design and Delivery

Indicator(s) Rating

EQ 9: Was the Transition Project delivered as designed (completed within budget/timeline)?

Documentation that the Transition Project was delivered as designed


EQ 9 Findings

Environment Canada interviewees and workshop participants43 generally agree that the concept and design of the project were appropriate to achieve the desired outcomes. Some interviewees noted, however, that certain sub-components were problematic.

o Environment Canada interviewees, secondary data and reports indicate that some activities were only partially completed (e.g. decommissioning of sites and the development of quality assurance programs).

o Environment Canada interviewees and documents indicate that some monitoring activities were dropped (e.g., the water vapour data acquisition program) or delayed (e.g. Mould Bay and Isachsen sites).

o Resources had to be re-allocated, and some funds were not used.

o Resource allocations for each component for the five-year period of the Transition are presented in Annex A. Detailed financial analyses (i.e. budgeted vs. actual expenditures) for each component are presented below.

Component 1: Consolidation and Modernization of MSC’s Forecast Operations

Expenditures amounted to approximately $20.1M for the first four years of the Transition Project, whereas the original five-year budget for this component amounted to $13.068M (53.8% actual over budget).

43 Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, December 6, 2007.



Consolidation and modernization of the MSC’s forecast operations ran significantly over budget (the costs to complete the automation and consolidation activities of this component of the Transition Project were under-estimated).

Specifically, the sub-component “Consolidation of Weather Forecast Production Capabilities” has run approximately 61% over budget, and the sub-component “Automated Routine Forecast Production” has run approximately 72% over budget (figures are based on original five-year budget allocated to these two sub-components).

Resources were re-allocated from Component 3: Restoring and Developing Key Skill Sets.

Only about 16% of Transition Project funds allocated for communications have been spent thus far.

Component 2: Creation of National Services Offices and Outreach Capacity

Of the $12M allocated by Treasury Board Secretariat to this component, 70.7% ($8.5M) have been spent thus far.

Although all three National Service Offices are essentially staffed and operational, there are some current vacancies due to recent retirements, people moving on to other positions, etc. which could account for part of the difference between budgeted and actual (during the period between FY 2003–2004 and 2006–2007, $9.6M was budgeted for this component; only approximately $8.5M was actually spent).

Component 3: Restoring and Developing Key Skill Sets

$3.192M was spent in the first three FYs of the project out of a total budget of $11.66M.

Essentially all of the remaining resources were re-directed in FY 2004–2005 to automation and consolidation (as part of the Consolidation and Modernization of the MSC’s Forecast Operations component) as these activities ran well over budget.

However, starting in 2004–2005 through the MSC Focusing for the Future initiative, funding for recruitment was built into A-Base funding and added to resources for managers.

Component 4: Introducing Products and Service Enhancements and Innovation

$11.6M had been allocated for this component by Treasury Board Secretariat. Thus far, in the four years between FY 2003–2004 and 2006–2007, approximately $8.3M (71.5%) has been spent.



Only 67.1% ($737,600) had been spent thus far compared to the overall amount of funding ($1,100,000) allocated to the sub-component “High Impact Weather and Climate.”

Actual expenditures for “Extended Range and Seasonal Forecasts” were on target with budgeted amounts, while 73.3% of funds allocated for the Road Weather Information System between FY 2003–2004 and 2006–2007 were spent.

Component 5: Invigorating the MSC’s Monitoring Capacity

Overall, $15M (58.9%) has been spent of the $25M allocated to this component by Treasury Board Secretariat.

With regard to the sub-component, “Rationalizing the Monitoring System,” $17.11M had been allocated over the five-year period of the Transition Project. Of this amount $8M (of which $700,000 has been spent thus far) had been allocated to two specific projects — the decommissioning of Mould Bay and Isachsen. As a result of aboriginal land claims that came to light with regards to these two sites in 2004–2005 and 2005–2006, a decision was made to delay these two projects (they have subsequently become the responsibility of Assets Management under Contaminated Sites Management). Therefore, of the remaining $9.11M, $8.925M has been spent (approximately 98%).

o The $7.3M allocated for Mould Bay/Isachsen cannot be accounted for.

About 81% has been spent to date on Assuring Data Quality and Access, which includes the development of a life-cycle management support system and climate and water data archive capacity.

Only 32% has been spent thus far in comparison to budgeted amounts up to FY 2006–2007 on Aircraft Meteorological Data Relay (AMDAR) system.


Indicator(s) Rating

EQ 10: Was performance data collected against program activities/ outcomes? If so, was information used to inform decision makers?

Extent to which performance measurement system has been implemented and is used to manage and adjust

Little progress, attention needed

EQ 10 Findings

Detailed performance information at the activity level exists and is used particularly at a managerial level to improve products and services. Internal interviews and the document review indicate a wide variety of information.



o Examples include: the number of hits on MSC’s websites are tracked; the National Inquiry Response Team collects public feedback and provides it to the Outcome Project lead for action; product surveys, human resources surveys and public opinion surveys have been conducted; verification systems are in place or are being developed to measure accuracy and timeliness of forecasting models and forecasts; and more recently, aspects of the Transition Project have received ISO 9000 certification.

Although performance measures and a performance measurement strategy were approved by the Assistant Deputy Minister of MSC in 2005, performance information is rarely linked to outcomes at the immediate and ultimate outcome levels. Further, it is difficult to track/trace resource budgets versus expenditures particularly after 2005. This means that the department is unable to track up to 17% of the funds allocated by Treasury Board Secretariat.

In the past, ad hoc reports were prepared in response to requests of management committees or boards. Now with ISO certification, management reviews are being conducted and the results are presented to the WES Board on a regular basis.


Indicator(s) Rating

EQ 11: To what extent are various stakeholders satisfied with the Transition Project?44

Reported satisfaction among stakeholders


Overall Findings

Environment Canada staff voice satisfaction with specific Transition Project aspects, as well as concern with the ability or capacity to achieve outcomes of the project. The external stakeholders were generally satisfied with the relationships but express concern about the ability of MSC to provide data that stakeholders rely on, about the lack of media access to a forecaster at the local level, and about the turnover of personnel and perceived lack of succession planning. The general public appears to be satisfied with MSC's services except in local situations where regional offices were closed down.

Detailed Findings

Environment Canada Employees

Environment Canada employees initially supported the vision, although there was some dissatisfaction with management’s handling of the Transition Project. Although some satisfaction with specific Transition Project aspects was expressed,

44 External stakeholders did not know specifically about the Transition Project. In order to get their input, external interviewees were asked questions about 1) elements of the Transition Project that related to their area of work and 2) their relationship with Environment Canada over the duration of the Transition Project period (2003–2004 to 2007–2008).



Environment Canada program staff voice concern over the ability to achieve outcomes and the need for management direction.45

o The results of the 2003 and 2004 Transition Employees Surveys indicate that MSC employees initially supported the vision, although there was some dissatisfaction with management’s handling of the Transition Project.

o While the results of the 2007 Transition Employee Survey have not been

released to date, interviews conducted under this evaluation show that MSC and internal partners46 are generally satisfied with management’s handling of the Transition Project but are of the view that management could have done a better job of handling priorities to make sure that goals were met.

o Environment Canada program staff interviewees expressed the view that Human Resources’ handling of the Transition Project was very good.

o Some Environment Canada program staff interviewed were satisfied with the training provided while some staff expressed concerns. As one Environment Canada interviewee noted, “while [MSC] tried there, I think we’ve failed in the sense that I don’t think we had the capacity to allow staff, particularly forecast staff, to take advantage of what we promised…to allow 20% of [the staff’s] time to develop and improve scientific skills.”

Mixed responses were noted by Environment Canada program staff interviewees regarding the handling of relationships between partners and stakeholders. Respondents noted that for the most part partners seem happy; however, frustration that data was not being provided to partners who rely on the data was expressed. When commenting on satisfaction with the handling of the relationship between the department and the general public, Environment Canada program staff interviewees felt that the general public was largely unaffected by or unaware of the Transition Project, except where a regional weather forecast centre was to be closed in their community.

o For example, public concern over the closure of the weather forecast office in Gander contributed to the re-opening of weather forecast operations.

Federal Partners and Stakeholders, Provincial and Municipal Stakeholders

Federal partner and stakeholder interviewees generally express satisfaction with their relationships with MSC, particularly in terms of communication, cooperation and consultation. Federal partners and stakeholders are concerned, however, that data is not as available or timely as needed.

Provincial and municipal interviewees note that new initiatives and partnerships are on the rise and are seen as improving in recent years. Increased data sharing and collaboration was also noted in the interviews.

45 Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, December 6, 2007. 46 Internal partners refer to Environment Canada staff within the Science and Technology and Environmental Stewardship branches.



Non-Federal Partners and Stakeholders

Non-federal partner and stakeholder interviewees express satisfaction with relationships created over the past five years and dissatisfaction with certain gaps in service. For example, some non-federal interviewees note that the information available on the Internet serves the general public but not specific industry needs. Comments were also made regarding a lack of media access to a live local person within the MSC for severe weather information and the charging for the provision of MSC weather information on media websites.

o Non-federal partner and stakeholder interviewees also express concern regarding turnover in personnel and are of the view that succession planning is not being addressed.

Regarding the handling of the relationship between Environment Canada and the general public, non-federal partner and stakeholder interviewees expressed the point of view that relationships are generally good; however, there were some negative comments, such as the sense of connectivity at the community level being lost due to the closing of the regional weather offices.

International Partners and Stakeholders

In an interview with an international partner and stakeholder, satisfaction with the relationship was expressed and the relationship was described as open and accommodating.

General Public

Of the 29 articles on the MSC Transition Project identified by the media scan, 17 articles can be seen as negative press on the MSC Transition Project, while 12 articles report positively on Transition Project activities. The most negative aspect of the Transition Project in the media scan is in regard to the weather station closures and the impact on MSC employees and residents living in those areas where a weather station was closed.

According to a survey, 75% of respondents are satisfied or extremely satisfied with the media website, with 59% of respondents visiting the site more than once per day.47


Indicator(s) Rating

EQ 12: What are the best practices and lessons learned from the Transition Project?

Identified learnings and best practices

Not applicable

47 MSC Media Website Survey Final Report, July 2007.



The findings related to best practices and lessons learned are based on an analysis of findings for the previous 11 evaluation questions. The results of this analysis have been grouped into best practices and lessons learned. Best practices are aspects that have contributed to the achievement of results, while lessons learned identify aspects that have detracted from the ability to demonstrate the achievement of results. Each of the best practices and lessons learned described in this section speaks to the design and delivery of the MSC Transition Project. These lessons learned could also be applicable to the design, implementation and monitoring of other projects introduced to accelerate the achievement of A-Base programs and goals.

Best Practices

A clear vision facilitates employee support for a new initiative. Employees bought into the vision for the Transition Project and continued to be supportive of the vision throughout the duration the project.

A logic model and performance measures provide a base upon which a project can be assessed. In 2005, MSC developed and approved a framework (logic model and performance measurement strategy). This framework enabled the reporting of ultimate outcomes and provided a structure by which the project could be evaluated.

Lessons Learned

Findings from the evaluation show that the roles and responsibilities of the National Service Offices and outreach officers were not clearly understood. It is important to ensure aspects and roles specific to an initiative are clearly defined and understood even if they evolve over time.

Findings also show that Environment Canada employees and external stakeholders, particularly non-federal stakeholders, did not understand the impact of changes created by the 2005 departmental transformation process on the mandate and services of MSC. As a result, these stakeholders held the view that MSC was becoming a more fragmented and less stable organization. Communication strategies play a key role in shaping stakeholders’ perceptions of an initiative.

Project managers underestimated the costs of some activities,48 did not anticipate the role and impact of internal and external factors, and were overly optimistic about the ability to achieve immediate and ultimate outcomes.

While the Transition Project was able to track the expenditures of the Transition Project by creating a separate Outcome Project under the Results Management Structure, this mechanism was not sufficient to determine the use of Treasury Board Secretariat and internally reallocated funds. Methods of tracking project specific financial data and linking it to a larger entity (e.g. ongoing A-Base programs) are essential to determine the value for money received.

The logic model for the Transition Project identified ambitious outcomes for a five-year project. Many of the outcomes of the project, particularly those related to the

48 For example, the cost of consolidating and modernizing forecast operations.



introduction of new products and services, take time to manifest. Accordingly, logic models for short-term projects need to be realistic in terms of timeframes.

In 2005, a performance measurement and an evaluation strategy were introduced through the development of an evaluation framework. Yet many of the Environment Canada interviewees were not aware of the performance measures and information requirements for an eventual evaluation. Thus, a great deal of time was taken to collect performance information for the evaluation. The performance measurement strategy that was developed needs to be implemented. A performance measurement strategy is only useful if efforts are taken to monitor the progress of the initiative against the performance measurement strategy. Methods of tracking performance data, especially when the project is part of a larger entity, are essential to determine the extent of achievements of a project.

5.0 CONCLUSIONS

In conclusion, the objectives and intended outcomes of the MSC Transition Project were relevant to Environment Canada’s mandate for meteorological services, departmental and government priorities. The Transition Project has made progress on many of the activities undertaken. Examples of achievements include: the consolidation of forecast offices; the focus on high-impact weather; movement toward an increased client/sector focus; new products and services; and the introduction of a life-cycle approach to management of infrastructure. Despite the achievements of these activities, all of the intended outcomes at the immediate and ultimate outcome levels will not be fully achieved by project-end (2007–2008). Of particular concern to both Environment Canada staff and external stakeholders is that, while progress has been made, the overall objective of making “MSC a more sustainable organization” has not been achieved to date.

At the same time, the evaluation concludes that the project used the most appropriate and effective means of achieving the intended outcomes, and that there were no obvious alternatives. Although qualitative evidence was provided on the value for the money received, a complete assessment could not be done due to a lack of performance and financial information. Finally, evidence shows pockets of dissatisfaction with the project exist among Environment Canada interviewees and some external stakeholders. Environment Canada interviewees are concerned about their ability to achieve the outcomes sought and some stakeholders express concern over the delivery and quality of Environment Canada’s weather services and the adequacy of the department’s efforts to address an aging workforce.



6.0 RECOMMENDATIONS

The following recommendations are based on the findings and conclusions of the evaluation. The first three recommendations are directed to the Weather and Environmental Services (WES) Board and the fourth recommendation is addressed both to the WES Board and the Departmental Management Services (DMS) Board.

1. Take stock and communicate which activities need to be continued in support of the modernization and sustainability of Environment Canada’s weather services.

Given the ongoing importance of modernizing Environment Canada’s weather services, attention is needed to maintain the progress and momentum achieved to date. Priority attention, in particular, is required to ensure the sustainability of Environment Canada’s weather services. In deciding which of the activities initiated under the Transition Project need to continue, the WES Board needs to


The overarching objective of the Transition Project was to help MSC become a more sustainable organization and infrastructure. The achievement of the other ultimate outcomes — strengthened linkages between production, science and service; improved services for Canadians and key stakeholders; and adaptation of Canadians to the environment in ways that safeguard security, health, the economy and quality of the environment — depend on the existence of a sustainable organization and infrastructure. Thus, in considering which activities should be continued, the WES Board needs to assess how these activities relate to the sustainability sought under the project and, further, how these activities contribute to the overall sustainability of Environment Canada’s weather services funded by existing A-Base resources.

In this context, the WES Board also needs to consider the risk to the mandates and priorities of internal stakeholders (e.g. the Environmental Protection Board) and external stakeholders, including other federal departments and agencies, other levels of government, the private sector and Canadians at large to align with what was sent for response.


The modernization efforts of the Transition Project were to be complemented by other A-Base programs; hence resource allocation should be considered within the broader context of A-Base funding for weather services. Seven million dollars annually has been allocated for the implementation of activities initiated under the Transition Project. In considering the budget allocations, the WES Board needs to consider how to best use and leverage the existing resources (financial and human) provided under the project with MSC’s other A-Base programs.




External stakeholders in particular voiced concern over the impact of an aging workforce on the capacity of Environment Canada to deliver high-quality weather services. The WES Board needs to assess the capacity of Environment Canada and the adequacy of current succession planning efforts to support sustainable and high-quality weather services to stakeholders.


Based on the information collected above, the WES Board needs to decide which activities will continue to be funded, how the resources will be allocated and linked to existing A-Base programs, and what needs to be done to ensure the capacity needed to deliver quality weather services. The strategic direction should be accompanied by an action plan which would set out specific measurable deliverables, timelines and accountabilities.

2. Communicate the strategic direction and action plan

The strategic direction and the action plan articulated by the WES Board need to be communicated to

a) MSC staff and other internal stakeholders

The vision for the Transition Project was clear and, overall, MSC staff was motivated to achieve the intended outcomes. Over time, Environment Canada interviewees became discouraged by their perceived inability to achieve the intended outcomes. Modernization of Environment Canada’s weather services is an ongoing requirement of the department, and employees need to be clear about the direction of programming. The strategic direction and the action plan formulated by the WES Board need to be communicated to the MSC, to the Science and Technology Branch and to the staff of the Chief Information Officer Branch.


Many external stakeholders are of the view that the 2004–2005 changes in the organizational structure of MSC and the turnover in personnel have moved Environment Canada away from being a sustainable organization. External stakeholders are concerned that Environment Canada does not have the capacity or succession planning efforts in place needed to create a stable organization. The changes in governance at Environment Canada, along with MSC’s efforts to address succession planning issues, need to be addressed with external stakeholders to ensure their continued support and the continued credibility of Environment Canada’s weather services.



3. Strengthen performance measurement, monitoring and reporting at the outcome level

While information on the progress of the project was collected at the activity level, information on how these activities contributed to the achievement of immediate and ultimate outcomes was not consistently collected and aggregated. Despite the existence of a performance measurement strategy, that strategy was not implemented and information at the outcome level was not readily available. The absence of performance information weakened the ability of the department to tell its full performance story as to how weather services have been modernized over the past five years and to identify ongoing requirements for programming and resources. To demonstrate good stewardship and accountability for the use of public funds and informed decision making, the WES Board needs to strengthen performance measurement, monitoring and reporting at the outcome level.


The financial information system did not specifically track the use of Treasury Board Secretariat funds from the use of other A-Base funds or indicate the impact of government and department-wide budget reductions on the Transition Project. Financial information also needs to be linked to results to enable Environment Canada to determine and report on the cost-effectiveness or value for money of programming. To ensure accountability of funds and to demonstrate good stewardship over the use of funds, the WES Board, in conjunction with Departmental Management Services (DMS) Board, needs to clarify who is responsible for tracking the allocations, budget cuts and expenditures of new and A-Base funds (e.g. program managers or Corporate Finance) and what mechanisms will be used.



7.0 MANAGEMENT RESPONSE

It must be noted that the Meteorological Service of Canada is a much different organization now than it was in 2003. Specifically, atmospheric and climate research is now conducted under the auspices of the Science and Technology Branch. Similarly, information technology activities are carried out under the Chief Information Officer Branch. While Weather and Environmental Services Board strategically manages the work of MSC and Science and Technology related to weather and environmental services, the Information Technology infrastructure critical to everything WES Board does is under the purview of the Departmental Management Services Board. Therefore it will be critical to ensure that all recommendations are implemented in an integrated and horizontal fashion across organizations and Boards to ensure results.

1. Take stock and communicate which activities need to be continued in support of the modernization and sustainability of Environment Canada’s weather services

WES Board agrees with the recommendation. The ultimate outcome of a more sustainable organization and infrastructure with strengthened linkages between science, production and service remains a key goal. Significant work was done in first considering alternate service delivery models and ultimately in determining resource requirements to deliver the mandated responsibilities of a sustainable weather service. This was the basis for the MSC Transition. As the A-Base on which Transition was predicated has steadily eroded and as other internal and external drivers have evolved, it is timely to re-evaluate the path forward.

There are a number of “stock-taking” initiatives that are already underway that are larger in scope than just the Transition activities and that will necessarily include them. Given that CIOB is critical to the delivery of Weather and Environmental Services, it will be imperative that DMS Board is engaged in these WES Board activities:

WES Board will contribute to an Environment Canada Strategic Review by validating its mandate, conducting a full review of the costs associated with its programs, identifying lowest priorities and associated risks by Sept 2008;

Concurrently, WES Board will define a long-term strategic way forward that is aligned with the directions of society and government;

Both of these “review initiatives” will guide recommendations for adjustments to core programs and services including the Transition activities in the form of a business case by fall/winter 2008/2009 designed to ensure continued relevance in a sustainable fashion; and

MSC People-planning will underpin everything to ensure the right level of skill-sets are available to the MSC/WES Board to be a continuous learning organization into the future and to put in place the succession plans to get us there.

The following sections respond to the more detailed sub-recommendations by expanding on these themes and identifying timelines.




Management Response to Recommendation 1a)

WES Board agrees with the recommendation. The present Transition evaluation report goes a long way to identifying the status of the Transition activities. An Office of the Auditor General evaluation of the MSC’s warning program is also expected to provide valuable insight into the continued relevance of those activities in contributing to ongoing sustainability (the first draft report is expected in summer 2008). These reports will inform the review initiatives and will result in recommendations as to which Transition activities need to continue. This will lead to the development of a business case by fall/winter 2008/2009.



The review initiatives will result in an understanding of the relevance and resource needs down to the program level and position us to recommend the path forward to meet our stakeholders’ and partners’ ongoing and increasing demands for environmental information and services. As above, completion is anticipated by fall/winter 2008/2009.


Management Response to Recommendation 1c)

As a knowledge-based organization, MSC is critically dependent on its people. The MSC has developed a People Plan to address succession planning and much more. The plan comprehensively addresses the needs of the meteorological enterprise of Environment Canada to effectively integrate business and human resource planning to facilitate the attraction, development and retention of a highly effective and innovative workforce. Again, integration with similar planning at Science and Technology Branch and CIOB will be critical in ensuring that the WES Board is fully equipped with the skill-sets required to move forward on its strategic direction and achieve the desired results.

The MSC People Plan is based on a departmental framework built around the People Component of the Management Accountability Framework developed for the public service at large. The plan is built around three key elements: Inclusive Culture, Building People Capacity and the Healthy and Respectful Workplace. Under these elements, there were 8 key areas identified and MSC Management Champions have been assigned to each one. The Management Champions have established working groups and have begun working on implementation of the tasks identified in the plan.

For succession planning in particular the MSC has already identified critical positions, instituted a coaching and mentoring program for employees interested in moving into management and instituted learning Plans for 95% of its employees. While the MSC is



making progress in succession planning, there are challenges with a potential bow wave of retirements as well as the need to maintain unique, national and international subject matter experts. It is recognized that mentoring, coaching, job shadowing and developmental assignments are excellent mechanisms to mitigate this challenge, but scarce resources are impeding the MSC’s ability to move ahead as fast as it would like.

Financial resources needed to implement this people plan will be a part of the business case being developed. MSC will also review the succession action plan to establish a mechanism to better communicate our approach to stakeholders.


Management response to Recommendation 1d)

The current review initiatives as described in this response so far are focused on delivering just this: a recommendation for a strategic direction and implementation plan will be developed by fall/winter 2008/2009.

2. Communicate the strategic direction and action plan to:

a) MSC Staff and other internal stakeholders

Management response to Recommendation 2a)

WES Board agrees with Recommendation 2a).

Communications has always been a priority for WES Board/MSC. Recently the MSC Forum was created as a mechanism for managers to exchange information, and communicate the regional perspective. The MSC Forum is chaired by the Assistant Deputy Minister and includes Director Generals, Directors, Regional Directors and Planners. The forum meets every six weeks and ensures that all aspects of MSC/WES Board operations are considered at the management table. This has helped to ensure consistent messaging from management to staff throughout the MSC.

The ADM, MSC (in some sites with the ADM S &T Branch) is conducting cross-country Town Hall meetings with staff in the spring 2008. The purpose of these meetings is to engage MSC, as well as S&T Branch and CIOB staff critical to WES activities in a dialogue for the future. These meetings will focus on successes, on WES strategic directions and on people planning issues and will be conducted on an annual basis going forward.

Throughout the MSC Transition, employee surveys were conducted to assess the staff’s perceptions and identify concerns. We have recently completed the 2007 MSC Employee Survey and will be reviewing the results in the next few months as well as the next steps. In addition, we will be launching an Intranet site as approved by MSC Management to provide enhanced information such as bi-weekly ADM news items, progress on our strategic exercise, updates on our people plan and a connection with



WES activities. A generic email has also been created for all staff to provide their comments and views on the future directions of WES. ([email protected])

The ADM, MSC will continue to consult and communicate horizontally within Environment Canada through his Priority Management Board membership and participation.



WES Board agrees with Recommendation 2b). WES Board recognizes that the stakeholders’ concerns raised in this evaluation will need to be further explored and addressed.

We will continue to build on long-standing relationships with Other Government Departments, non-federal partners in academe, Emergency Measures Organizations and the provinces in communicating our strategic directions and action plans. The MSC National Advisory Board – key to the development of the previous MSC Transition strategy – will be re-invigorated as a forum to seek input. Board membership is being updated to include a broad range of key stakeholders. The first meeting of the new Advisory Board will be held in the fall 2008 and the focus will be the strategic review initiatives.

A proactive strategy is also being developed for the ADM MSC (WES Board delegate) to connect with other key stakeholders both internal and external to government on the evolution and direction-setting of WES programs. WES Board commits to the development and initiation of this strategy this spring.

3. Strengthen performance measurement monitoring and reporting at the outcome level


WES Board and DMS Board agree with the recommendation. Significant work has been carried out over the past several years in identifying a variety of performance indicators for the WES program. More recently, the WES Board Quality Management System has been working towards registration to the ISO 9001:2000 International Standard. This standard focuses on using performance information to continuously improve the quality of our information and services to better meet the needs of clients.

We have completed Phase I of the Quality Management System (QMS) project which saw sixteen activities (Outcome Projects), including four CIOB activities, successfully ISO registered in 2007 and we are moving forward to expanding the registration to include an additional twenty one activities by the fall of 2008, essentially registering all WES Board operations. ISO registration will certainly contribute to meeting this recommendation.



The QMS work will be entirely in step with the performance framework being developed in support of the Program Activity Architecture (PAA). This performance framework will identify metrics relative to all PAA levels right up to the ultimate Outcomes. The QMS will stay linked to this effort to ensure consistency with the PAA objectives and metrics. Together, these initiatives will produce an upward cascade of metrics showing the contribution of activities to intermediate results and ultimate outcomes.

WES Board commits to implementing phase two of the ISO 9001:2000 registration by fall 2008. WES Board commits to implementing a new PAA with performance measures up to the ultimate Outcome level by fall 2009. Through these activities and the lessons learned from the MSC Transition evaluation, MSC will be well-poised to propose an effective performance measurement framework for future endeavours.



WES Board and DMS Board agree with the recommendation.. For the first three years of the project the project funds including carry-forward were properly tracked. A disconnect happened in the 4th year where the carry-forward was not tracked and budgeting was based on previous year’s expenditures.

Tracking funds and expenditures is a shared accountability among all management structures at Environment Canada. DMS Board in consultation with the Chief Financial Officer will work with the WES Board to determine how the mechanism(s) currently available within the departmental financial systems are best used to enable the priority management boards and organizations to effectively exercise their financial accountability.

Proposed Implementation Plan

Following is an outline of the implementation plan for the Management Response to the Recommendations contained in the Evaluation of the MSC Transition Project. Financial resources required for this implementation are not addressed herein. In many cases, activities will fall under the responsibility of WES Outcome Project Group 2B3 and as such will depend on adequate resource availability.

Recommendation Task Lead Target Date1. Articulate a strategy and a path forward to sustainability

- Conduct a sustainability review

- Strategic review of programs and resource requirements

- WES Board strategic directions

- Succession planning as part

ADM, MSC

Sept 2008

Early winter 2008

Ongoing with regular updatesFall/winter



of the implementation of a comprehensive People Plan

- Development of a business case

2008/2009

2. Communication of strategic direction to staff and external stakeholders

MSC staff and internal stakeholders(e.g. S&T and CIOB)

- ADM Town Halls to engage staff in the vision exercise

- Email for staff feedback - Establish MSC Intranet site

External stakeholders- ADM Engagement Strategy - MSC Advisory Board

Meeting and regular schedule established

ADM, MSC

April - June 2008

AnnuallyOngoing with regular updates

April-June 2008

Fall 20083. Strengthen performance measurement, monitoring and reporting at outcome level

QMS and phase 2 ISO Certification

PAA review and identification of performance measures at Outcome levels

ADM, MSC

Fall 2008

Fall 2009


Departmental Management Services (DMS) to clarify process for tracking allocations, budget cuts and expenditures of new and A-based funds and mechanisms.

ADM, Finance and Corporate Branch

Schedule to be determined by DMS Board by fall 2008



Annex APlanned Resource Allocations by Transition

Component49

SUMMARY OF INITIATIVES

(Million dollars)

5-YEAR INVESTMENT ON-GOING (POST 5-YEAR) INVESTMENT

TBS

Funding

Internal Re-allocation

TotalTBS

Funding

Internal Re-allocation

TotalA. CONSOLIDATION AND MODERNIZATION OF THE MSC’S FORECAST OPERATIONS Consolidation of Weather Forecast Production Capabilities 10.178 2.500 12.678 0.000 0.500 0.500

Automated Routine Forecast Production 1.900 0.000 1.900 0.000 0.000 0.000Communicating Program Consolidation to Clients, the Public and Staff 0.990 0.000 0.990 0.000 0.000 0.000

SUB-TOTAL A 13.068 2.500 15.568 0.000 0.500 0.500B. CREATION OF NATIONAL SERVICE OFFICES AND OUTREACH CAPACITY

SUB-TOTAL B 12.000 8.750 20.750 2.400 1.750 4.150C. RESTORING AND DEVELOPING KEY SKILL SETS

SUB-TOTAL C 11.660 0.000 11.660 0.000 0.000 0.000D. INTRODUCING PRODUCT AND SERVICE ENHANCEMENTS AND INNOVATION

Extended Range and Seasonal Forecasts 5.000 0.000 5.000 1.000 0.000 1.000

Road Weather Information System 5.500 0.000 5.500 0.600 0.000 0.600High Impact Weather and Climate – Vulnerabilities and Response Strategies 1.100 0.000 1.100 0.500 0.000 0.500

SUB-TOTAL D 11.600 0.000 11.600 2.100 0.000 2.100E. INVIGORATING THE MSC’S MONITORING CAPACITY

Rationalizing the Monitoring System 17.110 0.000 17.110 0.000 0.000 0.000

Assuring Data Quality and Access 6.962 0.000 6.962 0.000 0.000 0.000

Aircraft-Based Atmospheric Sensing System (AMDAR)

1.400 0.000 1.400 0.500 0.000 0.500

SUB-TOTAL E 25.472 0.000 25.472 0.500 0.000 0.500TOTAL INVESTMENT STRATEGY 73.800 11.250 85.050 5.000 2.250 7.250

49 Evaluation Framework for the Meteorological Service of Canada Transition, April 2005.



Annex BEvaluation Issues and Questions50

Question Statement of what should be observed

Indicator(s) Sources/Methods

Issue: RelevanceWas the initiative consistent with federal priorities and requirements?

1. Was the Transition Project aligned with the WES Board, Environment Canada, and government priorities?

Alignment with the WES Board, departmental and government priorities

Extent to which the strategic directions documented in the Transition Project matched that of the WES Board, Environment Canada, and government priorities

Outcome Project Plan/Group/Board priorities

WES Board documents Departmental

Performance Reports and Report on Plans and Priorities

Federal priorities (Speech from the Throne, budgets, policies and legislation)

Key informant interviews2. Did the

Transition Project address the needs identified by MSC to realign their function?

The Transition Project did/did not serve the needs of MSC

Demonstration that the Transition Project did/did not serve the needs of MSC

Transition reports National Survey on

Meteorological Products and Services (Survey I)

Key informant interviews Facilitated workshop

3. How has the ability of the Transition Project been affected by changes in the governance of the department?

Changes had / did not have an impact on meeting requirements of initiative

Presence / absence of positive, negative impacts

Transition reports Outcome Project

Plan/Group/Board/Business Line priorities

Treasury Board Submission

Key informant interviews

Issue: SuccessTo what extent has the initiative met its intended outcomes?

4. To what extent has the Transition Project achieved its immediate and ultimate outcomes?

Demonstrated results and outcomes and linkages between activities and outcomes



Departmental Performance Reports and Report on Plans and Priorities

Transition reports Financial reports Logic model Key Informant Interviews Survey I and MSC

Transition Employee Survey (Survey II)

5. What external Factors external to Presence or Transition reports

50 In a discussion at the WES Board meeting held February 5, 2008, members emphasized that the initiative is a project, not an ongoing program. Issue and evaluation question wording has therefore been modified from wording used in the evaluation plan to reflect this point.





factors influenced the success of the Transition Project?

the Transition Project are identified which have influenced the Transition, its implementation, and the achievement of results

absence of external influencing factors

Impacts of budget cuts

Key informant interviews Facilitated workshop Media scan

6. Were there any unintended (positive or negative) outcomes? Were any actions taken as a result?

Unintended outcomes were present that can be attributed to the Transition Project

Where appropriate, actions to address unintended outcomes were taken

Presence/absence of unintended outcomes

Where appropriate, documented management actions and/or learning from unintended outcomes

Transition reports Board documents Key informant interviews Survey II

Issue: Cost-Effectiveness / Alternatives Were the most appropriate and cost-effective means used to achieve outcomes?

7. Has the Transition Project provided value for federal dollars spent?

Transition Project demonstrated value for money (achievement of outcomes in relation to cost of delivery)

Extent to which Transition Project budget/ expenditures are appropriate in consideration of the stated objectives

Identified suggestions for improvement which made the Transition Project more cost-effective

Financial breakdowns for each component of the Transition Project (intended versus actual)

Resources leveraged

Transition financial reports

Key informant interviews Research pieces Treasury Board

Submission Analysis completed for

question 4

8. Were there alternative ways of achieving the stated outcomes of the Transition Project?

Alternative delivery methods

Analysis of various delivery options/ approaches

World Meteorological Organization standards and guidelines

Key informant interviews Facilitated workshop Review of benchmarking

documents and information





Issue: Design and Delivery Was the initiative delivered in the best possible way; were these the correct set of activities / outputs to

achieve goals?9. Was the

Transition Project delivered as designed (completed within budget / timeline)?

The Transition Project was consistent with and follows its defined approach

Documentation that the Transition Project was delivered as designed


Logic model Transition reports Key informant interviews Financial reports

10. Was performance data collected against program activities/ outcomes? If so, was information used to inform decision makers?

Performance data was collected against activities/outcomes and is used to inform decision making

Extent to which performance measurement system has been implemented and is used to manage and adjust

Review of planning documents, decisions

Review of performance information and data systems

Evaluation framework and other planning documents

Key informant interviews

11. To what extent are various stakeholders satisfied with the Transition Project?

Level of satisfaction with the Transition Project

Reported satisfaction among stakeholders

Survey I and Survey II Media scan Key informant interviews

12. What are the best practices and lessons learned from the Transition Project?



Analyses completed for Q 1 – 11

Key informant interviews Facilitated workshop



Annex CRelationship between Expenditure Review Questions and the MSC Transition Project Evaluation Questions

Expenditure Review Questions Addressed by Evaluation Question51

1. Public Interest - Does the program area or activity continue to serve the public interest?

Question 1

2. Role of Government - Is there a legitimate and necessary role for government in this program area or activity?

Questions 1, 8

3. Federalism - Is the current role of the federal government appropriate, or is the program a candidate for realignment with the provinces?

Questions 1, 8

4. Partnership - What activities or programs should or could be transferred in whole or in part to the private/voluntary sector?

Question 8

5. Value-for-Money - Are Canadians getting value for their tax dollars?

Question 7

6. Efficiency - If the program or activity continues, how could its efficiency be improved?

Questions 7, 8, 9

7. Affordability - Is the resultant package of programs and activities affordable? If not, what programs or activities would be abandoned?

Questions 7, 8, 9

51 Refer to Annex B for the evaluation issues and questions.



Annex DKey Documents

Document Title Date(if known)

Format(e.g. hard copy, electronic, website)

Government of Canada DocumentsSpeeches from the Throne 2002–2007 Electronic copiesFederal Budgets 2003–2007 http://www.fin.gc.ca/access/budinfoe.html

Environment Canada DocumentsEnvironment Canada Departmental Reports on Plans and Priorities

2003–20042004–20052005–2006 2006–20072007–2008

Electronic and hard copies (2003–2004 – 2006–2007);

Electronic copy for 2007–2008

Environment Canada Departmental Performance Reports

2003–20042004–20052005–2006 2006–2007

Electronic and hard copies

Weather and Environmental Predictions Business Line Plan 01/02 – 02/03

Nov 01 Electronic copy

Draft Weather and Environmental Predictions Business Line Plan Addendum

2004–05 Hard and electronic copies

Presentation to Executive Management Council 2004–05 September Financial Situation

Nov 04 Electronic copy

Evaluation Plan for the Meteorological Service of Canada (MSC) Transition

Sept 07 Electronic copy

Evaluation Framework for the Meteorological Service of Canada Transition

Apr 05 Electronic copy

Report on Workshops for the MSC Transition Project Evaluation: Final Report

6 Dec 07 Electronic copy

OP WorkplansContaminated Sites 2006–2007 Electronic copyLead Office 2006–2007 Electronic copyMonitoring 2006–2007 Electronic copyProduct Enhancements 2006–2007 Electronic copyRoad Weather 2006–2007 Electronic copyAircraft Meteorological Data Relay Program (AMDAR)

2006–2007 Electronic copy

Labs 2006–2007 Electronic copyLife-cycle 2006–2007 Electronic copyNational Service Office (NSO) 2006–2007 Electronic copyRelocations 2006–2007 Electronic copySurvey 2006–2007 Electronic copy

MSC DocumentsAnnual Reports 2002–2003

2003–20042005–2006

02/03: http://www.msc-smc.ec.gc.ca/media/annual_report/2002-03/chartingthecourse_e.html


http://www.msc-smc.ec.gc.ca/media/annual_report/2002-03/chartingthecourse_e.html

http://www.msc-smc.ec.gc.ca/media/annual_report/2002-03/chartingthecourse_e.html

http://www.fin.gc.ca/access/budinfoe.html


2006–2007 Hard copies (02/03 – 03/04)Electronic copies (05/06 – 06/07)

Budgets 2003–20042004–2005 2005–2006 2006–2007

Electronic copies

Mid-Year Reviews 2006–2007 Electronic copiesProject Proposals /Budget Requests 2003–2004

2004–20052005–20062006–20072007–2008

Electronic copies

Financial Pressures 14 Oct 04 Electronic copy“How MSC’s Financial Situation has Evolved: Presentation to MSC Advisory Board”

30 Sept 04 Electronic copy

“Progress Made Against Treasury Board Submission Funding”

Electronic copy

2B3g MSC Transition: Status Report to WES Board

21 Aug 07 Electronic copy

Project Reports 2002–20032003–20042004–20052005–20062006–2007

Electronic copies

Report Highlights 2002–20032003–20042004–2005

Electronic copies

Quarterly Reports 2003–20042004–20052005–2006

Electronic copies

Transition Reports 2005–2006 Electronic copiesJune Transition Report 03 Aug 04 Hard and electronic copiesMSC Case Study Report 31 Mar 04 Hard and electronic copiesADM Bulletins http://adm-msc.tor.ec.gc.ca/bulletins_e.htmlTransition Bulletins http://qww.quebec.int.ec.gc.ca/site.asp?

l=en&menu=o&dossier=/smc/transition&page=transition_00000

Atlantic Region Transition Newsletters 2003–2004 http://wwwi.ns.ec.gc.ca/internal/msc/default.asp?lang=En&n=958B5B77-1

Ontario Region Transition Progress Reports

http://infolane.on.ec.gc.ca/mscor-transition/index-e.html

Pacific and Yukon Region Priorities Plan Apr 2004 http://iws.pyr.ec.gc.ca/ECPYR_Priorities_plan_final.pdf

Prairie and Northern Region Transition Homepage

2004 http://pnrinternal.pnr.ec.gc.ca/msc/transition.asp

Improving Canada’s Weather Service: Regional Considerations

http://www.ec.gc.ca/media_archive/press/2003/030313-2_b_e.htm

MSC Transformation: Priority Area: Transformation of the MSC

http://www.ec.gc.ca/rpp/2004/en/msc.cfm

Atmospheric Environment Program Alternative Service Delivery Study Summary Report to the Steering Committee

1998 Electronic copies


http://pnrinternal.pnr.ec.gc.ca/msc/transition.asp

http://pnrinternal.pnr.ec.gc.ca/msc/transition.asp

http://iws.pyr.ec.gc.ca/ECPYR_Priorities_plan_final.pdf

http://iws.pyr.ec.gc.ca/ECPYR_Priorities_plan_final.pdf



http://wwwi.ns.ec.gc.ca/internal/msc/default.asp?lang=En&n=958B5B77-1

http://wwwi.ns.ec.gc.ca/internal/msc/default.asp?lang=En&n=958B5B77-1

http://qww.quebec.int.ec.gc.ca/site.asp?l=en&menu=o&dossier=/smc/transition&page=transition_00000



http://adm-msc.tor.ec.gc.ca/bulletins_e.html


Public Opinion Research in the Warning Preparedness Meteorologist (WPM) Program (ppt)

14 Sept 04 Hard and electronic copies

National Survey on Meteorological Products and Services – 2002: Final Report

May 2002 Electronic copy

National WES Products and Services Survey 2007: Draft Final Report

April 2007 Electronic copy

Meteorological Service of Canada Transition Employee Survey 2003: Final Report

15 Jan 04 Electronic copy

Meteorological Service of Canada Transition Employee Survey 2004: Final Report

March 05 Electronic copy

Webstats http://webstats.cmc.ec.gc.ca/index_en.htmlMonthly WXO Stats Urchin 2001–2007 Electronic copySurvey on Client Satisfaction with the MSC Media Website

July 07 Electronic copy

Activity-Specific DocumentsBrief Summary of Projects in the National Laboratory for Nowcasting and Remote Sensing Meteorology

Dec 07 Electronic copy

The Research Support Desk Initiative at the Ontario Storm Prediction Centre

9 March 05 Electronic copy

NIRT September Monthly Report 2007 Electronic copyMeteorological Technologist Apprenticeship or Professional Training Program

Dec 03 Electronic copy

Hydrometric Technician Apprenticeship or Professional Training Program

8 May 03 Electronic copy

Meteorologist Operational Internship Program Description

Electronic copy

Internship Assessment Responses (Summary)

23 Oct 07 Electronic copy

Succession planning pilot project in MSC (draft)

25 Oct 06 Electronic copy

Meteorological Service of Canada Strategy for People

Update: 23 Oct 06; file saved as: Oct 24

final

Electronic copy

MSC Transition Progress Reports for Human Resources

2003–2004 Electronic copies

EPS Publications List 02 Jun 06 http://collaboration.cmc.ec.gc.ca/science/arma/enkf/index_e.html

Canadian week-2 forecast http://www.weatheroffice.gc.ca/ensemble/semaine2_combinee_e.html

EPS Course Feedback Jun 07 Electronic copyEPS Training Material 08 Jun 07 http://collaboration.cmc.ec.gc.ca/cmc/

ensemble/Formation-Training/Read-me.html

Long-Range Forecast Skill Maps http://www.weatheroffice.gc.ca/saisons/info_pc_e.html


http://www.weatheroffice.gc.ca/saisons/info_pc_e.html

http://www.weatheroffice.gc.ca/saisons/info_pc_e.html

http://collaboration.cmc.ec.gc.ca/cmc/ensemble/Formation-Training/Read-me.html



http://www.weatheroffice.gc.ca/ensemble/semaine2_combinee_e.html

http://www.weatheroffice.gc.ca/ensemble/semaine2_combinee_e.html

http://collaboration.cmc.ec.gc.ca/science/arma/enkf/index_e.html

http://collaboration.cmc.ec.gc.ca/science/arma/enkf/index_e.html

http://webstats.cmc.ec.gc.ca/index_en.html


Users’ Guide of the Long-Range Probabilistic Forecasts

http://www.weatheroffice.gc.ca/saisons/info_prev_proba_e.html





Seasonal Forecasts based on a Statistical Model

http://www.weatheroffice.gc.ca/saisons/howto_seasonal_3-12_e.html

Long-Range Forecast Users Guide http://www.weatheroffice.gc.ca/saisons/info_prev_e.html

Seasonal forecasts based on numerical weather prediction models


Products – deterministic forecast for temperature

http://www.weatheroffice.gc.ca/saisons/image_e.html?img=sfe1t_s&title=Map

Products – probabilistic forecast for temperature

http://www.weatheroffice.gc.ca/saisons/image_e.html?img=spfe1t&title=Map

Report on Accuracy of Products V1.1 Jan 08 Electronic copy2006 Report of the CESD – Chapter 4 28 Sept 06 Electronic copyList of sources re: extended weather forecasts

15 Aug 07 Electronic copy (email)

RWIS Documents & Provincial Agreements (B.C., Yukon, P.E.I., Que.)

Electronic copies

Environment Canada’s Current and Future Contributions to Surface Transportation in Canada (ppt)

2007 Electronic copy

Road Weather Information System (RWIS) Data Sharing Agreement

20 May 04 Electronic copy

Atmospheric Hazards Canada http://www.hazards.ca/Hazards Poster Electronic copyWeb Server Statistics for Hazards Nov 07 Electronic copyData Mining – Standards Related Issues (ppt)

2007 Electronic copy

AMDAR progress report Nov 07 Electronic copyTransforming the MSC – MSC and Aviation Weather Services

Electronic copy

The Canadian AMDAR Program: A Presentation to Jazz

6 June 06 Electronic copy

Benchmarking DocumentsInternational Comparisons Apr 98 Electronic copyEC statement on the role and operation of National Meteorological and Hydrological Services (World Meteorological Organization)

Electronic copy

Other DocumentsMedia articles on MSC Transition (Sources: national and city newspapers; online articles)

2003–2007 Electronic copies


http://www.hazards.ca/







http://www.weatheroffice.gc.ca/saisons/info_prev_e.html

http://www.weatheroffice.gc.ca/saisons/info_prev_e.html




Annex EMaster Interview Guide52

Introduction:

The Audit and Evaluation Branch of Environment Canada is conducting an evaluation of the Meteorological Service of Canada (MSC) Transition. The purpose of this evaluation is to assess the relevance, success, cost-effectiveness, and design and delivery of the MSC Transition project for the five year period from 2003/4 to 2007/8.

As one component of the research, we are conducting key informant interviews in order to obtain a range of perspectives from those who are or who have been involved in this project. The questions below serve to guide this interview process.

Overview Question:

1. Briefly describe the nature of your involvement/role within the MSC Transition project. How long have you been involved?

a) What particular component(s) of the Transition have you been most involved with? The components of the MSC Transition project are:

Consolidating and modernizing MSC’s forecast operations Creating National Service Offices and outreach capacity Restoring and developing key skill sets Introducing product and service enhancements and innovation Invigorating the MSC’s monitoring capacity

b) Within this particular component, who were the primary stakeholders and beneficiaries with whom you were involved?

Prompt: MSC Staff, Canadian Public, OGDs, Private Sector, NGOs, Media, Provincial and Municipal Governments, and International Partners

2. Briefly describe your interaction with the Meteorological Service of Canada over the last 5 years.

a) Are you aware of the MSC Transition project? If yes, please describe your understanding of this project.

Prompt: The intention of the MSC Transition funding was to allow the MSC to transition to a modern, sustainable organization providing quality services to Canadians. See Annex 1 for a list of Transition components.

52 The master list was used to prepare interview guides for each of the interview groups.



Relevance Questions:

3. To what degree was/is the MSC Transition project aligned to: a) Departmental strategic outcomes?b) WES Board/WEP Table/MSC priorities? c) Government of Canada priorities?Please describe. (REF EQ1)

4. The intention of the MSC Transition funding was to allow the MSC to transition to a modern, sustainable organization providing quality services to Canadians. In order to do this 5 needs/areas were identified, these include:

Consolidating and modernizing MSC’s forecast operations Creating National Service Offices and outreach capacity Restoring and developing key skill sets Introducing product and service enhancements and innovation Invigorating the MSC’s monitoring capacity

In your opinion, to what degree has the MSC Transition project addressed this need?

Please describe. (REF: EQ2)

5. How do the Government of Canada’s meteorological research and development, products and services currently contribute to or advance:

a) The work of your organization?b) The international meteorological community?

Please explain. (REF EQ2)

6. In 2004, Environment Canada began a Transformation from a line-based organization to a results-based management structure. This Transformation integrated common departmental functions under a new Board structure. This resulted in the shift from WEP Table to WES Board.

What impacts, if any, have changes in the governance structure of the Department had on the MSC Transition project? Please describe. (REF EQ3)

Success Questions:

7. You mentioned during question 1 that you had been most involved with the following component(s) of the MSC Transition project: To what degree have the activities and outputs associated with this/these component(s) contributed to the achievement of their intended outcomes? (See Annex 1) (REF EQ4)

8. To what degree have the outcomes associated with this/these component(s) (outlined above) contributed to the achievement of the following final outcomes of the MSC Transition project?

a) A more sustainable organization and infrastructure;b) Strengthened linkages between production, science, and service;



c) Improved services for Canadians and key stakeholders;d) Adaptation of Canadians in ways that safeguard security, health and

safety, economic prosperity, and environmental quality? (REF EQ4)

9. Overall, to what degree has the MSC Transition project achieved the following final outcomes:

a) A more sustainable organization and infrastructure;b) Strengthened linkages between production, science, and service; c) Improved services for Canadians and key stakeholders;d) Adaptation of Canadians in ways that safeguard security, health and


10. In the last 5 years, to what degree has the MSC demonstrated movement toward achieving the following Transition outcomes:

a) A more sustainable organization and infrastructure;b) Strengthened linkages between production, science, and service; c) Improved services for Canadians and key stakeholders;d) Adaptation of Canadians in ways that safeguard security, health and


11. What external factors (outside of the MSC Transition project), either positive or negative, have influenced the MSC Transition project, its implementation, and/or the achievement of its outcomes?

a) What have been the impacts of these external factors?

b) Were these factors addressed? Please describe. (REF EQ5)

12. Have there been any unintended/unexpected outcomes, either positive or negative, resulting from the MSC Transition project? Please describe. (REF EQ6)

Cost-Effectiveness/Alternatives Questions:

13. To what degree do you feel the MSC Transition project provided value for federal dollars spent? Was Transition funding allocated and used appropriately in order to address MSC/WES needs? Please describe with examples. (REF EQ7)

14. Was adequate funding (Treasury Board and internal reallocation) provided to each of the five components of the MSC Transition project? Components include:

Consolidating and modernizing MSC’s forecast operations Creating National Service Offices and outreach capacity Restoring and developing key skill sets Introducing product and service enhancements and innovation Invigorating the MSC’s monitoring capacity (REF EQ7)



15. Given the component(s) you are most involved with, to what degree did the MSC Transition project use the most effective approaches/strategies to achieve its outcomes? Please describe.

a) Are there alternative approaches/strategies that would have made the MSC Transition project more effective? Please describe. (REF EQ8)

Design and Delivery Questions:

16. To what degree has the MSC Transition project been delivered as designed/intended? Please describe aspects of the Transition project that were not delivered as designed. (REF EQ9)

17. Was the MSC Transition project completed within its identified budget?

a) How were MSC Transition project priorities determined?

b) Did reallocations occur as originally planned?

c) Were additional reallocations required? If so, please describe. (REF EQ9)

18. Has performance data been collected and reported against MSC Transition project activities/outcomes? Please provide specific examples.

a) If yes, how has the information been used? (REF EQ10)

19. As an MSC employee, to what extent are you satisfied with the following aspects of the MSC Transition project?

a) Management’s handling of the Transition b) Human Resource professionals’ handling of the Transition c) Information regarding and availability of training and development

required in order to fill new roles (REF EQ11)Please describe.

20. During the MSC Transition project, how do you feel relationships with the following groups were handled?

a) Management and staffb) Management and partners/stakeholdersc) Environment Canada and the general public (REF EQ11)Please identify any issues that arose and if/how they were handled.

21. Over the last 5 years, to what extent has the MSC been successful in handling its relationships with:

a) Partners and stakeholdersb) The general public (REF EQ11)

22. What are some examples of best practices and/or lessons learned from the MSC Transition project? (REF EQ12)



Concluding Questions:

23. In your opinion, what are the key accomplishments and challenges of the MSC Transition project? (REF EQ12)

24. Over the last 5 years, what are the key accomplishments and challenges of the MSC as they relate to your organization? (REF EQ12)

25. Do you have anything additional to add?

Information from Annex 1 of Master List of Interview Questions:

1. Consolidating and Modernizing MSC’s Forecast Operationsa) Improved efficiency through consolidation of forecast operationsb) Improved warnings and forecasts of high impact eventsc) Improved technology transfer through the creation of five new National research

and development labs to support Storm Prediction Centres

2. Creating National Services Offices and Outreach Capacitya) Improved services for and strengthened partnerships with clientsb) Increased use and effectiveness of atmospheric data and services through a

new outreach network

3. Restoring and Developing Key Skill Setsa) More productive workforce with the capacity to meet MSC needsb) Sustainable workforce

4. Introducing Product and Service Enhancements and Innovationa) Improved extended range and seasonal forecastsb) Improved winter road safety weather servicesc) Enhanced scientific analysis of high-impact vulnerabilities and adaptations

5. Invigorating the MSC’s Monitoring Capacitya) Modernized monitoring networks through network rationalizationb) Enhanced quality assurance of and access to key atmospheric, water, ice and air

quality datac) Improved measurement of the upper atmosphere conditions via AMDAR to

enable improved forecasts



Annex F Summary of FindingsEvaluationquestion (EQ)

Outcome Achieved Progress made, attention needed

Little progress made, priority

attention needed

Too earlyto see

Not applicable (A rating is

not applicable)

Relevance EQ1 Ö EQ2 ÖSuccess EQ3 Ö EQ4 Immediate 1 X

Immediate 2 XImmediate 3 ~ÖImmediate 4 XImmediate 5 XUltimate 1 XUltimate 2 XUltimate 3 XUltimate 4 Ö

EQ5 Ö EQ6 ÖCost-effectiveness / alternatives EQ7 ~Ö EQ8 ÖDesign and delivery EQ9 X EQ10 X EQ11 X EQ 12 Ö

Note: ~ = Although there is compelling subjective evidence that the program is doing well in the given issue area, a complete assessment cannot be done due to lack of performance data.



Annex GBudgeted and Actual Expenditures versus Treasury

Board Funding for the First Four Years of the Transition Project (2003–2004 to 2006–2007)

Total: FY 2003-2004 to FY 2006-2007A: Consolidation and Modernization of MSC’s Forecast Operations

Sub-Component Budget Actual Difference - $ Difference - %Consolidation of Weather Forecast Production Capabilities $10,081,000.00 $16,676,100.00 ($6,595,100.00) 165.42%

Automated Routine Forecast Production $1,900,000.00 $3,266,300.00 ($1,366,300.00) 171.91%

Communicating Program Consolidation to Clients, the Public and Staff

$780,000.00 $157,400.00 $622,600.00 20.18%

Total: $12,761,000.00 $20,099,800.00 ($7,338,800.00) 157.51%B: Creation of National Service Offices and Outreach Capacity

Sub-Component Budget Actual Difference - $ Difference - %National Services Office and Outreach $9,600,000.00 $8,483,200.00 $1,116,800.00 88.37%

Total: $9,600,000.00 $8,483,200.00 $1,116,800.00 88.37%C: Restoring and Developing Key Skill Sets

Sub-Component Budget Actual Difference - $ Difference - %Restoring and Developing Key Skill Sets $11,012,000.00 $3,192,000.00 $7,820,000.00 28.99%

Total: $11,012,000.00 $3,192,000.00 $7,820,000.00 28.99%D: Introducing Product and Service Enhancements and Innovation

Sub-Component Budget Actual Difference - $ Difference - %Extended Range and Seasonal Forecasts $4,000,000.00 $4,024,100.00 ($24,100.00) 100.60%

Road Weather Information System $4,822,000.00 $3,534,400.00 $1,287,600.00 73.30%High Impact Weather and Climate – Vulnerabilities and Response Strategies

$600,000.00 $737,600.00 ($137,600.00) 122.93%

Total: $9,422,000.00 $8,296,100.00 $1,125,900.00 88.05%E: Invigorating the MSC’s Monitoring Capacity

Sub-Component Budget Actual Difference - $ Difference - %Rationalizing the Monitoring System $13,904,000.00 $9,625,100.00 $4,278,900.00 69.23%Assuring Data Quality and Access $6,299,000.00 $5,095,100.00 $1,203,900.00 80.89%Aircraft-Based Atmospheric Sensing System (AMDAR) $876,000.00 $280,000.00 $596,000.00 31.96%

Total: $21,079,000.00 $15,000,200.00 $6,078,800.00 71.16%

Note: Budgeted and actual expenditures represent Treasury Board Secretariat funds; these figures do not include $11.25M in internal re-allocations agreed to by the department as part of the submission agreement. These figures also do not include amounts for FY 2007–2008, as these were not yet available.
