gate 5 review guideline - digital.nsw
TRANSCRIPT
JUNE 2020
GATE 5 REVIEW
GUIDELINE Pre-Commissioning
GATE 5 REVIEW GUIDELINE Pre-commissioning
2
About this guideline This guideline assists review teams and delivering agencies working on Gate 5: Pre-commissioning of the ICT
Assurance Framework (IAF) Gateway review process. It should be read alongside the ‘Gate 5 Review Report’
template and ‘Guidance to Review Teams’, both available from https://www.digital.nsw. gov.au/policy/ict-assurance.
The Gateway Review process examines projects at key decision points (gates) and looks ahead to provide assurance
that projects can progress to the next stage (or gate). This can also include health checks between gates.
Gateway reviews are independent peer reviews of a project’s viability and development. Independent practitioners
from outside the project examine the progress and likelihood of successful delivery at a certain point in each project –
this provides a valuable new perspective on the project’s issues, while challenging the robustness of plans and
processes.
PAGE
3
The Gateway Review process Purpose of the review Focus areas
PAGE
10
Topics to probe within each of the four key review scope areas: 1. Business Case and stakeholders 2. Risk Management 3. Review of current phase 4. Readiness for next phase:
Post-implementation
PAGE
7
The Gateway Review Framework Conducting a Gateway Review Assessing risk in ICT Assurance Developing the report
PAGE
20
Typical project documentation for Gate 5 review
GATE 5 REVIEW GUIDELINE Pre-commissioning
3
The Gateway Review process
GATE 5 REVIEW GUIDELINE Pre-commissioning
4
Purpose of the review - Gate 5: Pre-commissioning
Gate 5: Pre-commissioning investigates whether the delivery agency is ready to adopt the solution to achieve the
benefits stated in the Business Case and manage the change required. The review also confirm that relevant
whole-of-government ICT policies, standards and priorities have been considered.
The Gate 5: Pre-commissioning review is designed to:
• Check the current phase of the contract is properly completed and documentation completed;
• Ensure contractual arrangements are up-to-date;
• Assess if the Business Case is still valid and unaffected by internal and external events or changes;
• Assess if the original projected business benefit is likely to be achieved;
• Ensure processes and procedures will achieve long-term success of the project;
• Confirm that all necessary testing is done (e.g. Commissioning, business integration and user acceptance
testing) to the delivery agency’s satisfaction and that the delivery agency is ready to approve implementation;
• Check that there are feasible and tested business contingency, continuity and/or reversion arrangements;
• Ensure all risks and issues are being managed effectively and do not threaten implementation;
• Evaluate the risk of proceeding with implementation if there are unresolved issues;
• Confirm the delivery agency is resourced and ready to implement the services and the business change;
• Confirm that the delivery agency and supplier implementation plans are achievable;
• Confirm there are controls to manage the project through implementation and operation;
• Confirm contract arrangements are in place to manage the operational phase of the contract;
• Confirm arrangements for handover from Project Sponsor to the operational business owner;
• Confirm all parties have agreed plans for training, communication, rollout, production release and support;
• Confirm all parties have agreed plans for managing risk;
• Confirm there are client-side (delivery agency) plans for managing the working relationship, with appropriate
reporting arrangements, reciprocated on the supplier side;
• Confirm information assurance accreditation/certification;
• Confirm defects or incomplete works are identified and recorded;
• Check that lessons for future projects are identified and recorded;
• Evaluate actions to implement recommendations made in any earlier assessment of deliverability; and
• Confirm relevant whole-of-government ICT policies, standards and priorities have been considered.
This guideline details topics to be assessed and the evidence the review team should expect, in four key review
scope areas:
• Business Case and stakeholders;
• Risk Management;
• Review of current phase; and
• Readiness for next phase: Post-implementation.
GATE 5 REVIEW GUIDELINE Pre-commissioning
5
These key review scope areas will help to structure the Gate 5 report.
The guideline provides examples of evidence the review team should seek. This should not be considered
prescriptive; each review team should consider if broader topics should be addressed, or different evidence required
– this will depend on the context of the project.
Focus Areas
The review team should be mindful of the seven focus areas. The seven focus areas are a set of themes common
across the project lifecycle that the NSW Government has determined as requiring assessment. They are referred
to in the key review scope areas and are used in the review report.
Focus area Description
Affordability and value for money
A clear case for change and consideration of technology and market options to show evidence that the
proposed changes will be delivered to the highest quality within an acceptable time and at a competitive and
affordable price. There must be sufficient financial, physical and human resource to deliver the project and
expenditure of these resources must provide value for money over the project’s life.
Risk Management
Risk to scope, cost, procurement, time and quality should be identified and managed, as should risks inherent
to the nature of new or changing technology, such as data privacy and cyber security risks, reputational risks
and risks to continuity or quality of business services. Risk management plans must be developed.
Governance
Consideration of project governance (roles and responsibilities to deliver the project, resource allocation, time
management and process management) and alignment with business as usual agency activities and broader
NSW Government and stakeholder governance
Stakeholder Management
Consideration of the stakeholders that may contribute to or be affected by new ICT environments and
capabilities, including end-users, government staff, citizens, business service managers and executive
owners, technology providers, and both government and external vendors and service providers.
Change Management
Consideration of how the change will affect stakeholders, expected acceptance or resistance and actions
required to move to new ways of working.
Service Delivery
Consideration of the effect of new technology capabilities on business service delivery, such as more efficient
business services; maintaining or improving service delivery, such as better access to government services;
quality improvements; or enabling new services.
Sustainability
Considerations of benefits realisation planning and tracking; service transition planning and implementation;
whether vendor management offices will be required; continuous improvement capabilities and solution road
maps; and how data will be archived or retained to meet current and future legislative requirements and data
migration requirements.
GATE 5 REVIEW GUIDELINE Pre-commissioning
6
The Gateway Review Framework
provides more details of the Gateway Review process.
Review teams should:
• Engage and meet with a Project Sponsor from the delivery agency prior to the review
• Where possible, engage early with the relevant agency’s project management office (PMO) to understand
the project’s background and to adequately plan for interviews and required documentation.
GATE 5 REVIEW GUIDELINE Pre-commissioning
NSW GOVERNMENT SENSITIVE 7
The Gateway Review Framework
GATE 5 REVIEW GUIDELINE Pre-commissioning
8
Conducting a Gateway Review
Step 1 – Initiate Step 2 – Prepare Step 3 – Conduct Step 4 – Report
WH
AT
• Register project
• Confirm risk tier and
assurance plan
• Agree review dates
• Draft and approve
terms of reference
• Nominate and agree review
team
• Draft review team
agreements
• Project documents
uploaded to SharePoint by
agency Coordinator
• Interview logistics
completed by agency
• Review team briefed by
assurance team
• Planning meeting
• Interviews held
• Daily Sponsor feedback
sessions
• End-of-review Sponsor
debrief
• Review team draft and issue
report to ICT Assurance/Sponsor
• Sponsor reviews report and
completes close-out plan and Sponsor comments
• Review team and ICT
Assurance validate Sponsor input
• Issue final report
• Issue clearance letter
• Survey completed by Sponsor
and review team
• Invoicing and charge-back
WH
O
• ISSI Working Group
• Sponsor, Project
Director / Manager (agency)
• Assurance Director,
Principal Manager, Case Officer (DCS)
• Sponsor, Project Director /
Manager, Coordinator
(agency)
• Assurance Director,
Principal Manager, Case
Officer (DCS)
• Review team
• Sponsor, Project Director /
Manager, Coordinator
(agency)
• Interviewees including
project stakeholders,
Treasury, end users, third parties
• Assurance Director,
Principal Manager, Case Officer (DCS)
• Review team
• Sponsor, Project Director /
Manager, Coordinator
(agency)
• GCIDO
• Assurance Executive Director,
Director, Principal Manager, Case Officer, Finance (DCS)
• Review team
WH
EN
Varied Up to 4 weeks 1–3 weeks 1–3 weeks
GATE 5 REVIEW GUIDELINE Pre-commissioning
9
Assessing risk in ICT Assurance
Each gate in the Gateway Review process requires the review team to assess a project’s level of risk. Before the
Gateway Process starts, each project is allocated a risk tier to quantify the level of assurance required. The risk tier
– a rating between 1 and 5, with 1 being the largest and most complex – is determined through a self-assessment
of risks and complexities which is then compared against estimated costs. The risk tier ensures there will be
sufficient assurance to larger projects and less regulation for smaller projects.
As project risks or complexities can change, each gate or health check should reassess project risk and complexity
by reviewing risk and issue logs, specifically that:
• Inherent and residual ratings are provided for all risks and issues;
• All risks and issues have action plans, with owners and dates against each action;
• Each action plan and seniority of owner reflects the significance of the risk or issue; and
• All dates must be in the future (if an action is late then a revised action plan should be documented).
If the risk tier needs to be changed or the assurance plan updated, this must be discussed with the Project Sponsor,
with any change in tier requiring Government Chief Information and Digital Officer (GCIDO) endorsement under the
terms of the IAF.
Tier classification and assessment
Developing the report
A review report is the key output of each gate. Each report must follow the report template and be written in a
concise way that a third party could understand. Commentary should be included for each section, to support
recommendations by the Review Team. Where possible, examples should be provided especially for items that
require further work and action.
The review report lists recommendations, defined as either critical, essential or recommended. These should:
• Link to project milestones;
• Follow the SMART approach (S – specific; M – measurable; A – attainable; R – realistic; T – timely); and
• Align to the seven focus areas.
Reports will remain in Microsoft Word and named as per the following file naming protocol:
Project Name – Gateway Review Name – (DRAFT / FINAL) Report_Ver 1-1
The review team leader emails all reports to the ICT Assurance Director.
GATE 5 REVIEW GUIDELINE Pre-commissioning
10
1. Business Case and stakeholders Each numbered item below is an area to be probed.
1.1 Is the project still required?
Evidence expected Status/Ref
• Where relevant, approval of changes to requirement defined at Gate 4;
• Communication with stakeholders;
• Project board endorsement of:
– Updated Business Case and benefits plans;
– Reviews of the solution against the requirement;
– Reconciliation of government and organisation objectives with those defined at Gate 4; and
• Plans for modular/incremental implementation, where required.
1.2 Does the project meet the business need?
Evidence expected Status/Ref
• Confirmation the operational service or facility (or partnering contract, where applicable) is approved by
stakeholders.
1.3 Is the Business Case valid?
Evidence expected Status/Ref
• Updated project (or program, if appropriate) plan and Business Case that justify how the implementation
will meet business need; deliver value for money; be affordable and achievable, with implementation
broken down into modules/increments where appropriate.
1.4 Do changes between award of contract and completion of transition/testing affect
plans for business change?
Evidence expected Status/Ref
• Change management documentation for impact analysis; products, design or operational changes; and
justified and approved changes;
• Updated Business Case and benefit plan; and
• Updated processes, procedures and activities.
1.5 Is the delivery agency ready for business change?
Evidence expected Status/Ref
• Agreed plans for business preparation, transition and operational phases and, where appropriate,
readiness of ICT and/or new facilities;
• Change management plan developed with stakeholders;
• Change management strategy builds understanding of stakeholders’ views, organisational and business
process implications and communication requirements;
• Communications plan;
• Informed and trained staff; and
• Defined service management function/organisation.
GATE 5 REVIEW GUIDELINE Pre-commissioning
11
1.6 Can the delivery agency implement the new services and maintain existing services?
Evidence expected Status/Ref
• Resource plan, showing capacity and capability, and that resources are available to meet commitments.
1.7 Are resources available with, where required, appropriate skills and experience?
Evidence expected Status/Ref
• Internal and external commitment to provide required resources;
• Job descriptions for key project staff;
• Skills appraisal and plans to address shortfalls; and
• Allocation of project roles between internal staff and consultants or contractors.
1.8 Have benefits identified in the Business Case changed in a way that could affect the
value of the project?
Evidence expected Status/Ref
• Updated benefits realisation strategy and benefits realisation register.
1.9 Is there a strategy to plan and manage benefits?
Evidence expected Status/Ref
• Updated benefits realisation strategy and benefits realisation register document the identified benefits
and the realisation of these benefits.
1.10 Is any remaining development focused on the minimum viable product (MVP)?
Evidence expected Status/Ref
• Control of product backlogs.
1.11 Are delegations (decisions and budget) defined?
Evidence expected Status/Ref
• Business Case or program structure document details individual and group delegations.
1.12 Does the live service operating model provide the resources to deliver the MVP?
Evidence expected Status/Ref
• Resource plan mapped against operating model, required resourcing or the MVP.
1.13 Are business users empowered to effect change if required?
Evidence expected Status/Ref
• Effectiveness of product managers to deliver change in the business.
GATE 5 REVIEW GUIDELINE Pre-commissioning
12
1.14 What is the strategy for continued development of the service or growing a portfolio
of Digital by Default services?
Evidence expected Status/Ref
• Department and program policy/strategy; and
• Business Case.
1.15 Are user and business needs reviewed and benefits tracked?
Evidence expected Status/Ref
• Feedback from business stakeholders; and
• Benefits management arrangements reflect the changing environment.
GATE 5 REVIEW GUIDELINE Pre-commissioning
13
2. Risk Management Each numbered item below is an area to be probed.
2.1 Are risks and issues identified at contract award resolved?
Evidence expected Status/Ref
• Risks satisfactorily resolved – no outstanding issues.
2.2 Are risks and issues at implementation phase being identified and managed?
Evidence expected Status/Ref
• Risks satisfactorily resolved – with remaining risks only associated with commissioning and service
delivery;
• Risks fully quantified with appropriate risk management plans in place; and
• Mitigation of risks identified during vulnerability and penetration testing.
2.3 If there are unresolved issues, what are the risks of implementing versus
delay?
Evidence expected Status/Ref
• Project risk management strategy developed in line with best practice;
• Remaining issues and risks assessed with responsibility for management defined;
• Evaluation report on the risk and impact of cancelling, delaying or proceeding with implementation
considers project outcome and wider program of change; benefits realisation; consequences for supplier,
client, business, stakeholders, users; other factors such as financial outcome, political issues, information
security and delivery;
• Options and management plans for all scenarios and a recommendation based on sensitivity analysis;
and
• Project board ratifies any recommendation to delay or proceed with implementation.
2.4 Is the budget under control? Is a higher spend burn rate required, e.g. for
developers or coders to maintain pace?
Evidence expected Status/Ref
• Examination of financial management data;
• Evidence of regular financial data linked to each Sprint cycle; and
• Reports considered by program board/steering committee.
GATE 5 REVIEW GUIDELINE Pre-commissioning
14
3. Review of current phase Each numbered item below is an area to be probed.
3.1 Does the total service or facility meet the acceptance criteria?
Evidence expected Status/Ref
• Justification and authorisation of changes to original specification; and
• Analysis of as-built/products show the solution complies with acceptance criteria.
3.2 Is the project under control? Is it running according to plan and
budget?
Evidence expected Status/Ref
• Reconciliations of cost with budget and actual schedule with planned schedule;
• Updated risk register and issue log;
• Status reports for communication and external relations activities;
• Reports on environmental performance, where applicable;
• Compliance with statutory requirements (e.g. data protection, health and safety);
• Contractual issues resolved and recorded; and
• Compliance with security standards such as information assurance.
3.3 Have all the stakeholder issues been addressed?
Evidence expected Status/Ref
• Progress reports completed and circulated as part of the communication plan for stakeholders.
3.4 Have all process testing and commissioning/acceptance (or transition) procedures
and activities been completed?
Evidence expected Status/Ref
• Commissioning/test plans show results and analysis of products against acceptance criteria and conform
to the pre-defined criteria;
• Ratified test reports and logs;
• Commissioning/testing team with relevant skills and experience;
• Confirmed end-to-end testing, including changed or new business processes;
• Testing accounts for future modules or deliverables; and
• Missing or incomplete items and agreed corrective action documented.
3.5 Have all parties accepted relevant commissioning/test results and any action
plans required?
Evidence expected Status/Ref
• Plans and procedures by supplier and delivery agency, as client, are appropriate;
• Testing methodologies and outcomes are acceptable; and
• Information is secure and appropriate security protocols and practices in place to mitigate information
security and cyber security threats.
GATE 5 REVIEW GUIDELINE Pre-commissioning
15
3.6 Are there workable and tested business contingency, continuity and/or
reversion plans for rollout, implementation and operation?
Evidence expected Status/Ref
• Documented and timetabled decision paths for key aspects (e.g. go/no go decisions on rollout) with
decision makers identified and informed;
• Where appropriate, plans cover IT components and business processes;
• Endorsement by project board and supplier;
• Roles and responsibilities listed, resources allocated and staff trained;
• Commissioning/testing represents expected scenario(s);
• Plans for transition to new ways of working, where applicable;
• Plans for handover to facilities management, where applicable;
• Training plans and relevant supporting material, if required; and
• Plans for a user support helpdesk, where applicable.
3.7 Have internal/external parties agreed change management plans; plans for
migration and data transfer; client and supplier implementation; or rollout?
Evidence expected Status/Ref
• All required plans in the contract;
• All parties or their representatives aware of and agreed to their responsibilities;
• Where relevant, partnering agreement in place or planned; and
• Shared understanding of the change control process.
3.8 Have changes to the contract been previously forecast, accurately recorded and
approved?
Evidence expected Status/Ref
• Contractual basis for ‘manage and operate contract’ reviewed and agreed; and
• Contract documentation with appropriate authority for all changes since award, including rationale for the
change.
3.9 Is the delivery agency operationally ready to manage the contract?
Evidence expected Status/Ref
• Appropriate involvement of future operational contract management team;
• Handover arrangements regarding knowledge and learning between provision of assets (where required)
and contract management teams;
• Project team who will be available to the contract management team over the first year of operation
identified; and
• Expected issues related to defects in the finished product identified and addressed.
3.10 Does the project meet whole-of-government ICT policies, standards and priorities?
Evidence expected Status/Ref
• Assessment against whole-of-government ICT policies, standards and priorities in completed self-
Assessment template (available from ICT Assurance).
GATE 5 REVIEW GUIDELINE Pre-commissioning
16
3.11 Does the project align with the NSW Government Sustainability Plan?
Evidence expected Status/Ref
• Project aligns to sustainability plans/policies where applicable.
3.12 If the project is replacing an existing system or infrastructure, are NSW e-Waste
policies considered?
Evidence expected Status/Ref
• Project aligns to sustainability plans/policies where applicable.
3.13 Can non-functional requirements (NFRs) be tested and is testing of NFRs
provided for?
Evidence expected Status/Ref
• Origin and validity of NFRs readily assessable; and
• Performance, volume and stress testing being planned.
3.14 Is there a system to track, report and, if required, correct progress?
Evidence expected Status/Ref
• Backlogs monitored with evidence of realignment if required;
• Earned value properly measured; and
• Timely reports/dashboard to the program board/steering committee.
3.15 If there are legacy systems, are plans to transfer data, integrate with them and exit
them adequate?
Evidence expected Status/Ref
• Review of plans to establish viability of approach.
3.16 Is the incremental planning approach overloading resources or the schedule?
Evidence expected Status/Ref
• Monitoring of progress and backlog.
3.17 What are the contingency plans and estimates for non-digital services?
Evidence expected Status/Ref
• Evidence of a suitable non-digital solution where required.
3.18 Is change managed/controlled effectively?
Evidence expected Status/Ref
• Change management strategy and log details when, what, why and who.
GATE 5 REVIEW GUIDELINE Pre-commissioning
17
3.19 Has change management for cyber security been successful?
Evidence expected Status/Ref
• Evidence through the change register and interviews.
3.20 What lessons are captured and considered from past or public releases?
Evidence expected Status/Ref
• Evidence of the systematic and sound identification, capture, retention and dissemination/use of lessons
learned information; and
• Learnings from cyber security implementations considered.
GATE 5 REVIEW GUIDELINE Pre-commissioning
18
4. Readiness for next phase: Post-implementation
Each numbered item below is an area to be probed.
4.1 Are all project elements ready for service?
Evidence expected Status/Ref
• Updated schedules;
• Health and safety file;
• Handover certificates;
• Test and commissioning data;
• Plans for transition are in place;
• Plans for ‘operate contract’/service phase;
• Contingency plan in place, if required; and
• Technical documentation, including delivered drawings, operating manuals and instructions, and
information assurance documentation.
4.2 Is ownership after handover clearly understood?
Evidence expected Status/Ref
• Project Sponsor has identified business owner for the operational service, where applicable;
• Project Sponsor has identified and agreed the critical success factors with the business owner; and
• Handover responsibilities and arrangements documented and agreed.
4.3 Is the delivery agency ready to adopt new ways of working, where applicable?
Evidence expected Status/Ref
• New business processes identified, tested and ready to go-live;
• Information and support available (e.g. customer information at call centres); and
• Where applicable, members of the public as end-users aware of the new service and can find out more.
4.4 Is the long-term contract management process in place?
Evidence expected Status/Ref
• Performance management plans in place;
• Performance enhancement process agreed with service provider and documented in contract; and
• Means of measuring performance agreed with supplier or partners.
4.5 Is there a process to manage and measure benefits?
Evidence expected Status/Ref
• Benefits management plans linked to program outcomes where applicable;
• Means of measuring benefits agreed with supplier/partners; and
• For collaborative projects, all parties understand and agree responsibilities and arrangements for benefits
realisation.
GATE 5 REVIEW GUIDELINE Pre-commissioning
19
4.6 Has ongoing operation and maintenance been considered?
Evidence expected Status/Ref
• Issues and ongoing costs for maintenance of ICT infrastructure and applications monitored against
expectations and addressed.
4.7 Is there a process for post-implementation reviews?
Evidence expected Status/Ref
• Plan for post-implementation reviews endorsed by supplier and internal and external parties.
4.8 What communications are planned for releases or for live transition?
Evidence expected Status/Ref
• NFRs scaled to likely consumer demand; and
• Communication products address this, and potential mismatch of surge demand identified.
4.9 Are time and resources allowed for product integration and operational readiness
testing over and above testing as part of development iterations?
Evidence expected Status/Ref
• Testing plan details the nature and rationale for planned testing;
• The need for full system and end-to-end process testing recognised, especially in multi-vendor
environments; and
• Test schedules do not assume success at first pass and allow for faults identification and rectification.
4.10 Are end users adequately prepared for the transition?
Evidence expected Status/Ref
• User research and engagement; and
• Communication strategy defines customers and end users, and how they will be engaged.
4.11 Has the transition from project to business as usual been effective?
Evidence expected Status/Ref
• Support service handover arrangements defined and, if externally supplied, timing and handover
arrangements defined in contracts.
GATE 5 REVIEW GUIDELINE Pre-commissioning
20
Gate 5 Review: Typical project documentation The review team should expect to receive evidence as noted below.
Governance, requirements, policy and resourcing
• An updated requirements definition with any changes agreed during the period up to Gate 5;
• Updated Business Case and plans for benefits realisation that reflect the effect of any requirements changes,
and the plans for service delivery;
• Governance arrangements for the management of the operational contract;
• Active management of the product backlog and sprint backlog (Agile);
• Conducting sprint planning, review and retrospective meetings (Agile);
• Close-out (if the project ends at implementation) and status reports and reconciliations for adherence to statutory
requirements;
• Lessons learned during the project (if the project ends at implementation); and
• The agency self-assessment template showing compliance with whole-of-government ICT policies, standards
and priorities.
Stakeholder engagement and change management
• Training plan;
• Close-out (if the project ends at implementation) and status reports and reconciliations for communication and
external relations; and
• Plan for management of change, including expected changes to requirements over time.
Quality Management
• Test plan and test reports;
• Close-out (if the project ends at implementation) and status reports and reconciliations for environmental
performance;
• A plan for performance measurement; and
• Information assurance documentation (accreditation) and operational and maintenance instructions and
warranties.
Financial Management
• Close-out (if the project ends at implementation) and status reports and reconciliations for cost versus budget,
and actual versus planned schedule.
Procurement and commercials
• The updated contract; and
• An assessment of contractual issues during the project to date.
Risk Management
• Close-out (if the project ends at implementation) and status reports and reconciliations for risk management;
GATE 5 REVIEW GUIDELINE Pre-commissioning
21
and
• Updated risk registers and issues log, including residual risks.
Planning and control
• Outline project plans through to completion and detailed plans for the next stage;
• Updated contingency and reversion plans;
• Active management of the Scrum Board/holding stand-ups (Agile); and
• Tracking of the Sprint Burndown Chart (Agile).
Benefits Management
• Benefits management plan.