DKIST Project Execution Plan & Construction Review

Panel Report

12/15/2015

Executive Summary The Daniel K. Inouye Solar Telescope (DKIST) construction project was reviewed by an external panel on December 7-9, 2016 at NOAO Headquarters in Tucson, AZ. The Panel reviewed project status and performance, and compared the formal processes used by the project in areas such as earned-value reporting and risk estimation for consistency with the requirements outlined in the National Science Foundation Large Facilities Manual (NSF LFM). Minor recommendations to improve the project processes have been identified. The Panel finds DKIST’s budget, schedule and scope performance against plan to be excellent, and the project office appears well-supported by a first-class set of tools covering all areas of project needs. Budget performance has been strong, and we believe adequate contingency remains to complete the project in most scenarios. Minor schedule delays are apparent, and periods of more complex scheduling are approaching, but the Panel is confident the Project will be able to address these issues. Two remaining significant risks (threat of delays due to Hawaiian permitting issues, delivery of the VTF instrument) are being watched closely, and may require assistance and support from NSF and AURA to be addressed. In all cases we find the DKIST Project Office is doing an outstanding job managing the construction process, and we have confidence in this team being able to successfully execute the remainder of the project. Introduction This review takes place two years after the implementation of the revised baseline and approximately two months after the closeout of the National Science Foundation (NSF) ARRA funding line. The goal of the review is to explore the overall status of the DKIST construction project relative to the most recent version of the Project Execution Plan, including examination of DKIST’s scope, budget, schedule, and risk management performance. A key question to be addressed: is there a high degree of confidence that the planned scope of the facility can be delivered within the parameters defined in the project baseline definition? This report contains two primary sections – (a) a narrative section indicating finding, recommendations and comments in several key areas presented to the panel, and (b) specific responses to questions directed to the panel by the NSF. A list of Panel Members and bios, and the Charge to the Panel, can be found in Appendices A & B.

Science (Talk reference: Rimmele “DKIST Science”) FINDING: the 4-m, off-axis Gregorian alt-azimuth design of DKIST promises to revolutionize solar astronomy. Integrated adaptive optics, combined with a thermally-controlled enclosure will provide superb solar imaging over wavelengths ranging from 0.3 – 28 microns (near-UV to infrared), with an angular resolution <0.03 arcsec. First-light instruments planned include a Visible Broadband Imager (VBI), a Visible Spectropolarimeter (ViSP), a Visible Tunable Filter (VTF), a Diffraction-Limited Near-Infrared Spectropolarimeter (DL-NIRSP), and a Cryogenic Near-Infrared Spectropolarimeter (Cryo-NIRSP) . DKIST will probe the magnetic fields at all levels of the solar atmosphere down to length scales (20-30 km), with a polarization accuracy of <0.01%. It will enable photospheric high-spatial resolution measurements of the quiet-Sun magnetic fields, studying the magnetic field distribution and the effects of a local (turbulent) dynamo. Observations of sunspots will provide information about magneto-convective processes, while spectroscopic and spectropolarimetric measurements of the chromospheric layers will probe the three-dimensional thermal, magnetic and material flow dynamics. Polarimetric measurements in combination with fast imaging capabilities will shed light on solar flare physics. The telescope’s off-axis all-reflective design on a dark coronal site (Haleakala, HI) will enable the study of the solar corona, the dynamics of the coronal magnetic field during filament and prominence eruptions, flaring events and Coronal Mass Ejections. COMMENT: The outstanding science potential of DKIST is well understood and appreciated by the panel; we recommend that in future reviews, more clearly placing DKIST in the broader context of the existing instruments in the field, and highlighting the enormous step in capability that DKIST represents, would assist in transmitting the excitement of the initiative, and connecting the data the telescope will produce to the important theories and results expected from this new era in solar physics. Project Performance/PMCS (Talk reference: McMullin “DKIST Status and Performance”) (Talk reference: McVeigh “DKIST Project Management Controls System”) FINDING: After six years of construction, the project is close to schedule and within budget projections, with an adequate remaining contingency. In most areas (particularly telescope), design and fabrication activities have proceeded well, with accepted components meeting specifications. System Engineering processes (as summarized by the DKIST V-diagram) appear well-integrated to project activities. FINDING: The current schedule shows an 80-day slip through the end of the project, which the panel was told has arisen largely through weather impacts to telescope construction. The project is underway trying to

recover this slip, with several opportunities for replanning to address some/all of the concerns. The project has not yet funded the slip, and still holds 37 days of schedule contingency in addition to a contingency lien for 12-months of funded work. In the Panel’s opinion, the schedule risks associated with instrument development and deployment will probably be more significant. We note that to date the project has done an outstanding job dealing with major schedule impacts (M1 blank crack, permitting/demonstration issues, ARRA spendout). RECOMMENDATION: The Panel strongly recommends the project makes the enclosure and adjacent support & operations buildings watertight as quickly as possible to enable interior work to continue independent of weather, and perhaps to avoid further project delays due to permit problems associated with exterior work. FINDING: Staffing across the project appears adequate, though the panel noted that the inevitable impacts of having the project team spread across three sites (AZ, CO, NM, HI) should be monitored. The proposed retention bonuses should help incentivize people to remain with the project through the end of construction. FINDING: Budget performance is good, with reasonable CPI/SPI performance and remarkable EV tracking to plan. More discussion of contingency management is found later in this document. FINDING: Project controls appear mature and efficient. All key areas (cost estimation, IPS, budgeting and EVMS, reporting, risk management, change control) were explored (with some commented on specifically later in the document), with excellent tools and buy-in from the project team apparent. Contingency Management (Talk reference: McMullin “DKIST Status and Performance”) FINDING: The project is actively managing risks and contingency. At rebaselining (Jan. 2013), the project had $47.5M of contingency which was 20% of the remaining costs. As of Oct. 2015 $36.9M contingency remains. This is 32% of the remaining costs. Identified risks account for 88% of the remaining contingency at an 80% confidence level. Project is tracking 92 risks (all levels of severity) in the risk register. $12M of the risks have a high assigned probability (80%) of occurrence. FINDING: The risk and contingency associated with the delivery of the instruments has been collected into one item. When split up into individual instruments, the liens on project contingency for each may be less than $1M and thus not fall into the high-risk category. The dollar value of the risk on the instruments doesn't capture their importance to the project because in some cases the risk is shared or covered by the partners, not the project. RECOMMENDATION: The instrument risks and contingency liens should be separated into instrument-based quantities, to enable fine-tuning of the analysis and planning to address issues in the individual developments.

RECOMMENDATION: The project should also identify an ideal delivery and commissioning sequence for each instrument; this might help the project and instrument builders focus on required infrastructure and related logistics and on the instruments that are required for the first stages of commissioning. FINDING: The project was baselined with 13 months of schedule contingency. Standing army costs of 6 months accounts for $5.5M of the liens of contingency. The project is behind schedule by 80 days and over the course of the project has slipped about 1 month/year. Some schedule recovery may be possible. In April 2016, when the enclosure is completed, the schedule will be adjusted to reflect the delay that is predicted at that time. Potential disruptions due to permitting could introduce long delays. The project has a robust and effective change control process and a detailed contingency usage log. 654 change requests have been processed to date, roughly 11 per month. 201 (31%) of these involved contingency usage. High level risks are defined as having an impact >$1M, and these are actively managed. Mitigation plans are developed and implemented. We note that enhanced approaches to Monte Carlo analysis of contingency estimation might be useful. FINDING: The cost associated with a long permitting delay could exceed the remaining cost and schedule contingency. This eventuality may have to be addressed by the NSF. RECOMMENDATION: The project should develop with NSF a plan for handling a potential delay due to reassessment of the existing permits, clearly assigning responsibilities for the cost and schedule impacts. COMMENT: Post 2017, the remaining cost in the project is dominated by labor and could potentially be associated with larger contingency usage than in the past, which was dominated by contracts. The project has demonstrated success in mitigating a wide range of risks and is focused on managing future known risks. They expressed confidence that their list has been updated and informed by a broad base of experience. FINDING: The project has excellent tools for evaluating descope options, risks and contingency tracking. RECOMMENDATION: A careful plan for when risks are realized or eliminated should be developed and used to make decisions on how and when to apply the remaining contingency (a contingency burn-down plan). For example in April 2016 the enclosure should be complete. This presumably retires some risk and frees up some contingency lien that could be used to reduce the risk in instrument delivery. A detailed plan would include decision point dates and potential trades and contingency usage from now to the end of the project. The project has a good chance of finishing within the remaining contingency with two potential exceptions: 1) a long delay from permit reassessment, 2) the VTF instrument, while not a project cost, the instrument is critical to the science of DKIST and a long delay (beyond the end of the construction project) is a distinct possibility. Project contingency could potentially speed up the development of that instrument.

FINDING: While the project has a detailed and integrated approach to risk and contingency management, the panel was concerned that the strong focus on “known unknowns” could lead to an inability to think out of the box regarding risks – unable to see “black swans”. RECOMMENDATION: Project leadership should step back and make a new high level, out-of-the–box assessment of worries, risks and potential draws on contingency to uncover anything that might be simmering but has not been fully captured in the existing lists. Risk Management (Talk reference: McMullin “DKIST Risk Management”) FINDINGS: The project presented a risk register that follows the industry (Project Management Institute) standard treatment of risk. It includes risk descriptions, implications, severity, likelihood, mitigation and provides for tracking the implementation of mitigation and risk reduction strategies. The risk register methodology was stated as compliant with the NSF Large Facility Manual. The risk register tracks the contingency adjustment to the project cost that is estimated for addressing each risk should it develop. This supports calls on contingency funds through the change control process. The risk management process includes Monte Carlo estimates of the likely contingency cost. The project currently carries 92 top level project risks. From an initial contingency pool of $47.5 million, DKIST now retains $36.9 million for a project that is slightly more than 60% complete. The current risks are valued at approximately $30 million and are estimated to cost no more than $33.0 million for an 80% probability of success. The risks are tracked in regular engineering management meetings. The highest-level risks were described. A list of scope contingency opportunities was prepared in the past to provide additional flexibility to address the situation where the emerged risks require resources beyond those estimated in the risk register. COMMENTS: The risk register is comprehensive, employs state of the art techniques, is thorough and is actively managed and updated. DKIST has done an effective job in risk and contingency planning and has been prudent, even conservative, in allocating contingency to emerging risks. The register process is compliant with the Project Execution Plan and the NSF Large Facility Office requirements. It is reported to the NSF in a transparent manner. The web-based interactive tools are effective, especially important for a geographically-distributed project. The use of the Monte Carlo method to assess risk likelihood and contingency cost is appropriate. The DKIST risk register is a comprehensive means of achieving visibility and tracking for the broad range of foreseeable risks. While the risk register properly identifies the significant risks from the instrument subprojects, it does so in an aggregate manner. The separate risks in each of the instrument subprojects suggest that the sum cost of the instrument risks may not be properly covered by the single aggregate risk register item. Furthermore, by aggregating the instrument risks, the particular mitigations and risk management opportunities are obscured. The risk register should include separate register items for each instrument.

The legal risks from the pending Hawaiian court actions are potentially very serious and the recent Hawai’i Supreme Court ruling in the Mauna kea case for TMT signals significant risk for DKIST. While the risk register contains an item for legal risks, the project assessment of that risk may need to recognize a larger and more imminent threat. The risk from a court-ordered interruption in DKIST construction is so large that NSF may need to consider it to be an NSF-carried risk. The estimated cost of addressing all of the register risks is about 88% of the remaining contingency pool. The project’s estimate indicates that the contingency pool may be adequate to complete DKIST, but there is a significant chance that it may be exhausted. Against this eventuality, DKIST has prepared a list of scope contingency items that may extend the opportunities for risk management. However, the scope contingency analysis was prepared at an earlier stage of the project and has not been recently updated or reevaluated. A quick assessment, during the review, of the scope opportunity list indicates that approximately $14 million may be available from future expenditures as additional ability to manage the project endgame risks. The risk register is comprehensive but it focuses DKIST on managing the foreseen and carefully analyzed risks within the register. DKIST construction has been ably and successfully managed to date. As mentioned in the previous section, this success to date, as well as the focus on the risk register, may leave the project vulnerable to being blind-sided by “black swan” risks that emerge, such as an adverse Hawaiian Supreme Court ruling. The project and the NSF should assess these other threats as best they can and be careful to think out of the box defined by the risk register lest they be surprised by a black swan. RECOMMENDATIONS:

1. The risk register treatment of instrument risks should be divided into separate entries and risk assessments carried out for each instrument.

2. Reassess the risk from the two pending Hawai’i Supreme Court cases and work with the NSF to develop a strategy for reaction to adverse court rulings that may result in 1 to 2 year halts in onsite construction, and the resulting DKIST rebaseline.

3. The project and the NSF should assess potential significant but unexpected threats as best they can and be careful to think out of the box defined by the comprehensive risk register lest they be surprised by an unexpected but consequential development.

Software (Talk reference: McMullin “Project Software Summary Presentation”) The review panel asked the DKIST team to prepare a presentation that outlined the software development plans, the associated documentation and the planned verification and validation test concept. The panel was taken to a test area for the Observatory Control System software package and provided an overview of the

Graphical User Interfaces that observatory operators will use to control the various systems and instruments as well as monitor health and safety of the facility. FINDING: The software development plans are defined within the top-level systems engineering documentation, and requirements are derived and documented for each component, subsystem and system. The overall software design employs a Common Services Framework with a modular approach to software packages. High-level packages include the Observatory Control System, the Instrument Control System, the Telescope Control System, and the Data Handling System. These large system software packages are each well-defined in scope, and requirements definitions are adequately documented. Each of these packages has independent documentation for the flow down from operational requirements, to system requirements, and design specifications, as well as compliance documents and operation manuals. The project maintains an end-to-end simulator that will be used to integrate and test each software component. A test plan has been developed that details the various unit, system and recursive/regression tests that will be done for the individual and merged packages as the full software suite is being completed. FINDING: The component software development plans follow the same development approach as outlined for the major systems. Interface Control Documents define the interactions for the various subsystems and instruments that are developed within the larger software packages. These ICDs are shared with the instrument teams and component/system vendors. FINDING: The project has an acceptance test plan and schedule for components and systems and has considered the staffing associated with these activities. FINDING: The project identified version control software for configuration management but did not articulate a specific concept for the ongoing maintenance of the software packages or configuration controls for the software. COMMENT: We suggest project present more information about these systems in future reviews. Instruments (Talk reference: McMullin “DKIST Status and Performance”) (Talk reference: Warner “Instrument Overview/SWOT analysis”) Visible Broadband Imager (VBI): FINDING: The VBI is being developed by the National Solar Observatory staff and consists of two separate instrument channels, red and blue. The VBI is fed from the main telescope using a dichroic beam splitter and will be positioned on the Coudé deck of the facility. The instrument is well past Critical Design Review and is

actually nearing completion with no identified performance or programmatic issues. The VBI is the first instrument required in the commissioning phase for the facility. Visible Spectro-Polarimeter (ViSP) FINDING: The High Altitude Observatory of the National Center for Atmospheric Research (NCAR) in Boulder, CO is developing the ViSP instrument. As with the VBI, ViSP is mounted on the Coudé deck and fed using a dichroic beam splitter is designed to simultaneously measure three distinct wavelengths in the visible range of the solar spectrum (380 to 900 nm) with very high spatial and spectral resolution. The instrument has successfully passed Critical Design Review (CDR), but the fabrication contract is still under negotiation. This instrument is a relatively low complexity system and has a reasonable plan for fabrication, subsystem integration, testing and performance verification. DKIST is carrying a contingency lien of $360K for this instrument. The project noted that this instrument team is performing well, and the team is conforming to the programmatic reporting scenario employed by DKIST management. Cryogenic Near Infrared Spectro-Polarimeter (Cryo-NIRSP) FINDING: The Cryo-NIRSP will be provided by the Institute for Astronomy, University of Hawai’i. Cryo-NIRSP has passed CDR, and is a significant contribution from IfA to the project. Major fabrication contracts have been awarded. COMMENT: The Panel suggests that the project continue to monitor the development of this instrument at both the technical, programmatic and contractual levels. Visible Tunable Filter (VTF) FINDING: This instrument is being developed by the Kiepenheuer Institute for Solar Physics (KIS) through a Memorandum of Understanding between KIS and AURA. The VTF is a key science instrument for the scientific community, with many of the level 0 science requirements tied to the performance of this instrument. The VTF is a dual Fabry-Perot tunable filter that will provide rapid imaging spectroscopy, Stokes imaging polarimetry, and calibrated surface photometry. The instrument is very far behind in its development compared to the other instruments. It has failed a recent Preliminary Design Review, and the instrument team has proposed delivery of a reduced performance instrument (one Fabry-Perot etalon instead of two) nearly twenty months later than required for facility integration. These failures threaten the early science promise of DKIST, and potentially limit the appeal of the facility to the broader solar physics community. RECOMMENDATION: The DKIST project should request the AURA president, Dr. Matt Mountain, contact the KIS Director and VTF PI, Herr Prof. Oskar von der Lüehe and work through what will be required to recover the schedule and assure delivery of VTF within the construction phase of the DKIST facility. It is further recommended that the program work with the NSF, subsequent to the conversation between Dr. Mountain

and KIS, on a mitigation plan for an instrument delivery. The project should seriously consider the deployment of a systems engineer and program management expert to KIS to assist develop proper SE&I and program controls to help recover the instrument schedule. RECOMMENDATION: We suggest the project and/or NSF might begin consideration of alternative plans to deliver the VTF capabilities to DKIST. Scope Management (Talk reference: McMullin “DKIST Status and Performance”) (Talk reference: McMullin “Project Descope Plans with Timelines/impact”) The panel requested that the project share plans for scope reductions (descoping) in the event of the exhaustion or near exhaustion of contingency (either budgetary or schedule). FINDING: The project was able to supply a listing of potential candidate tasks that might be available for descoping, employing their progress/performance tools (chiefly Tableau), literally overnight. They clearly understood and had quickly updated existing information to preliminarily identify the tasks for which substantial remaining value existed, and noted deadlines for when commitments would render the task with insufficient remaining value to be useable for descope candidates. RECOMMENDATION: The panel recommends that the project regularly monitor a descope list. Integration, Testing & Commissioning (Talk reference: Craig “DKIST Integration, Testing and Commissioning Planning”) FINDINGS: the Panel found that the processes in place for planning and documentation for Integration, Testing and Commissioning (IT&C) were robust, comprehensive and highly supported by the project team. The project has developed a system of documentation and configuration control that has demonstrated its capability for maintaining good records of the systems’ requirements, designs, acceptance (with or without modification) and implementation. An interface matrix documents the coordination of the various interface connections of the various subsystems and the requisite testing regimes (many of which are in progress.) We find this system, which is actively being used, to be ready for the subsystems’ integration. The project noted that a relational database (DOORS) was being set up to help manage retention and provision of the many documents required to validate each subsystem; document the connection and testing of the subsystems; leading to verification of level 2 objectives culminating in the validation of level 1

(telescope system level) and 0 (telescope science) objectives. The project is planning to hire a systems engineer to continue and complete the transition to this new database. RECOMMENDATION: The panel supports this plan and fully supports the addition of this new systems engineering position, and recommends this be completed as soon as possible. The instrumentation IT&C plan is being prepared without the benefit of knowing the exact sequence of when the instruments will be arriving at the site. Although the exact IT&C plan is not entirely worked out at this stage due to this uncertainty, the process in place seems very flexible and capable of handling this activity. The project will, without doubt, have more information in the next six months on the expected delivery of at least two of the instrumentation systems needed to validate the telescope subsystems. RECOMMENDATION: The Panel recommends that the project continue to work toward securing the delivery dates of the instruments and establish an optimized sequence of installation to provide the most likely dates for proof of the level 1 objectives. Furthermore, the panel believes that the IT&C processes that the project has developed are capable of identifying risks associated with possible non-conformances and attendant schedule impacts. This gives the project the capability for mitigation or potential remediation of the non-conformances before they severely impact schedule and budget contingency. However, there is one instrument (the VTF, see the section on Instrumentation) that is presently found by the project to be tracking for late delivery. FINDING: The identified test plans and procedures within the IT&C plans flow from the level 0 observatory and science requirements documents. There is clear traceability from these level 0 requirements to the level 2 articles and to the test plans in place that will demonstrate performance verification. The systems engineering methodology employed to manage the various sub system and system and facility level verification metrics is robust and well developed. The documentation for many of the various activities is already mature, and there is adequate contingency and schedule margin for the further refinement for these processes. Additionally, the project has demonstrated a clear approach for management of the instrument verifications, notwithstanding the noted ambiguity in the expected delivery date for any specific instrument. Compliance/Safety/local interactions (Talk reference: McMullin “DKIST Communications and Compliance”) (Talk reference: Shimko “DKIST Project Safety”) FINDING: Over the past several years, the project has realized significant delays and additional issues regarding compliance issues and interactions with the local community over concerns about the project. At the current time, we note the two pending Hawaiian Supreme Court rulings in progress regarding DKIST, and a rapidly-evolving broader Hawaiian situation regarding telescope construction. While the Project Office acknowledges the potential threats, they are proceeding with development assuming no further permitting

delays. This is an appropriate strategy, but the Panel notes that a (“black swan”) sudden halt to construction due to a Supreme Court ruling against DKIST would be a major perturbation to the project. NSF is aware that they will own (and need to address) the impacts associated with this outcome. FINDING: The Safety program is well-developed, and a strong culture supporting safety is apparent from the project team. Safety performance during construction has been good. The Panel noted that details regarding the clean-room aspects of the Coudé platform were not discussed in the main presentation, and requested/received further information from the Project Manager (which was delivered). Other Topics

FINDING: The Panel notes that the review format (which excluded discussion of DKIST Operations and Construction to Operations transition issues) led to several instances where insufficient detail and odd language interpretations arose (example: the Project Office indicated that “Operations” does not begin until a fully-accepted facility is handed over; yet it is apparent that “Early Facility Operations” activities are taking place during the construction phase). RECOMMENDATION: In the next review (Transition from Construction to Operations) more information about construction to operations transition should be provided.

Committee Charge Responses

• Assess the DKIST Project’s performance (i.e. the is the project meeting scope as set by the Science

Requirements Document and Operational Concepts Definition, are costs, schedule, and risks close to those projected) to date, focusing on the Project’s performance since the re-baseline and its projections for completion. Guidance for NSF performance reporting is set forth in Section 4.5 of the NSF Large Facilities Manual (NSF 15089) (hereafter LFM).

The Panel believes that project performance against the plan defined in the Project Execution Plan is excellent, and the project is being carried out in compliance with the requirements and principles outlined in the LFM.

• Assess the DKIST Project’s processes for project management and controls including change control, risk management, contingency planning, requirement/compliance tracking and plans for managing the project scope. In the Panel’s opinion, the project has developed, implemented and is effectively using an exemplary suite of tools and processes to manage the project.

• Contracting & procurement: Review the Project's contracting and procurement strategies, techniques for verification and acceptance of deliverable items and lessons learned. Review the Project’s strategy for the development of Memoranda of Agreement/Understanding (MoA/MoU) for partner in-kind contributions to materials (i.e. instrumentation) and labor. The project has generally demonstrated skill and creativity in dealing with various contractual challenges, but the Panel notes some significant acquisition challenges remaining, most notably the VTF instrument.

• Project Change Controls: Review the Project’s methods for managing the scope, schedule and budget with guidance from Sections 4.2.5 and 5.2 of the LFM.

As described in earlier sections of this review report, the project’s scope definition, configuration management, document version control, integrated schedule planning and critical path analysis, budget estimation, budget contingency estimation, schedule contingency definition, earned value management, change control and risk management are found to be fully compliant with the referenced sections of the LFM. Planning for scope contingency has also been conducted compliant with the LFM. However, recent tracking and management of the remaining scope contingency may not be up to date and the project should carry out a review of scope reduction opportunities that may be useful in managing the project’s completion.

• Risk Management: Review the Project’s techniques for identification, assessment and monitoring of risks as well as the tactics to control and mitigate their impact with guidance from Section 5.2 of the LFM.

Risk management is also fully compliant with the LFM sections, including risk identification, qualitative and quantitative risk assessment and monitoring and control of defined risks. These constitute a comprehensive treatment of the identified risks. However, as stated earlier in this report, consideration of “unknown unknowns” and risks that are NSF responsibilities should be strengthened.

• Earned Value Management (EVM) System: Review the project’s system for tracking, calculating, and reporting earned value to ensure accurate and timely reporting, with guidance from Sections 4.2.5, 4.5, and 5.7 of the LFM.

See answer above. The Project’s EVMS and management support systems are seen as compliant with the LFM and are very impressive.

• Requirement & Compliance Tracking: Review the Project’s systems engineering methodology to ensure traceability of requirements (to safeguard quality designs), to provide the ability to manage change, and to direct and record the verification of deliverables.

Noting the pending installation of DOORS, at the current time we believe that traceability is being actively and effectively managed inside the project.

• Assess the Project’s cost estimation history, performance, and projections for the remaining work to be performed through the end of construction.

The project presented the detailed processes used to provide cost estimates throughout the development, and the Panel explored (at a high level) the success and fidelity of that framework, which appears robust and of high quality. We note the extraordinary performance against plan as indicated by EV; and we were impressed by the PMCS expertise demonstrated by the Cost Account Managers (CAMs).

• Assess the Project’s safety record including planning and management.

The Safety Program presented to the Panel is comprehensive and robust, and the site safety record is exemplary. The safety personnel demonstrate a strong commitment to ongoing monitoring, training and workforce well-being.

• Review the project’s engagement with stakeholders (e.g. the Hawaiian Department of Land and Natural Resources, the Hawaiian Department of Fish and Wildlife, the U.S. Fish and Wildlife Service, the Federal Aviation Administration, the National Park Service, and Native Hawaiian cultural practitioners) in compliance with federal, state and local environmental and cultural requirements. Assess the projects relations with the labor unions, the Native Hawaiian community and its outreach to the public.

Noting the significant challenges faced by the project in recent years, both the DKIST project management and overall AURA/NSO administrations and the NSF have provided effective and successful stewardship of a complex and rapidly-evolving situation.

Summary The Panel believes that the DKIST project is performing excellently relative to the Project Execution Plan baseline, with all major project processes in place and functioning well. Minor schedule delays apparent in the current status are being analyzed. The Project is aware of the possibility of additional delays arising from the Hawaiian Supreme Court situation, and is working closely with NSF on the matter. The Panel anticipates that the late stages of the project (when the instruments arrive) will be a complex time, and we urge the project to begin planning of that time as early as possible. With one exception, the instrument developments appear to be in good shape, and we urge AURA and the NSF to work with the DKIST German partner to recover the situation for VTF.

We commend the project director, project manager and the entire project team for their outstanding work over the past six years to bring DKIST to this point, and we wish them the best of luck in the remaining development.

APPENDIX A Panel: Tony Beasley NRAO (Director) Jonathan Cirtain NASA MSFC (Heliophysics Team Lead) Brenna Flaugher Fermilab (Department Head, Astrophysics) Paul LaMarche Princeton (Vice Provost, Space Programming & Planning) Gary Sanders TMT (Project Manager)

Panel Member Biographies: Dr. Tony Beasley Beasley received his Bachelor’s Degree in Physics with First Class Honours, and a PhD in Astrophysics from the University of Sydney in 1986 and 1991, respectively. His doctoral thesis examined magnetic field generation and solar-stellar activity in post-main-sequence stars. After his completing his PhD, he joined NRAO first as a post-doc, then as a scientific staff member and senior manager in Socorro, NM and Charlottesville, VA. His scientific interests include non-thermal stellar radio emission, Very Long Baseline Interferometry (VLBI) techniques, radio supernovae, and astrometry of stellar/interstellar masers. Beasley became the Director of the National Radio Astronomy Observatory (NRAO) on 21 May 2012. Dr. Jonathan Cirtain Dr. Jonathan Cirtain is the manager of the Heliophysics, Planetary and Space Weather Branch of the Space Sciences Office at NASA's Marshall Space Flight Center in Huntsville, Ala. He is the former project scientist for Hinode -- an international mission to study the sun -- and leads the solar instrumentation development programs and technology development effort. He has served on national and international science advisory committees, and he currently is the Co-Chair of the Hinode Science Working Group. Dr. Cirtain also is the institutional principal investigator for the Solar Wind Electrons, Alpha and Protons instrument, or SWEAP, slated to launch no later than 2018 aboard the Solar Probe + mission to explore unprecedented regions in space, transforming our understanding of the sun and its effects on the solar system. Dr. Cirtain also developed the secondary payloads capability for Exploration Mission 1, the first launch of the nation's new heavy launch vehicle, the Space Launch System currently planned for November 2018. He began his NASA career in 2007 as an astrophysicist. Dr. Brenna Flaugher

Dr. Brenna Flaugher is a Physicist and Department Head of Astrophysics in the Particle Physics Division at Fermi National Accelerator Laboratory (Fermilab) in Batavia, Illinois. She received her BS degree in physics from Bates College in Lewiston, Maine, and her M.S. and Ph. D degrees in physics from Rutgers, the State University of New Jersey. From 1986 to 2003 she was a member of the Collider Detector Facility (CDF) Collaboration, serving over the years as a convener of the QCD & Jet Analysis Group and the CDF Operations Manager. From 1996-2003 she led the assembly of the CDF Run IIa silicon vertex detector and the development of the CDF silicon vertex detector for Run IIb of the Fermilab Tevatron. In 2003, inspired by huge mystery of dark energy and the obvious need for more data to guide the theory, she helped initiate the Dark Energy Survey (DES), contributed to the development the international collaboration and to convincing funding agencies to support the project. From 2004-2012 she led the Dark Energy Camera (DECam) construction project that culminated in a very successful first light in Sept. 2012. During 2009 and 2010 she served as a Member of the National Academy of Sciences Astro2010 Decadal Survey Electromagnetic Observations from Space Program Prioritization Panel. In Nov. 2011 she was elected as a Fellow of the American Physical Society. She currently serves on the scientific advisory committee at Fermilab and on the APS Division of Astrophysics executive board. In 2013 she was appointed co-project scientist for the DESI (Dark Energy Spectroscopic Instrument) project. DESI will put a new wide-field multi-object spectrograph on the Mayall telescope on Kitt Peak in Tucson and use it to constrain dark energy by measuring Baryon Acoustic Oscillations over 14,000 sq. degrees of the northern sky. Dr. Paul LaMarche Paul LaMarche has been the Vice Provost for Space Programming and Planning at Princeton University since March of 2007. In that role, Paul works closely with the University's leadership team to support the formulation and assessment of ideas for new facilities and the renovation of existing buildings. He also assists the Provost in managing the fiscal and budgetary dimensions of the University's space planning and programming agenda. Paul also supports the budgetary process related to University funded research programs. From 1984 to 1996, LaMarche worked at the Princeton Plasma Physics Laboratory, serving as a research staff physicist and as head of various groups conducting experiments at the lab. After two years at a technology firm in Massachusetts, he returned to the University to serve as the Project Manager for the Borexino Solar Neutrino Detector Project, a joint effort with the National Science Foundation and the Instituto Nazionale Fisica Nucleare in Italy for three years. In 2000-01, he also was a research engineer at the Plasma Physics Lab. From 2001 to 2007, Paul was the Department Manager and Director of Operations at Princeton’s Physics Department where he managed the human, budgetary and physical resources for the department. LaMarche earned his bachelor's degree in physics from Boston College and his Ph.D. in physics from Yale University.

Dr. Gary Sanders Gary H. Sanders spent 25 years performing high-energy physics experiments at laboratories in the United States and Europe. He earned an AB degree in physics from Columbia University and a PhD in high-energy physics from the Massachusetts Institute of Technology. He has been a faculty member in physics at Princeton University and a scientist at Los Alamos National Laboratory. In 1994, Gary went to the California Institute of Technology to serve as the Project Manager and Deputy Director for the Laser Interferometer Gravitational Wave Observatory (LIGO) project. He joined the Thirty Meter Telescope (TMT) as its Project Manager in 2004. He is the author or a co-author of more than 175 peer-reviewed publications and he has been elected a Fellow of the American Physical Society.

APPENDIX B

DKIST Project Execution Plan and Construction Status Review - Charge Background When completed in 2019, the Daniel K. Inouye Solar Telescope (DKIST) will be the world's most powerful solar observatory, poised to answer fundamental questions in solar physics by providing transformative improvements over current ground-base facilities DKIST will enable the study of magnetic phenomena in the solar photosphere, chromosphere, and corona. Determining the role of magnetic fields in the outer regions of the Sun is crucial to understanding the solar dynamo, solar variability, and solar activity, including flares and coronal mass ejections. Solar activity can affect civil life on Earth through phenomena generally described as “space weather” and may have impact on the terrestrial climate. The relevance of DKIST’s science drivers was reaffirmed by the National Academy of Sciences 2010 Astronomy and Astrophysics Decadal Survey: New Worlds, New Horizons[1] as well as the 2012 Solar and Space Physics Decadal Survey: A Science for a Technological Society.[2] Beginning in 2001, NSF provided funds to the National Solar Observatory (NSO) for an eight-year design and development program for DKIST and its initial complement of instruments through the Division of Astronomical Sciences (AST) in the Directorate for Mathematical and Physical Sciences (MPS) and the Division of Atmospheric and Geospace Sciences (AGS) in the Directorate for Geosciences (GEO). The current design, cost, schedule, and risk were scrutinized in an NSF-conducted Preliminary Design Review in October-November 2006. The FDR held in May 2009 determined that the project was fully prepared to begin construction. The original total project cost to NSF, $297.93 million, was set after a Final Design Review (FDR) in May 2009. The NSB approved an award for this amount at the NSF Director’s discretion, contingent upon completion of compliance with relevant environmental and cultural/historic statutes. In FY 2009, $153.0 million was provided through the Major Research Equipment and Facilities Construction (MREFC) account to initiate construction. Of these MREFC funds, $146.0 million was appropriated through the American Recovery and Reinvestment Act (ARRA). Given the timing of the receipt of budget authority and the complexity of project contracting, the entire $153.0 million was carried over from FY 2009 and subsequently obligated in FY 2010. The environmental compliance requirements were completed on November 20, 2009, and the NSF Director signed the Record of Decision authorizing the construction on December 3, 2009. The Hawai’i Board on Land and Natural Resources (BLNR) approved the project’s application for a Conservation District Use Permit (CDUP) on December 1, 2010. The Hawai’i BLNR subsequently approved a Habitat Conservation Plan, designed to protect the habitats of the endangered Hawaiian petrel and Hawaiian goose that could potentially be impacted by DKIST construction. Formal consultation with the U.S. Fish and Wildlife Service with regard to the endangered Hawaiian petrel was completed in calendar year 2011. A challenge to the CDUP issuance was resolved favorably for the project in November 2012 and full access to the site atop Haleakala followed shortly thereafter. Site preparation and excavation began November 30, 2012.

Current Status, Next Milestones, and the Goal of this Review Due to the extended delay in obtaining permission to commence site construction as well as the cost growth that accompanied the delay and other cost drivers that were out of the control of the project, the NSF conducted a re-baseline review over the period from October 2012 through March 2013. The revised baseline and an increase in the total project cost of approximately $46.20 million was reviewed by an external panel of experts and subsequently considered by the NSB, which approved a revised total project cost of $344.13 million at their August 2013 meeting. DKIST is currently in the Construction Stage of the Major Research Equipment and Facility Construction process. DKIST started construction in 2010 and approximately 60% of construction is complete. Construction is expected to be completed in late 2019. Near-term milestones include:

• Assembly of the telescope enclosure (dome) continues in FY 2016. The enclosure will be commissioned, declared water-tight, and accepted by the end of FY 2016 (a Level 1 milestone).

• The Coudé rotator platform assembly inside the enclosure will begin in FY 2016. • The telescope mount assembly (TMA) has been shipped to Maui, and assembly of the TMA base inside

the enclosure will begin in FY 2016. • The main M1 mirror polishing will be completed by early FY 2016 and the mirror shipped to Maui for

storage. • The fabrication of all first-light instruments will continue through FY 2016.

This review takes place two years after the implementation of the revised baseline and approximately two months after the closeout of the ARRA funding line. It seeks to review the overall status of the DKIST construction project relative to the most recent version of the Project Execution Plan. In accordance with Sections 2.4.1 and 4.5 of the most recent version of the Large Facilities Manual (LFM)[3] reviews of construction awards are performed on a periodic basis. The last external review was performed in 2012. Charge to the Review Panel After careful examination of the DKIST Project’s performance to date including the scope, budget, schedule, and risk management plan; is there a high degree of confidence that the planned scope of the facility can be delivered within the parameters defined in the project baseline definition? To that end, we request that the panel:

• Assess the DKIST Project’s performance (i.e. the is the project meeting scope as set by the Science Requirements Document and Operational Concepts Definition, are costs, schedule, and risks close to those projected) to date, focusing on the Project’s performance since the re-baseline and its projections for completion. Guidance for NSF performance reporting is set forth in Section 4.5 of the LFM.

• Assess the DKIST Project’s processes for project management and controls including change control, risk management, contingency planning, requirement/compliance tracking and plans for managing the project scope. In particular the panel should:

• Contracting & procurement: Review the Project's contracting and procurement strategies, techniques for verification and acceptance of deliverable items and lessons learned. Review the Project’s strategy for the development of Memoranda of Agreement/Understanding (MoA/MoU) for partner in-kind contributions to materials (i.e. instrumentation) and labor.

• Project Change Controls: Review the Project’s methods for managing the scope, schedule and budget with guidance from Sections 4.2.5 and 5.2 of the LFM.

• Risk Management: Review the Project’s techniques for identification, assessment and monitoring of risks as well as the tactics to control and mitigate their impact with guidance from Section 5.2 of the LFM.

• Earned Value Management (EVM) System: Review the project’s system for tracking, calculating, and reporting earned value to ensure accurate and timely reporting, with guidance from Sections 4.2.5, 4.5, and 5.7 of the LFM.

• Requirement & Compliance Tracking: Review the Project’s systems engineering methodology to ensure traceability of requirements (to safeguard quality designs), to provide the ability to manage change, and to direct and record the verification of deliverables.

• Assess the Project’s cost estimation history, performance, and projections for the remaining work to be performed through the end of construction.

• Assess the Project’s safety record including planning and management. • Review the project’s engagement with stakeholders (e.g. the Hawaiian Department of Land and

Natural Resources, the Hawaiian Department of Fish and Wildlife, the U.S. Fish and Wildlife Service, the Federal Aviation Administration, the National Park Service, and Native Hawaiian cultural practitioners) in compliance with federal, state and local environmental and cultural requirements. Assess the projects relations with the labor unions, the Native Hawaiian community and its outreach to the public.

Organization of the Review: The DKIST Project will share all appropriate documentation with the review panel via an electronically accessible library prior to the review. Project planning and requirements documents (e.g. Configuration, Cost, Scope, Risk, Contingency, Procurement, Safety, etc. Management Plans) will be made available for review no later than November 6, 2015. Time critical documentation requiring the closeout of the fiscal year financials (e.g. expenditure actuals, EVM reports, contingency logs, risk register, integrated project schedule, etc.) will be made available for review no later than November 15, 2015. The panel will communicate via e-mail and will hold at least one teleconference prior to the review. The review will be held at the location of the DKIST Project office in the headquarters of the National Optical Astronomy Observatory in Tucson, AZ December 7-9, 2015. The review panel will discuss their findings and prepare a written report addressing their conclusions with respect to the elements of the charge. Ample time

will be provided during the meeting to organize the report and to debrief the NSF, NSO and the DKIST Project. A final version of the report should be provided to the NSF by January 15, 2016. [1] www.nap.edu/catalog.php?record_id=12951 [2] www.nap.edu/search/?term=13060&x=0&y=0 [3] www.nsf.gov/bfa/lfo/lfo_documents.jsp