enabling(vu(assessments(powers/vv.presentations/knupp.pdf ·...

Enabling VU Assessments A Design for a V&V and UQ Discovery Process (SAND2011-‐6677)

Patrick Knupp and Angel Urbina NEAMS VU Crosscut

Sandia National Laboratories SAND2011-‐6934C

Sandia National Laboratories is a multi-‐program laboratory managed and operated by Sandia

Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-‐AC04-‐94AL85000.

Motivating Issues

NEAMS VU Crosscut How can the Predictive Capability Maturity Model (PCMM) be implemented as part of the assessment process within large projects? NEAMS Waste IPSC How do we integrate modern VU methodology into large simulation projects? Neither of these questions has been adequately answered!

Conten

t

Increasing completeness and rigor

Decreasing risk

SAND2007-‐5948

Predictive Capability Maturity Model for Computational Modeling and Simulation

PCMM Input

PCMM Purpose

•  PCMM helps identify the current state of predictive capability for the intended application.

•  PCMM is a tool for measuring and communicating progress in predictive capability.

•  It is a tool for managing risk in the use of modeling and simulation.

Input to PCMM assessments: descriptions of project practices & results, organized by

•  VU Element and sub-‐Element,

•  levels of fidelity,

•  model,

•  code.

To perform PCMM assessments, the input must be

well-‐organized,

accessible, and

understandable.

Motivating Issues

NEAMS VU Crosscut How can the Predictive Capability Maturity Model (PCMM) be implemented as part of the assessment process within large projects? NEAMS Waste IPSC How do we integrate modern VU methodology into large simulation projects? Oberkampf & Roy (Verification & Validation in Scientific Computing, 2010) “… our experience and the experience of others has convinced us that while technical issues and computing resources are important, they are not the limiting factor in improving the credibility and usefulness of scientific computing used in a decision-‐making environment.”

Requirements in the Literature for State-‐0f-‐the Art VU Processes

Discovery Planning, Sequencing, Terminology, Context, Initiation, Consonance, Concurrent, Practical, Transferable, Deliberate, Multiple-‐Use, Transparency, Records, VU Requirements, Traceability

Accumulation Rightness, Well-‐defined, Control Human Error, Reproducible, Reusable, Flexible, Justified, Interpret-‐ability

Assessment Status, Interim assessment, accuracy, gaps & weaknesses, balance, economical, usable, completion

Discovery Requirements: Planning. The system of processes must assist in the planning of VU work and establish the basis for assessment. Terminology. The system must use clear terminology in order to provide a common language and understanding. Context. The system must provide context to define and manage relationships between components, data, people, and goals.

From the list of requirements, Planning, Sequencing, Terminology, Context, Initiation, Consonance, Concurrent, Practical, Transfer-‐able, Deliberate, Multiple-‐Use, Transparency, Records, VU Requirements, Traceability, Rightness, Well-‐defined, Control Human Error, Reproducible, Reusable, Flexible, Justified, Interpret-‐ability Status, Interim Assessment, Accuracy, Gaps & Weaknesses, Balance, Economical, Usable, Completion one can deduce that current VU processes are susceptible to: -‐ poor planning, -‐ use of overloaded or ambiguous terminology, -‐ failure to set proper contexts, -‐ failure to use best practices, -‐ lack of transparency, -‐ lack of interpret-‐ability, -‐ lack of proper documentation, -‐ unintentional omissions (human error), -‐ being unbalanced or incomplete, -‐ wasteful allocation of resources, -‐ collecting the wrong evidence, and -‐ superficial assessments

VU Processes: Current

Premise of this Work

Integration of modern VU into large projects requires a re-‐organization in the way we plan, accumulate, assess, and disseminate the work. Integration of state-‐of-‐the-‐art VU processes is potentially a better approach to building confidence. (We do not expect to fully accomplish this long-‐term re-‐organization within NEAMS, but we do expect to add value through increased transparency.) But what are state-‐of-‐the-‐art VU processes? •  Define them, •  Apply them, and •  Improve them. Definition starts with Requirements for modern VU processes. We divide the processes into three parts: •  Discovery (planning and integration) First •  Accumulation (obtaining the evidence) •  Assessment (interim feedback) Together (FY11 focused on defining the Discovery processes, using Waste as the example.)

What is to be Discovered?

We seek to Discover: A useful and specific VU-‐oriented contextual framework that accurately reflects the logical structure of VU within the project in its current and future states. The framework consists of hierarchically arranged entities of VU significance. These entities contain place-‐holders for ‘VU data’ which describes the ultimate organization of VU within the project and the content there-‐in. During Discovery, the hierarchy and place-‐holders are identified. During Accumulation, place-‐holders are filled with VU data. During Assessment, VU data is used to provide timely feedback, status reports, and high-‐level information to decision makers.

One begins with •  a given project, •  a pre-‐defined abstract hierarchy Pre-‐defined hierarchies and place-‐holders orient VU planning along the lines established by the VU community. 0. Determine the ISC 1. The entities within the abstract hierarchy are particularized to the given project. 2. Particularized hierarchy reviewed by the team for accuracy, completeness, and logical consistency. 3. The placeholders for VU data within each entity are particularized. 4. Placeholders are reviewed by the team for accuracy, completeness, and logical consistency. 5. The hierarchy, place-‐holders, and VU data are entered into a web-‐based Discovery & Assessment tool Observations: •  The Discovery Team needs objectivity and should not consist solely of modelers & developers •  No judgments as the to quality of the work are made during discovery •  Hierarchy may be updated later if project plans change

The Discovery Process

Integrated Simulation Capabilities (ISC’s)

Self-‐contained, complex systems of simulation codes that work together (the ‘finished’ product, as envisioned)

0. The Abstract Hierarchy

The abstract hierarchy describes a conceptual ISC from the perspective of VU. The hierarchy is composed of three nested sub-‐hierarchies: 1.  One ISC hierarchy, containing 2.  Multiple Landmark Simulation Capability (LSC ) hierarchies, each containing 3.  A VU hierarchy

The ISC Hierarchy ISC Hierarchy is a Development Sequence (lower level LSC’s in hierarchy become higher level LSC’s) Incorporates “validation hierarchy” concept

Legend

Not Implemented

Implemented

Ver. A

SIMULATION CAPABILITY:

Coupling

Ver. A


C (Chemical)

Ver. B


C (Chemical)

Ver. C


C (Chemical)

Ver. A

A SIMULATION CAPABILITY:

THC (Thermal, Hydraulic, Chemical)

Ver. A


M (Mechanical)

Ver. B


M (Mechanical)

Ver. A


THCM (Thermal, Hydraulic, Chemical, Mechanical)

Ver. B


THCM (Thermal, Hydraulic, Chemical, Mechanical)

Ver. A

A SIMULATION CAPABILITY:

TH (Thermal, Hydraulic)

Comparison: ISC and Validation Hierarchies

Similarities: •  Both hierarchies tend to include more complex physics as one moves up the hierarchy, •  Both hierarchies tend to model more complex systems as one move up the hierarchy. Differences: •  Validation hierarchy is a hierarchy of experiments; ISC is a hierarchy of simulation capabilities, •  ISC hierarchy can incorporate other VU objectives aside from Validation,

•  There may not be a one-‐to-‐one relation between experiments in the validation hierarchy and simulation capabilities in the ISC, but every experiment will appear somewhere in the ISC in conjunction with a simulation capability

Summary: •  The ISC hierarchy is a broader concept that intentionally incorporates the validation hierarchy, •  Information about the experiments in validation hierarchy become VU data in ISC hierarchy.

Abstract Landmark Simulation Capability Hierarchy

VU Objectives

Seven pre-‐defined VU Objectives: 1.  VE: Validation to Experiment, 2.  CD: Performing Simulations to Create Data, 3.  CLM: Performing Simulations to Create Local Models, 4.  CGM: Performing Simulations to Create Simplified Global Models, 5.  EU: Performing Simulations for End-‐Use Analysis, 6.  VS: Verification of Software, 7.  RF: Research Fitness of Imported or Pre-‐existing Capabilities

The Abstract VU Hierarchy

VU Simulation Ensembles

1.  Calibrations 2.  Verification Tests 3.  Numerical Model Sensitivity Studies 4.  Making the End-‐Use Predictions 5.  Physical Parameter Sensitivity Studies 6.  Simulations that Feed into MMD 7.  Simulations that Create Surrogate Models 8.  Other

Traversing the Hierarchy to Obtain Context

Example: This input data à that output data à a particular simulation ensemble à which pertains to a particular VU element à which appears in an element chain related to a particular VU objective à which pertains to a particular landmark simulation capability having à a known global math model which is à Implemented in a particular software version.

1. Particularization of the Hierarchies

Development of the hierarchies is done by •  Reading project documents, •  Interviewing project members, •  Use of online web tool

Start with the big picture (the ISC hierarchy).

Discovery could take 1-‐2 months (depending on complexity of project and availability of required information.)

Particularized Integrated Simulation Capability Hierarchy for Waste IPSC

The Landmark Glass-‐Brine Chemical Equilibrium Capability

Particularized VU Hierarchy for Glass Brine LSC

VE

MMD CV Val

RP/EQ36

Code-‐to-‐Code S vs. pH Narrative

& Claim Verify I/O

2. Review the Hierarchy

with modelers & developers for accuracy & completeness.

3. Particularizing the Placeholders Each VU entity within the hierarchy contains placeholders for VU data. Pre-‐defined Placeholders for Data (required to define hierarchy) •  Entity name •  Entity version •  Entity date •  Pointer to parent entity •  Number of child entities •  Pointers to child entities •  Pointer to corresponding item in central repository Example Supplementary Placeholders (vary with entity type) •  Simulation Ensemble points to computer platform that executed the simulations •  VU objective points to related local math model •  CV pre-‐simulation entity points to EQ3/6 code and documentation •  Validation element points to the experimental data •  VU Elements point to PCMM-‐based Questions

VU Elements contain pointers to PCMM-‐based Questions & Answers

PCMM asks many due-‐diligence questions. The particularized hierarchy allows these questions to be asked within a clear context. Example. Math Model Development Element (within the context of VE, Glass-‐Brine) 1.  What is the physical phenomenon? 2.  What is the global math model? 3.  What is the science basis for this model? 4.  What are the limits of applicability of this model? 5.  Is this model appropriate for the intended use? 6.  What are the local math models for glass-‐brine dissolution? 7.  Are the local models non-‐phenomenological? 8.  Describe the local models. 9.  Why was the Rimstedt model chosen? 10.  Have the local models been assessed for accuracy? 11.  Were the models calibrated?

During Accumulation….

Example. Math Model Development Element (within the context of VE, Glass-‐Brine) 1.  What is the physical phenomenon? Ans. Chemical Equilibrium for glass-‐brine solub 2.  What is the global math model? Ans. Gibb’s Free Energy (see ref.) 3.  What is the science basis for this model? Ans. It is well established (see ref.) 4.  What are the limits of applicability of this model? Ans. Null 5.  Is this model appropriate for the intended use? Ans. Null 6.  What are the local math models for glass-‐brine dissolution? Ans. Seven species for SiO2, etc. 7.  Are the local models non-‐phenomenological? Ans. Yes, because they are curve fits to data. 8.  Where is the data? Ans. In papers (ref. 1, ref. 2) 9.  Describe the local math models. Ans. Null 10.  Why was the Rimstedt model chosen? Ans. Null 11.  Have the local models been assessed for accuracy? Ans. Yes (need a ref.) 12.  Were the models calibrated? Ans. Yes, see papers. 13.  Where are the curve fits? Ans. In the Yucca Mtn. Database

4. Placeholder Review

Placeholders reviewed for accuracy, completeness, and logical consistency. Creates a contract between the people who will fill the place-‐holders with VU data and the people who will manage the VU data within the central repository. Discovery Process ends here. We are ready to accumulate VU data (=evidence) in an organized fashion, as well as perform interim assessments using the data.

The Discovery Process Addresses the Requirements

Discovery Planning, Sequencing, Terminology, Context, Initiation, Consonance, Concurrent, Practical, Transferable, Deliberate, Multiple-‐Use, Transparency, Records, VU Requirements, Traceability

Planning. If discovery occurs early within the project, the process defines the hierarchy and placeholders, which is a form of planning. Terminology. The hierarchies (both abstract and particular) standardize the VU terminology on the project. Context. Hierarchies are very effective in providing context. Initiation. The Discovery process may be initiated at any time during the project (the sooner the better) since it describes the end-‐result, not only what exists at the moment. Consonance. The hierarchy incorporates PCMM elements, sub-‐Elements, and questions. Transparency. Mutually agreed-‐upon Evidence presentable to a wide audience. Web-‐based Views of Project that many can access.

5. A Web-‐Based Discovery and Assessment Tool

Automatic Hierarchy Drawing Tool (front-‐end) with Database (back-‐end) Uses: •  Online customization of the Pre-‐defined Hierarchy and Place-‐holders, •  Efficient engagement of the Modelers and Developers, •  Standardize terminology, provide context, •  Provide Transparency, Status, •  Facilitate Dialogue, •  Upload data to fill place-‐holders, •  View Assessments, and •  Communicate with managers, stakeholders

Landmark Glass-‐Brine Chemical Equilibrium Simulation Capability Hierarchy

G

Glass-‐Brine Chem Eq LSC

G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE

MMD CVER SVER UQ MMV

CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub

EQ3/6 Design: C-‐C

Verify I/O Analysis: S vs. pH

Narrative

Sim-‐1 Sim-‐2 Sim-‐N


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


Glass-‐brine solub: SiO2 (aq), SiO2 (am), Na+, Cl-‐, OH6-‐, H+, HSiO3

-‐


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative



G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


G-‐B Solub Driver, version 3.2.1 VCS Solver, version 2.1.2 Workstation A 11/26/11


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative



G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


How were the components in the solub chosen? What is the science basis for the local model? What are the limits of applicability? Is this model appropriate for the intended use? Is the local model non-‐phenomenological? Were the models calibrated? What data was used in the calibration? What is the local model form? Where are the curve fits?


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


How were the components in the solub chosen? Null What is the science basis for the local model? See ref X What are the limits of applicability? Null Is this model appropriate for the intended use? Null Is the local model non-‐phenomenological? Yes (curve fits) Where are the curve fits? In Yucca Mtn Thermo-‐database Were the models calibrated? Yes What data was used in the calibration? See papers by X, Y, Z. What is the local model form? Null


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative



G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


Pointer-‐based Assessment

0

0.5

1

1.5 % ptrs filled


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


PMI-‐based Assessment Input: domain of applicability, the experiments, knobs, the predictions Output: Predictive Maturity Index


G


G-‐B Solub Driver

Cantera VCS Solver

GMM: Gibbs’ FE

GMM Data: Yucca TD

VU Objective: VE


CVER Sim Ens

SVER Sim Ens

MMV Sim Ens

G-‐B Solub



Narrative


PCMM-‐based Assessment Input: descriptions of activities & methods Output: Radar Plots, Due Diligence Metric

0

1

2

3 RGF

PMMF

CVER

SVER

VAL

UQ

Summary & Future Work

Summary •  Discovery Process Design vetted against the Waste IPSC.

•  The Design incorporates features from PCMM.

•  Hierarchy provides a (sorely needed) context for VU.

•  The Discovery Process is not a recipe (requires VU expertise).

•  The hierarchies can incorporate sub-‐continuum, continuum, and PA levels of fidelity.

•  Organization of central repository mirrors the hierarchy, placing everything under configuration control.

•  Discovery process is mainly about re-‐organizing to plan, integrate, accumulate, assess, and disseminate VU results within large simulation projects. Future Work 1.  Describe the Accumulation and Assessment Processes.

2.  Develop a web-‐based VU Discovery & Assessment tool. 3.  Apply to as many projects as possible in order to test and refine this approach.

enabling(vu(assessments(powers/vv.presentations/knupp.pdf ·...

Documents