vartan – validation reporting templates jürgen teutsch, nlr caats workshop, 16-feb-2006,...
TRANSCRIPT
VARTAN – Validation Reporting Templates
Jürgen Teutsch, NLRCAATS Workshop, 16-Feb-2006, Lanzarote
VARTAN – Project Objectives
VARTAN is concerned with...
Definition of Validation Reporting Templates in accordance with European Operational Concept Validation Methodology (E-OCVM)
Analysis of E-OCVM
Refinement of Life Cycle Model and Case-based View
Context of templates in Life Cycle Model
Study for automated transfer of template contents into the Validation Data Repository (VDR)
VARTAN – Analysis Issues
Performance expressed in structured way?– Key Performance Areas (PRC/ICAO)– Key Performance Indicators
Evidence with right level of quality?– Validity and accuracy of results (and objectives)
Right levels of detail at different decision-making levels?– Strategy, programme, project, and experiment level
Information needs of all decision makers addressed?– Airlines, service providers etc.
E-OCVM comprehensible for less proficient user?– Simplify methodology for different E-OCVM views
VARTAN – Results Overview
Suggested (or similar) refinements are envisaged for implementation in the updated methodology:
Improve comprehensibility of Life Cycle Phase Model Refine case-building process description Describe and give examples of possible stakeholder needs and
associated stakeholder analysis issues Consider confidence in results
Activities for preparation of document standards with automation:
Build templates at different planning levels and determine triggers for reporting
Provide automation example for experiment-level validation reporting template
Describe process for working with result templates
VARTAN – Suggested Life Cycle Refinements
Plan
Do
Check
Act
Define Goals and Cases
Decide onWay to Proceed
Carry OutExperiments
Study Results of Experiment
Deming Cycle for ATM Concept Validation
I mplementedProcedure or ProcessI nitial I dea
Scope Operability Performance I ndustrialisation I mplementation
V1 V2 V3 V4 V5
Concept Principles
Techniques•High-level Modelling•Fast-time Simulations•Expert J udgement
KPA/ Cases•Acceptability•Access and Equity•Capacity and Delay•Cost Efficiency•Efficiency•Environment•Flexibility•Interoperability•Safety•Security
Expected Results•Stakeholder I ssues•Time Planning•Scenario Definitions•Benefit Mechanisms•Potential Benefits•Showstoppers•Risks
Feasibility
Techniques•Requirement Capturing•Rapid Prototyping•Operational Simulations•Procedure Analysis•Scenario Evaluation
KPA/ Cases•Acceptability•Efficiency•Human Factors•Interoperability•Predictability•Safety
Expected Results•System Operability•User Operability•Procedure Description•Non-nominal Cases•HMI Prototype•Workload Estimates•Concept Baseline•Risks and Mitigation
Concept I ntegration
Techniques•Real-time Simulations•Live Trials•Shadow-mode Trials•Cost-Benefit Analyses•Safety Analysis
KPA/ Cases•Acceptability•Capacity and Delay•Cost Efficiency•Efficiency•Environment•Safety
Expected Results•Performance Data•Stakeholder Approval•Acquisition Process•Detailed Procedures•Detailed Concept
I ndustrialisation
•Procedure Approval•Acquisition•Factory Test
I mplementation
•In-Service Test•In-Service
VARTAN – Suggested Case-building Refinements
From KPA and KPI to case definition: which cases are necessary for decision making?
Interrelationships between defined areas (cases)
Differences in focus from V1 to V3; from identification of showstoppers (V1) to implementation decision (V3)
Discern Verification and Validation (especially in V2)
Different levels of detail in cases: what level of detail is necessary for decision making per lifecycle phase?
VARTAN – Proposals for Stakeholder Needs
Conflicting stakeholder objectives/goals or interests– Goal balancing– Harmonisation of concepts and ideas
Different stakeholder information needs– Different levels of reporting
Different levels of quality (validity/accuracy) of results – Use of standard methods to increase (or find a
common level of) validity
VARTAN – General Template Considerations
• Pre-defined introductory part (background, context, scope)
• Summary of planning document
• Differences between planning and actual achievements
• Report confidence in results
• Objective should be related to a certain KPA
• Correlation between results / conclusions / recommendations and objectives (from high-level to low-level to hypotheses)
VARTAN – Planning and Reporting LevelsPlanning Levels
Life Cycle Phase
Strategic Level (Decision Makers)Set overall and intermediate goalsDetermine timeframeDetermine Key Performance Areas (KPA)Solve stakeholder issuesPrioritise ATM needs
Programme Level (Programme Managers)Address particular intermediate goal(s)Focus on particular KPAIdentify interrelations between KPAsDetermine KPIs inherent to KPAsDefine cases necessary to support goals
Project Level (Project Managers)Define validation strategyImplement case(s)Identify hypotheses based on goals/objectivesProduce scenarios and experiment planProduce analysis plan
Experiment Level (Experiment Leader)Prepare platform or facilityPerform pre-exercise testing and trainingConduct validation experiment(s)Perform analysisStart reporting cycle on lowest level
Experiment Level Results Report
Summary of planningExperiment executionExperiment resultsExperiment analysisConclusionsRecommendations
Programme Level Results Report
Goals and ObjectivesKPA, KPI and casesProgramme resultsProgramme analysisConclusionsRecommendations
Project Level Results Report
Project ObjectivesValidation strategyImplemented case(s)Project resultsProject analysisConclusionsRecommendations
Continue ?
Continue ?
Continue ?
YES
YES
NO
NO
NO YES
Strategic Level Planning Report
KPAATM ProblemsATM NeedsPrioritisationOp. ImprovementsExpected BenefitsProgramme Outline
Programme Level Planning Report
ATM NeedsOperational ConceptOI ClustersProgr. ObjectivesProgr. StakeholdersProgr. DependenciesFactors of InfluenceTools and TechniquesProgramme OverviewResourcesDeliverablesRisks
Project Level Planning Report
Project ContextProj. ObjectivesProj. StakeholdersProj. DependenciesIdentified CasesFactors of InfluenceTools and PlatformsProject Overview ResourcesDeliverablesRisks
Experiment Level Planning Report
N/A
Considered to betoo prescriptive
Experiment Level
Strategy Level
Programme Level
Project Level
VARTAN – Planning and Reporting Levels
Planning Levels
Life Cycle Phase
Strategic Level (Decision Makers)Set overall and intermediate goalsDetermine timeframeDetermine Key Performance Areas (KPA)Solve stakeholder issuesPrioritise ATM needs
Programme Level (Programme Managers)Address particular intermediate goal(s)Focus on particular KPAIdentify interrelations between KPAsDetermine KPIs inherent to KPAsDefine cases necessary to support goals
Programme Level Results Report
Goals and ObjectivesKPA, KPI and casesProgramme resultsProgramme analysisConclusionsRecommendations
Continue ?
Continue ?
YES
NO
NO YES
Strategic Level Planning Report
KPAATM ProblemsATM NeedsPrioritisationOp. ImprovementsExpected BenefitsProgramme Outline
Programme Level Planning Report
ATM NeedsOperational ConceptOI ClustersProgr. ObjectivesProgr. StakeholdersProgr. DependenciesFactors of InfluenceTools and TechniquesProgramme OverviewResourcesDeliverablesRisks
VARTAN – Planning and Reporting Levels
Project Level (Project Managers)Define validation strategyImplement case(s)Identify hypotheses based on goals/objectivesProduce scenarios and experiment planProduce analysis plan
Experiment Level (Experiment Leader)Prepare platform or facilityPerform pre-exercise testing and trainingConduct validation experiment(s)Perform analysisStart reporting cycle on lowest level
Experiment Level Results Report
Summary of planningExperiment executionExperiment resultsExperiment analysisConclusionsRecommendations
Project Level Results Report
Project ObjectivesValidation strategyImplemented case(s)Project resultsProject analysisConclusionsRecommendations
Continue ?
YES
NO
Project Level Planning Report
Project ContextProj. ObjectivesProj. StakeholdersProj. DependenciesIdentified CasesFactors of InfluenceTools and PlatformsProject Overview ResourcesDeliverablesRisks
Experiment Level Planning Report
N/A
Considered to betoo prescriptive
relevantVDR information
VARTAN – Unexpected Results
Validation Reporting Triggers (Exceptional triggers)
Exceptional situations with unexpected results or events might lead to a re-evaluation of the current activity
Need to report to higher levels immediately and therefore trigger a special report
Important: there must be a way to stop the validation cycle and reconsider the complete approach
Generally, such reporting is part of a good management approach (not part of E-OCVM guideline structure)
VARTAN – Validation Reporting Process (1/3)
Computer with
MS-Word Client
I nternet
Validation ReportingTemplates
VDR Website
VARTAN – Template Life Cycle
Empty Template
Accepted Template
Prepared Template
Empty Template
Accepted Template
Prepared Template
Filled-in Template
Received Template
Approved Template
Filled-in Template
Received Template
Approved Template
Author accepts data
VDR Data Collectoradds data
Author does notaccept data
VDR Administratorapproves template
VDR Administratorrejects template Author
transmits template
Authorfills in template
Authorupdates template
VARTAN – Validation Reporting Process (2/3)
VDR(Oracle Database)
Computer with
MS-Word Client
I nternet
Filled-in Templates
VARTAN – Template Life Cycle
Empty Template
Accepted Template
Prepared Template
Empty Template
Accepted Template
Prepared Template
Filled-in Template
Received Template
Approved Template
Filled-in Template
Received Template
Approved Template
Author accepts data
VDR Data Collectoradds data
Author does notaccept data
VDR Administratorapproves template
VDR Administratorrejects template Author
transmits template
Authorfills in template
Authorupdates template
VARTAN – Validation Reporting Process (3/3)
Validation Expert
VDR Administrator
TCP/IP
HTTP
Web Services – SOAP (XML)
Apache Axis
Java Servlet
VDR (Oracle)
VDR Interface
SQLStatements
VARTAN – Template Life Cycle
Empty Template
Accepted Template
Prepared Template
Empty Template
Accepted Template
Prepared Template
Filled-in Template
Received Template
Approved Template
Filled-in Template
Received Template
Approved Template
Author accepts data
VDR Data Collectoradds data
Author does notaccept data
VDR Administratorapproves template
VDR Administratorrejects template Author
transmits template
Authorfills in template
Authorupdates template
VARTAN – Template Contents:Help Text
VARTAN – Template Contents:Introduction
1. Introduction
1.1 Project/Experiment Background and Context
1.2 Concept Overview
1.3 Purpose of Document
1.4 Scope and Structure of Document
1.5 References
1.6 Acronyms
VARTAN – Template Contents:Project Level Strategy and Planning
2. Summary of Project Strategy and Planning
2.1Stakeholder Requirements
2.2Supporting Cases
2.3Expected Project Outcomes and Objectives
2.3.1 Description of Expected Project Outcomes2.3.2 Summary of Expected Project Outcomes2.3.3 Description of Project Objectives2.3.4 Summary of Project Objectives
2.4Decision Processes in Experiment Definition
2.5Experiment Context and Planning
2.6Analysis Strategy
VARTAN – Template Contents:Project Level Conduct and Results
3. Conduct of Validation Project3.1Executed Experiments3.2Deviations from the Planning
4. Results of Experiments4.1Results of Different Experiments per Key Performance Area4.2Confidence in Results of Different Experiments
4.2.1 Quality of Results of Different Experiments
4.2.2 Significance of Results of Different Experiments
4.3Unexpected Project Results
5. Consolidation of Project Results5.1Consequences of Confidence in Experiment Results5.2Consequences of Experiment Results for Project Objectives
5.2.1 Interpretation of Results with respect to High-level Objectives
5.2.2 Interpretation of Results with respect to Key Performance Areas
6. Conclusions and Recommendations for Programme Level6.1Conclusions6.2 Recommendations
VARTAN – Template Contents:Experiment Level Strategy and Planning
2. Summary of Experiment and Strategy Planning
2.1 Expected Experiment Outcomes, Objectives and Hypotheses
2.1.1 Description of Expected Experiment Outcomes
2.1.2 Summary of Expected Experiment Outcomes
2.1.3 Description of Experiment Objectives
2.1.4 Summary of Experiment Objectives
2.1.5 Description of Experiment Hypotheses
2.1.6 Summary of Experiment Hypotheses
2.2 Choice of Metrics and Measurements
2.3 Choice of Methods and Techniques
2.4 Scenario Specifications
2.5 Experimental Variables and Design
VARTAN – Template Contents:Experiment Level Conduct and Results
3. Conduct of Validation Exercise Runs3.1Experiment Preparation3.2Executed Experiment Schedule3.3Deviations from the Planning
4. Experiment Results4.1Measured Experiment Results4.2Confidence in Experiment Results
4.2.1 Quality of Results of Experiment
4.2.2 Significance of Results of Experiment
4.3Unexpected Experiment Results
5. Analysis of Experiment Outcomes5.1Analysis of Outcomes on the Basis of Determined Hypotheses5.2Analysis of Consequences of Outcomes for Experiment Objectives
6. Conclusions and Recommendations for Project Level6.1Conclusions6.2 Recommendations
VARTAN - Template Automation: Basic Information
VARTAN - Template Automation: Standard Structure
VARTAN - Template Automation: Entry Fields
VARTAN – Outlook into the Future
Data definition and requirements for (static) document data and pre-formatted data
Data structures for VDR entry including mutual relations
Encryption standards for usernames and passwords
Data protection modes and protected sections
List of all property fields, bookmarks and macros in use by the automated solution to avoid conflicts
Define rollback mechanisms for all automated changes
Templates available at VALUG website: http://www.eurocontrol.int/valug/
Website offers contact information and is available for feedback on methodology and standard methods
Questions