Towards Appropriate Selection of Analysis Tools

and Methods

Overview

• Research to determine the appropriate tools and methods for effective application within a broad range of analysis categories.

• Longer term goal to complete whole activity– Summarised later

• This paper focuses on – categorisation of study and problem types

• Initial activities description

– Discussion on derivation of metrics • for assessing the value of study methods and tools vs

problem and study types.

Aim

• Overall Aim and Introduction• The aim of the whole research is to determine the

appropriate tools and methods for effective application within a broad range of analysis categories.

• The work will– investigate and analyse a broad range of differing analysis study

types and – research optimal methods and tools for utilisation within these

types. • The analysis study types

– will be example operational and system analysis studies – investigated to determine

• the approaches used, • the constraints and issues associated with them and • the overall success and outcomes.

Activities to date

• Initial Literature Review– OR Society– College Library– MORS Material

• Methods Handbook• Model Taxonomy

– SME Contact

• No evidence of complete approach • But

– Problem Structuring Methods – Paradigms

Problem Types

• Obeng’s Problem types– Fog – Open– Movie – Semi Open– Quest – Semi Closed – Paint by Numbers – Closed

• Mike Pidd– Problems– Puzzles– Messes

• Obeng preferred– More refined

– Organisational Level• Corps• Div• Bn

– Combat Type• High Intensity• PK• Hybrid

– Timescale of Battle• Hour• Day• Days

Proposed Factors affecting the problem

• The Question– Cost Effectiveness Comparison– Force Mix– Battlefield Effectiveness– Capability Gap– Tactics comparison– Numbers

• The Customer

• Soft Issues

Proposed Factors affecting the problem II

Factors affecting the solution• Skill set• Analysis Timescales• Analysis Budget• Model Types

– Hard– Soft– etc

Factors affecting the solution• Skill setDefence EnvironmentAnalysis within Acquisition and Systems Process

UsesCapability AnalysisAcquisition Processes

Business CaseBOICOEIA

Operational SupportWargamingSeBATraining

Systems ApproachValidation

Factors affecting the solution – Skill set

Tools and Methods OR Techniques

Soft MethodsSpreadsheetsMathematical ModellingEngineering Modelling

Modelling and SimulationVirtualLiveConstructiveStochasticDeterministic etcGamesSimulationsAnalytical methods – lanchester etcSEs

Model Development Process

Factors affecting the solution – Skill setDesign of Experiments

Analysis and interpretation of resultsDesign of Experimental Studies

ConfidenceVerificationCommunication and Understanding

Model Components Networking/ Distribution of SimulationHCIProtocolsBehavioural Representation/Intelligent Systems

DataData Analysis & Databases

Combat Sys, TerrainScenariosSoftware Development

Skill set

• Level of Skill required for differing roles– Senior Decision Maker– Problem Owner– Study Leader– Analyst– Developer– Player– Project Manager– Technical Manager

Success Criteria - Measuring how well the method(s) Measuring how well the method(s) fits the problemfits the problem• A. Answer obtained

– Within time/budget/other constraints

• B Answer accepted – Validated

• C Answer Used

• D Repeat work

Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem

• A. Answer obtained– Within time/budget/other constraints– Scale

• On budget/time• Ahead of budget/time• 20% slip• 50% slip• 100% slip

Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem

• B Answer accepted – Validated– By peer/QA review– By acceptance– Accepted after changes

Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem

• C Answer Used– By one user– By several– Citations

Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem

• D Repeat work– Once– Extension– Frequent

Next StepsStudy Examples - • Drawn from industry and Dstl where possible. • For each study

– What was done, – what techniques used – resources used – Output measured

Study Analysis• Studies categorised according to the range of criteria or dimensions,

– Problem type – Effort levels available– Timescales– Tools available– Skills available– Audience/Customer– Other Soft issues – political

Comparative Analysis of UK MoD Guidelines for conduct of Analysis

Outline• Evaluation of Study Approaches

– Identification of a full range of approaches including tools and methods for potential use within these types of analyses.

– Identification of the approaches and the associated tools and methods used within each example study.

– Derivation of metrics for assessing the value of each of approaches and the tools and methods in achieving successful outcomes in the studies.

• Derivation of Optimising Approaches– Derivation of theoretical/initial rule sets for identifying

the optimal approaches and tools and methods for use within study categories.

Outline• Analytical Review of Example Study

– Testing of the initial theoretical rule sets against the example studies to determine where non-optimal methods were applied and identification of the improved approaches which might have been employed and the impact these would have had on the study outcome.

• Test phase using New Studies– Experimental phase using real study examples to

develop approaches and test the hypothesised rule sets. • Conclusions

– Identified optimised approaches including the risk areas and constraints and the ‘region of validity’ of the conclusions.

Towards Appropriate Selection of Analysis Tools

and Methods