ncw/nec metrics workshop 17-19 december, 2002

NCW/NEC Metrics Workshop17-19 December, 2002

Report from Work Group 4

Evidence and Assessment

Work Group 4 - Objectives

• To discuss ways to develop a knowledge base relevant to NCW/NEC metrics.

• Key tasks:– Identify and evaluate the utility of sources of evidence;

– Identify the “meta-data” needed to organize and integrate the knowledge base.

Work Group 4 - Strategy

• Journalist’s questions– Who? Why? What? When? Where? How?

• Who? - Identified range of knowledge base user groups.

• Why? - For each, identified why they need the knowledge...

• What? - …and what types of knowledge they need.

• Used NCW Framework to identify NCW-specific knowledge needs.

• When? Considered qualities of evidence for different needs.

• Where? Considered qualities provided by different type of evidence source.

• How? Considered constraints on format/management of KB.

Themes for Evidence Workgroup• Source of evidence

– history, experiment, model, judgement,..

– Multi-national availability

• What you want evidence for?– persuade participation/commitment,

– inform investment (BOI, individual cases, substantiate/test claims),

– inform planning (strategic, campaign, mission),

– generate usable heuristics.

• What are the claims of NCW/NEC?

• Nature of evidence– Strength,credibility, reliability, fidelity, specific/ generic, cost to gather

– Risk of deciding the “wrong” thing

– Types of evidence

• What is this “knowledge base” (and why do we need it)?– What type of knowledge? (claims, evidence, …) - part of a process

– What characteristics does the KB have.

• Who, What, When, Where, Why?

Who? and Why?• Overall WHY? - Justify decisions, generate of commitment to action.

• Policy and Force Structure folks– WHY? - Mission/Context drivers for capabilities. Asymmetric threats. Policy

effectiveness. Economics.

• Materiel folks – WHY? - Where are the new needs? What technologies? Trade-off assessment,

effectiveness - against baseline of current/legacy. Efficiency/cost.

• Doctrine developers – WHY? - Limits of capability. What is effective? Areas for promising

experimentation.

• Operators – WHY? - What have we discovered? ( heuristics). Under what conditions was this

knowledge derived? (reliability). What doesn’t work?

• Trainers– WHY? - What technologies are in-service or emerging? Awareness of current

capabilities, concepts of use, emerging doctrine. [National differences here?]

What?

• Evidence about the nature of NCW/NEC itself.

• Evidence to:– Inform the generation of ideas, goals, concepts.

– Justify decision and persuade others to endorse/participate.

– Enable bringing into being, engineering, production.

Policy and Force Structure

• WHY - Mission/Context drivers for capabilities. Asymmetric threats. Policy effectiveness. Economics.

• WHAT - Evidence to inform, justify, enable...

• Mission requirements - What missions? Where/when? Constraints?

• Capability requirements

• Desired force structure attributes, e.g.– Ability to fight for decision superiority

– Speed of command

– Information access + shared awareness

– Dispersed forces in non-contiguous/continuous operations

– “De-massification”

– Elimination of process lines (boundaries and comm chains)

– Elimination of structural lines

– Dynamic, self-synchronisation of forces

– Compressed levels of operation (collapse of the levels)

Materiel folks

• WHY - Where are the new needs? What technologies? Trade-off assessment, effectiveness - against baseline of current/legacy. Efficiency/cost.

• WHAT - Evidence to inform, justify, enable…• Operational requirements (implies goals, threats, contexts, constraints)

• Systemic requirements (addressable accessible entities, level of sophistication of information exchange allowed)

• Capability gaps (implies baseline capability and capability needs, value gap)

• Technologies, etc. to close gaps (maturity, costs, risks, potential synergies)

• People issues (manning, skills, link to training and doctrine/processes)

• Networking concepts (synergies, concepts for exploitation, potential to enable operations/activities, net-readiness requirements)

Doctrine developers

• WHY - Limits of capability. What is effective? Areas for promising experimentation.

• WHAT Thing itself, and Evidence to inform, justify, enable…

• National Military strategy

• Operational concept/concept of operations; joint doctrine

• Organisational structure

• Specific processes: effects based planning, information sharing, subscription lists, post and pull

• Coevolution of doctrine/materiale by experimentation

• TTPs

Operators

• WHY - What have we discovered? ( heuristics). Under what conditions was this knowledge derived? (reliability). What doesn’t work?

• WHAT - Evidence to inform, justify, enable…

– Effectiveness of different NCW methods under different circumstances

– Range of applicability--circumstances under which types of equipment should/should not be used

– Robustness--

– Resource and information requirements for different kinds of actions

– Rules of thumb/stories/heuristics for suggesting methods and judging applicability; execution level of detail

Trainers

• WHY - What technologies are in-service or emerging? Awareness of current capabilities, concepts of use, emerging doctrine. [National differences here?]

• WHAT - Evidence to inform, justify, enable…

– Doctrine, TTP

– Technologies; when to apply, how to use, human impact

NCW Conceptual Framework

AwarenessUnderstanding

Degree of Decision/ Plan Synchronization

Degree of Actions/ Entities Synchronized

Degree of Effectiveness/ Agility

Degree of Information “Share-ability”

Degree of Networking

Force

Quality of Individual Information Degree of Shared Information

Quality of Individual SensemakingShared Awareness

Shared Understanding

Degree of Shared Sensemaking

Operating Environments

Quality of Organic Information

C2 EffectorsValue Added

Services

Quality of Collaborative DecisionsQuality of Individual Decisions

Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

NCW knowledge needs for Trainers

Nothing much








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources


Nothing much

When to apply, how to use different info sources

Nothing much

Nothing much

When/how to exploit sharing capabilities•Consistent ontology, epistemology, context








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources


Nothing much


Factors for“Net-ready people”•Frames of mind, heuristics•Sharing behaviour modes•Knowledge of networks and how to exploit them•Holistic habits, supra-task views

Nothing much

Nothing much

When/how to exploit sharing capabilities








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources


Nothing much


Factors for“Net-ready people”


Nothing much

Nothing much


Factors for effective interaction•Trust, confidence, comfort, … (esp. in computer-mediated interaction)•Ownership of enterprise goals•Feedback and motivation, altruism?•Holistic habits, Basic business rules•Generational changes?








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources


Nothing much




Nothing much

Nothing much


Factors for shared sense-making•Consistent mental constructs•Cultural affinity•Prior interaction experience•Consistent Doctrines

Factors for effective interaction

Factors for collaborative DM•Consistent Doctrines, TTP•Consistent goals/objectives•Will to collaborate•Requisite explicitness•Feedback, confirmation








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources


Nothing much




Nothing much

Nothing much


Factors for shared sense-making

Factors for effective interaction

Factors for collaborative DM

“Gearing” from decision quality to synchronization•What quality factors and how much of each are needed to achieve the behaviour goal








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

NCW knowledge needed for Policy & Force Structure folks








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

Doctrine developers








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

Operators








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

Materiel








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

NCW knowledge needs for Materiel

Nothing much•Individual sensor information that is not shared

•State of legacy network•Info systems/comms•Human co-location•Equipment capability gaps (implies requirements)

•Emerging/available technologies•Cost/capability trade-offs








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

•Information user groups•Security architecture requirements•Interoperation doctrine/TTP•Organisation agility requirements

NCW knowledge needs for Material

•Information service requirements•Driven by TTP? Driving TTP?

•Emerging service concepts•e.g. addressable accessible entities

•Criteria for “net-readiness”•…and for “net-dependency”

•speaks to quality of service and priority for net-readiness








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

•Human factors/design•Non-organic information requirements•Cognitive process requirements









Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources


•Legacy collaboration capabilities•Human factors/design for collaboration

•Collaboration Technologies•Information requirements for collaboration•Collaborative cognitive process requirements








Force








Services


Physical Domain

Social Domain

Information Domain

Cognitive Domain

Quality of

Inter-actions

Information Sources

… ?


When?• Qualities of evidence for different user needs.• Need for good evidence

– Manage the risk of reaching incorrect conclusion or knowledge claims.

• Dimensions of the evidence:– Strength - the extent to which the evidence supports the hypothesis claim

(the probability of the hypothesis given the evidence).• Credibility – the extent to which the decision-maker is predisposed to believe

the evidence based in the source and prior experience with evidence. The extent to which it speaks to his understanding or is coherent with his criteria (or comes from sources he trusts).

• Reliability – reproducibility under insignificantly context changes (e.g., different observer)

• Fidelity – depth of detail or fineness of focus• Specific/ generic – relevance of evidence to wide range of context, robustness

across variable conditions, applicability to the future context

• Case to case comparability• Cost to gather evidence traded with quality

– minimum acceptable quality– maximum effective quality

Qualities of evidence required

Evidence user

Strength Credibility Reliability Specific/Generic

Fidelity Risk of "error"

Policy/ Force Structure

HIGH HIGH HIGH Generic LOW LOW

Doctrine developers

LOW HIGH MEDIUM Generic LOW MEDIUM

Materiel folks

MEDIUM HIGH HIGH Specific HIGH MEDIUM

Operators MEDIUM HIGH HIGH? Specific HIGH LOW

Trainers ? MEDIUM MEDIUM Generic HIGH MEDIUM

Researchers LOW LOW LOW ? HIGH HIGH

Where?

• ALL evidence, of whatever quality, can be used provided one knows the quality

• Sources of evidence– Historical records, personal experiences, synthetic

experiences, expert opinion, controlled experiments, “forensic” analysis.

Qualities of evidence sources

Evidence source

Strength Credibility Reliability Specific/Generic Fidelity Risk of "error"

Historical experiences

LOW LOW MED Specific ? HIGH

Personal experiences (Post deployment hot wash)

MED MED LOW Specific HIGH? ?

Synthetic experiences (modelling, exercises, experiments)

MED MED MED Specific HIGH MED

Expert opinion MED MED MED Generic MED MED

Controlled experiments

MED MED HIGH Specific HIGH LOW

"Forensic" Analysis

HIGH MED MED Generic HIGH LOW

How?

• Network-centric knowledge representation – Knowledge distributed and self-synchronised?

– Maintenance policy by central dictat or enlightened self-interest?

– Using NCW principles in relation to creating/exploiting evidence

– Dissemination

• National differences?

Summary

• Knowledge relevant to NCW serves diverse user communities (e.g., policy makers, force structure, operators, trainers, materiel people, doctrine developers)– Information in the knowledge base can be mapped on to the NCW

framework

– Information needs to be organized to support each user community

– There are technical and national policy issues to build a knowledge base

• Evidence (meta-data) to substantiate the information in the knowledge base is available from multiple sources– Historical experiences, personal experience, synthetic, expert opinion,

controlled experiments, analysis & synthesis

– Information from multiple sources can increase strength

– Required strength of evidence varies with how the information is going to be used

ncw/nec metrics workshop 17-19 december, 2002

Documents

qualities of evidence

type of knowledge

types of knowledge

knowledge base relevant

ncwspecific knowledge

policy effectiveness

claims of ncwnec

nature of ncwnec