A Modelling Framework:

Supporting Evidence-Based Decisions

ModSimWorld

Montreal, Quebec June 8-9, 2009

2

Main MessagesMain Messages

• Models are needed to understand and predict the behavior of complex systems.

• Models are needed to fulfill an agency’s mandate and support its core business.

• Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability.

• Models are needed to understand and predict the behavior of complex systems.

• Models are needed to fulfill an agency’s mandate and support its core business.

• Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability.

Models should be used wisely

3

OutlineOutline

I. Underlying Concepts

Scientific underpinning

II. Decision Guide

Decision making

III. Glossary

Common understanding

I. Underlying Concepts

Scientific underpinning

II. Decision Guide

Decision making

III. Glossary

Common understanding

4

About ModelsAbout Models

• What are they?

– Simplified representations of reality.

– Transform data, information, and knowledge into outputs.

• Why do we use them?

– Reality is too complex

– Experiments are infeasible

– Predict consequences

– Increase understanding

• What are they?

– Simplified representations of reality.

– Transform data, information, and knowledge into outputs.

• Why do we use them?

– Reality is too complex

– Experiments are infeasible

– Predict consequences

– Increase understanding

....

Nonaka (2000)

Concepts

5

What is a Framework?What is a Framework?

“Structural outline of the components of an organization, system, or process and the relationships among them.”

Understanding Knowledge Services NRCan (2006)

Concepts

6

Framework ObjectivesFramework Objectives

• Support needs-driven and science-driven analysis.

• Promote dialogue among modelers, managers, & users.

• Reduce wasted time, effort, & money.

• Provide a basis for planning and action.

• Document and justify decisions.

• Support needs-driven and science-driven analysis.

• Promote dialogue among modelers, managers, & users.

• Reduce wasted time, effort, & money.

• Provide a basis for planning and action.

• Document and justify decisions.

Concepts

7

Framework DesignFramework Design

• Reflect modelling, management, and user perspectives.

• Balance efficiency and effectiveness with cost and effort.

• Applicable to both demand and supply approaches to modelling.

• Applicable to both logical and computational models

• Reflect modelling, management, and user perspectives.

• Balance efficiency and effectiveness with cost and effort.

• Applicable to both demand and supply approaches to modelling.

• Applicable to both logical and computational models

Concepts

8

Different PerspectivesDifferent Perspectives

What developers proposed

What managers funded

What stakeholders wanted

What users needed

Concepts

9

Supply & DemandSupply & Demand

Concepts

Supply: I have a model that solves your problem.

Demand: I have a problem that needs a model.

10

Modelling SystemModelling System

External Models

DevelopDevelop

Nature, Society

Internal Models

UseUseManageManage

Lost Models

ShareShare

PreservePreserve

Knowledge Management

Concepts

11

Modelling ProcessModelling Process

Modelling combines science & computers; judgement & experience; insight & intuition.

• Principles: effort, simple, data, knowledge, transparent, understandable.

• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.

• Development: techniques are well-understood; management less understood and practiced.

• Use: Decision making under uncertainty, unknown elements, outcome probabilities.

Modelling combines science & computers; judgement & experience; insight & intuition.

• Principles: effort, simple, data, knowledge, transparent, understandable.

• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.

• Development: techniques are well-understood; management less understood and practiced.

• Use: Decision making under uncertainty, unknown elements, outcome probabilities.

Concepts

12

Systems HierarchySystems Hierarchy

Data

ModelsDecision SupportInformatio

nKnowledge

ManagementManagement

Mandate

Business

Policies

Processes

Concepts

13

Data Data

A model and its data are inseparable; they succeed or fail as one.

– Data Needs: Situation may involve nature, the system, and/or intervention.

– Sampling: Statistics are essential to determine how much data is needed.

– Source: Ownership? Use rights? Privacy & security concerns?

– Scale: Time, space, and process scale must match the situation.

– Quality: Level of accuracy, detail, and completeness are needed?

A model and its data are inseparable; they succeed or fail as one.

– Data Needs: Situation may involve nature, the system, and/or intervention.

– Sampling: Statistics are essential to determine how much data is needed.

– Source: Ownership? Use rights? Privacy & security concerns?

– Scale: Time, space, and process scale must match the situation.

– Quality: Level of accuracy, detail, and completeness are needed?

Concepts

14

OutputsAcquisition ProcessingStorage

Knowledge

Organization

Society

Environment

Events

Economy

Channels

Access Search Retrieval

InterfaceProcessingDatabase

Inputs Audience UseMedia

InterfaceSystemDataModel

Outputs

Concepts

Information SystemInformation System

Interoperability IntegrationAvailability Utility

151. Common

•Flow-through (-)•Fixed (1:1)•Planning•Mechanistic•Automated•Data, facts

2. Complicated •Feedback•Linear (1:n)•Mathematics•Deterministic•Certainty•Explicit knowledge

3. Complex •Predictive feedback (+)•Non-linear (1:?)•Simulation•Stochastic•Uncertainty •Tacit knowledge

4. Chaotic

•Emergent•Disorganized•Scenario analysis•Mental•Reaction•Intuition

System:Behavior:Approach:Model:Decision:Basis:

Concepts

Models and Knowledge Models and Knowledge

16

OutlineOutline

I. Underlying Concepts

Scientific underpinning

II. Decision Guide

Decision making

III. Glossary

Common understanding

I. Underlying Concepts

Scientific underpinning

II. Decision Guide

Decision making

III. Glossary

Common understanding

17

Decision Guide - HierarchyDecision Guide - Hierarchy

• Phase: (3) demand, supply, project

• Stage: (7) approach, design, establish, develop, evaluate, implement, conclude

• Step: (34) screening, problem definition, suitability, knowledge, data

• Consideration (132): recurrence, importance, problem space, existence

• Phase: (3) demand, supply, project

• Stage: (7) approach, design, establish, develop, evaluate, implement, conclude

• Step: (34) screening, problem definition, suitability, knowledge, data

• Consideration (132): recurrence, importance, problem space, existence

18

Design

Acquire Data

Generate Knowledge

Approach

Out

Evaluation

IssueStart: (Demand)

Implementation

Outputs

Conclusion

End

Development(Modeller)

(Manager)

(User)

D1

D2

3

Applicability

S2

ModelStart (Supply) identification

S1

5

7

6

Establishment

4

(Manager)

(All)

Decision Guide - StagesGuide

19

Issue Model

Demand Supply

Project

End

Decision Guide PhasesDecision Guide Phases

Guide

20

Start (use)

Demand-drivenbackward chaining, closed question

Model

Supply-drivenforward chaining, open question

Start (model) Uses

Supply & DemandSupply & Demand

Guide

21

Issue

D1. Approach

D2. Design

Development

Acquire Data

Generate Knowledge

Out

Demand PhaseDemand Phase

Guide

22

Issue

Initial Screening

Problem Definition

Suitability

Knowledge Evaluation

Data Availability

Design

Recurrence ImportanceProblem spaceExistence

Business FunctionIntended use

Time available Situation

NeedsExistingGap

NeedsAttributesAccessibilityProcessing

Continue

Continue

Continue

Continue

Continue

Below threshold

Can’t define

Unsuitable

Excess gap

Inadequate

Generate?

Acquire ?

Yes

Yes

Out

No

No

D1.1

D1.2

D1.3

D1.4

D1.5

Guide Approach Stage

23

Decision Guide ConsiderationsDecision Guide Considerations

• Explains the question.

• Classify a situation or write a short description.

• Complete a statement template.

• Decide where to go next.

• Not a cookbook to be followed without interpretation.

• Compliments experience & judgement; doesn’t replace them.

• Explains the question.

• Classify a situation or write a short description.

• Complete a statement template.

• Decide where to go next.

• Not a cookbook to be followed without interpretation.

• Compliments experience & judgement; doesn’t replace them.

Guide

24

Development

Model

S1. Identification

S2. Applicability

Evaluation

Conclusion

Supply PhaseSupply Phase

Guide

25

Existing Model

Knowledge base

Suitability

Evaluation

Specification criteria (4)

Knowledge criteria (3)

Business lineFunction Development

S2.2

S2.3

S2.4

IdentificationSearchDescription

S1

modify

Data AvailabilityData criteria (5)

Data Acquisition

inaccessible

Conclusion

unsuitable

unjustified

Specifications

S2.1

Development criteria (3) Development

S2.5

Applicability Stage

Guide

26

Project

Phase

End

Design

Applicability

4. Development

5. Evaluation

6. Implementation

7. Project Conclusion

Out3. Project Establishment

Situation

Guide

27

Project Establishment

Hierarchy Relationships Indicators Review

Logic Computation Debugging Review

Attributes ConsistencyReview

Uncertainty Representation Review

Conceptualization

Construction

Evaluation

Verification

Validation

Inventory

exit

exit

exit

exit

4.2

4.3

4.4

4.5

continue

continue

continue

continue

Interaction

Awareness Understanding Consensus

4.1

Development Stage

Guide

28

OutlineOutline

I. Underlying Concepts

Scientific underpinning

II. Decision Guide

Decision making

III. Glossary

Common understanding

I. Underlying Concepts

Scientific underpinning

II. Decision Guide

Decision making

III. Glossary

Common understanding

29

GlossaryGlossary

• Background (introduction, methods, references).

• Taxonomy (organization, nature, risk analysis, content, modelling, concepts).

• Definitions - 650 terms from six sources.

• Links to taxonomy and related terms.

• Background (introduction, methods, references).

• Taxonomy (organization, nature, risk analysis, content, modelling, concepts).

• Definitions - 650 terms from six sources.

• Links to taxonomy and related terms.

Glossary

30

Sample DefinitionSample Definition

Model: Abstract and simplified construct or representation of reality in the form of a pattern, description, or definition that shows the essential structure, relationships, and workings of a concept, process, or system.

(see modelling approach, function, modelling methods, process, relationship, representation, system)

Glossary

31

Modelling Framework:

• Supports an agency’s business

• Facilitates horizontal integration

• Minimizes waste & inefficiency

• Maximizes likely success

• Documents & justifies decisions

“Using a clear blueprint first prevents chaos latter.”

Carla O’Dell (1998)