EMER: Engineering Simulations for scientific research

EME : 2

Complex systems simulation

• Complex systems comprise component systems that interact with each other and their environment

• Which is itself comprised of complex systems

• Scientific research in complex systems has many challenges:– Observation interferes with the system

– Too many factors and variables for comprehension

– Over-reliance on simplistic models

EME : 3

Example: immune systems

• Immunologists need to understand how diseases attack organisms in order to target therapies– Diseases only develop naturally in living organisms

– Limitations to how disease development can be observed• Kill the subject and work out what has been going on

– Disease progress involves killing 1000s of mice

• Invasive procedures involving nano-cameras or xrays, chemical dyes etc

– Expensive, and possibly interferes with disease itself

• Can simulation provide an alternative experimental subject?– Can a simulation be realistic enough to do real science on?

EME : 4

Models of complex systems

• Simulations are undertaken for different reasons– Developers must be clear about purpose

• Models that mimic and “predict” or project– e.g. mathematical equations that reproduce observed

measurements and statistics

• differential equations

• Models that describe– models made up of components and an environment

• Mathematical individual-based models (IBMs)

• Agent-based models (ABMs)

• All inadequate, but in different ways

EME : 5

Mathematical models to “predict”

• Simulating observed behaviour – e.g. population dynamics using differential

equations• Lotka-Volterra model of predator-prey interaction

• Logistic equation models of population vs resources

– Population-level integrated view

– Analytic rigour

• Good for general trends, but…– Minimal insight to system dynamics

– Intractable if too many parameters

– Many (vague/unrealistic) assumptions

EME : 6

Component-based simulations

• Design components and environment; initialise lots of components; see what happens– Capability to express heterogeneity

– “realistic” interactions

• Good for exploring, but…– Often poorly engineered

– Scalability often limited by inefficient programming or execution

– Mapping between reality and simulation often weak

EME : 7

Commentary on simulation

• Many scientists like the analytic strength of mathematical models of behaviour

• ODEs, PDEs, logistic maps, etc

– But parameter-tweaking is rarely validated properly

• Many scientists distrust computer-based simulations– Because the analogies to real systems are not made clear

• Concerns about component-based simulation can be reduced by working closely with scientists – Clarity over modelling and design assumptions

– Exploration of scientific meaning and assumptions

– Challenged assumptions can lead to new experiments• In the science and/or in the simulation

EME : 8

Complex system simulations: problems

• Simulations seek to achieve behaviours that are observed in real systems– Emergent behaviours should emerge in simulation

• Is a simulation valid just because it gives apparently realistic behaviours?

• e.g. ALife systems often look like the real thing, but the comparison is barely skin deep

– Need realistic behaviour from realistic structure and behaviour

• We need to engineer realistic simulations– Simulations that scientists trust

www.cogs.susx.ac.uk/users/ezequiel/wheeler-etal-2002.pdf

EME : 9

A PRINCIPLED APPROACH TO SIMULATION

The CoSMoS Project, 2007-11

Complex Systems Modelling andSimulation Infrastructure

York, Kent, Abertay, UWE

EME : 10

CoSMoS

www.cs.york.ac.uk/ftpdir/reports/2010/YCS/453/YCS-2010-453.pdf

EME : 11

Domain

• It is not the role of the developer to make decisions about science

• A simulation is built for a purpose in a domain– Domain information comes from domain experts

• responsible for scientific scope of the simulation

• provide disambiguation, answer questions, etc.

– Information in the form of papers, expert insight, lab experiments and results, text books…

• Domain experts and developers negotiate scope and detail level for simulation, and purpose of simulation exercise

EME : 12Some information on domain of lymphocyte capture

Andrews et al: www.cosmos-research.org/docs/cosmos2008-understanding.pdf

EME : 13

Domain Model

• Built by developer, checked by domain expert

• Represents developer’s interpretation of the domain

• Strict scoping– The domain expert may want more detail, wider scope, etc.

– The developer has to focus on the established purpose

• Different purposes require different domain modelsSee also: Read et al: www.cosmos-research.org/docs/cosmos2009-read.pdf and www.cosmos-research.org/docs/icaris2009-eae.pdf

⁻ Focuses on the scientific understanding• no implementation details

EME : 14

Domain issues

• Domain experts do not know everything

• Authoritative scientific papers may be:– Economical with the truth

– Following accepted conventions that have no firm basis

– Inconsistent with other scientists’ observations

• Data accuracy may be low– Did all the data come from the same individual?

• Or even the same (sub)species?

– How accurately can a data point be measured?• Often only to within several orders of magnitude

– Is the datum “real” or an estimate?

• Some things cannot be measured or understood with current scientific techniques

EME : 15

Research context

• All the assumptions and decisions

• Record sources and assumptions

• Achieve deep understanding of the purpose of simulation

e.g. Some assumptions from lymphocyte capture study:– The detail provided by domain experts is correct

– Lymphocyte migration only takes place in the HEV areas of the lymph node

– All lymphocytes are “the same”, and there is no interaction between lymphocytes: they do not collide

– Once a suitable chemokine signal has been received by a lymphocyte, it always migrates

– Lymphocytes are created, die, and flow through the HEV at a constant rateAndrews et al: www.cosmos-research.org/docs/cosmos2008-understanding.pdf

⁻ Separating expert and developer roles encourages developers to challenge domain understanding

EME : 16Platform Model, Simulation Platform

• The developer’s domain

• Design and implementation– Must be traceably linked to domain model and research

context

• Must record design decisions, simplifications, computational necessities– In order to understand any results from simulations

• Simulation Platform encodes the platform model in software and hardware– Simulations are run on the simulation platform (simulator)

– Simulations equate to scientific experiments

– Simulator may be modified for different simulations

EME : 17

Results Model

• A simulator is not the same as the domain– Results of simulation must be mapped back to the domain

• In a simple case, each domain concept maps directly to a computational concept

• In the usual case, computational artefacts do not map cleanly– Results model records mappings

– With research context, presents understanding of mappings, assumptions, limitations

• Put results in context

EME : 18

CoSMoS Purposes and responsibilities

• A simulation is built for a purpose– The purpose relates to the domain

– Domain information comes from domain experts• responsible for scientific scope of the simulation

• provide disambiguation, answer questions, etc.

• Engineering systematically develops – from a model of the domain

– to a platform model

– to a platform, or simulator• used to run a series of experiments or simulations

EME : 19

Assurance and documentation

• Domain modelling includes recording of sources and assumptions

• Platform modelling includes recording design decisions

• Assurance uses evidence to support a case that the simulator is adequate to support a specific set of simulation experiments– Justification of assumptions, simplifications, design

decisions

– Mappings from domain concepts to platform components

• Assurance also calls on evidence of engineering quality– Testing and other verification activities

EME : 20

Engineering computer simulations

• Two main sources of inspiration for an engineering approach to computer simulation– Simulation approaches devised for conventional systems

– Critical systems engineering

• Key features– Validation and arguments of validity

– Modelling approaches that support verification

• Consider what these mean in relation to complex system simulations that could be used to support scientific research

EME : 21

Conventional engineering of simulations

• Work on simulation by R. G. Sargent over 30 years

– Conventional systems

• A lifecycle model:

– Elements of a development

• Verification and validation

citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.17.438

EME : 22

Validation of computerised models

Some of Sargent’s proposals:– Comparison with valid models, with real events

– Domain expert appraisal• Turing test on the expert using the simulation

– Historic or predictive data validity

– Combination of rationalisation of assumptions, empirical validity checks and sound theory

• Most assume that reality is understood or already expressed in validated models– Not realistic for most complex systems simulations

• We can construct assurance arguments

See: Polack, www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf Ghetiu et al: www.cosmos-research.org/docs/valid2010-arguments.pdf

EME : 23

Verification: software

• Complex systems have complex interactions– A lot of processes going on in parallel, with periodic

synchronisations

• Simulations need to consider how to check the protocols and structures used– e.g. systematic consideration of assumptions made about

time and space• Including assumptions made to turn parallel reality into

sequential and/or OO code

– e.g. formal proof of protocols• occam-pi synchronisations can be proved deadlock-free

• Systematic recording and challenging of design assumptions

EME : 24

Validation: Assurance arguments

• We can adapt techniques used in safety case arguments– We can use Goal Structuring Notation

• or something very similar

• Present the structure of an argument– Expose the argument to external review

– Identify evidence on which the argument is built

• Attaining this level of understanding helps establish trust between scientists and developers– Potentially helps convince other scientists

See: Polack, www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

EME : 25

Recap: Basic Goal Structuring Notation

See T.P.Kelly, PhD thesis, www.cs.york.ac.uk/ftpdir/reports/99/YCST/05/YCST-99-05.pdfR. Weaver, PhD these, www.cs.york.ac.uk/ftpdir/reports/2004/YCST/01/YCST-2004-01.pdfand papers by Kelly’s group, www-users.cs.york.ac.uk/~tpk/pubs.html

EME : 26

GSN: arguing simulation adequacy

Polack, CoSMoS 2010: www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

EME : 27

GSN: justification

Polack, CoSMoS 2010: www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

EME : 28

Elaborating parts of the argument

www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

EME : 29

An alternative argument style

Ghetiu et al: Valid 2010: www.cosmos-research.org/docs/valid2010-arguments.pdf

EME : 30

Something different: arguing trust!

Ghetiu

et a

l: w

ww

.cosm

os-re

search

.org

/docs/v

alid

20

10

-arg

um

ents-slid

es.p

df

EME : 31

Validation: common problems

• Design steps typically use diagrams and related text or mathematics– To represent state, operations, structures of behaviour

• We’d like to use these in engineering simulations

• Existing diagram approaches do not express all that is needed for complex systems– Time

– Space

– Relative numbers of components

• Specialist formalisms exist but need experts– eg. process algebras

See www-users.cs.york.ac.uk/~susan/bib/ss/nonstd/alife08c.pdf

EME : 32

Notes for 2013

• The CoSMoS website has been hacked: most material can be found under – www-users.cs.york.ac.uk/~susan/bib/ss/nonstd/

– www-users.cs.york.ac.uk/~fiona/PUBS/recent.html

• We have identified that fitness for purpose is a better term than validation:– Validation in CS tends to imply a Boolean

• It is valid, it is not valid

– In complex systems, validity is contingent• Valid in this context for this purpose