validation of simulation models
TRANSCRIPT
This article was downloaded by: [Lund University Libraries]On: 29 October 2014, At: 03:43Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Journal of Experimental & TheoreticalArtificial IntelligencePublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/teta20
Validation of simulation modelsMuniza Rehman a & Stig Andur Pedersen ba Section for Philosophy and Science Studies , Roskilde University ,Universitetsvej 1, 3.1.3, 4000 Roskilde , Denmarkb Section for Philosophy and Science Studies , Roskilde University ,Universitetsvej 1, 3.1.3, 4000 Roskilde , DenmarkPublished online: 04 Sep 2012.
To cite this article: Muniza Rehman & Stig Andur Pedersen (2012) Validation of simulationmodels, Journal of Experimental & Theoretical Artificial Intelligence, 24:3, 351-363, DOI:10.1080/0952813X.2012.695459
To link to this article: http://dx.doi.org/10.1080/0952813X.2012.695459
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Journal of Experimental & Theoretical Artificial IntelligenceVol. 24, No. 3, September 2012, 351–363
Validation of simulation models
Muniza Rehmana* and Stig Andur Pedersenb
aSection for Philosophy and Science Studies, Roskilde University, Universitetsvej 1, 3.1.3,4000 Roskilde, Denmark; bSection for Philosophy and Science Studies, Roskilde University,
Universitetsvej 1, 3.1.3, 4000 Roskilde, Denmark
(Received 26 March 2012; final version received 29 April 2012)
In philosophy of science, the interest for computational models and simulationshas increased heavily during the past decades. Different positions regarding thevalidity of models have emerged but the views have not succeeded in capturing thediversity of validation methods. The wide variety of models with regards to theirpurpose, character, field of application and time dimension inherently calls for asimilar diversity in validation approaches. A classification of models in terms ofthe mentioned elements is presented and used to shed light on possible types ofvalidation leading to a categorisation of validation methods. Through differentexamples it is shown that the methods of validation depend on a number of thingsas the context of the model, the mathematical nature, the data available and therepresentational power of the model. In philosophy of science many discussionsof validation of models has been somewhat narrow-minded reducing the notionof validation to establishment of truth. This article puts forward the diversity inapplications of simulation models that demands a corresponding diversity in thenotion of validation.
Keywords: validation; simulations; categorisation; diversity
1. Introduction
Technological development has increased the computational power available and as aresult many mathematical problems can be either solved or approximated to a large extent.The number of mathematical models available for implementation on computers haveexploded, enabling cross-disciplinary knowledge sharing. A lot of effort has been put intoapplying these computational tools to new disciplines hitherto unknown to the possibilitiesof computational models. The models have a representational purpose of describingphenomena as close to the observed reality as possible and these models can be used tomimic a phenomena or predict what will happen if some initial conditions for the modelare modified. The technological development has made it possible to implement complexmathematical models and run realistic simulations within chemistry, economics,climatology, etc.
In the past 20 years, philosophy of science has gained interest in these models and triedto shed light on the epistemology of models and simulations. When talking of theepistemology an important aspect is the validation of models or the legitimacy of the finalmodel scientists arrive at. This is especially important in complex decision-making
*Corresponding author. Email: [email protected]
ISSN 0952–813X print/ISSN 1362–3079 online
� 2012 Taylor & Francis
http://dx.doi.org/10.1080/0952813X.2012.695459
http://www.tandfonline.com
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
situations where simulations provide incentives for the decision either due to lack of accessto the system or as a sort of prior assessment to experimentation. An informed decisionrequires a certain quality of the model, and so validation, one could say, is a kind ofquality assurance of the model. Within each discipline, different schemes for evaluating themodels before releasing them or trusting their results are developed. Philosophers togetherwith scientists have discussed the value of models based on validation, and coming fromdifferent worlds they have developed significantly different perspectives on validation.Some are pragmatic while others find it to be incommensurable with the establishment oftruth. In this article, some of the most important factors influencing the validation will beexamined, and it will be argued that depending on the nature of models, differentvalidation strategies will be used.
2. Validation
In order to understand the term validation a definition should be in place. Dee (1995) havedefined it as the following:
Validation of a computational model is the process of formulating and substantiating explicitclaims about the applicability and accuracy of computational results, with reference to theintended purposes of the model as well as to the natural system it represents.
This is in our view a good definition as it takes as a point of departure the intendedpurpose of the model and makes the validation particular to the application situation.Validation methods cannot, in general, be universal because different models are based onvery different kinds of data, or on specific construction methods, or they are developed forparticular systems which constrain their application to the specific context. The limitationof the definition is that it does not say when a validation is good enough. However, this isinevitable as the validation depends very much on the experience of the modeller, theintended purpose of the model, the data used for construction, the theoretical basis,etc. Furthermore, a specification of ‘applicability’ is necessary to sharpen thedefinition. In general, we define it as being implementable in a specific context with acertain purpose.
Oreskes, Shrader-Frechette, and Belitz (1994) argue that validation is the establishmentof legitimacy. According to them it is mistakenly interchanged with verification1 which isonly possible in closed systems, but most systems which we want to simulate are opensystems based on incomplete knowledge, and often several different models can producethe same results. They argue that if the model fails to reproduce observed data it is deemedfaulty, but that a model could, in principle, be faulty and represent the data well.Furthermore, a variety of different models based on very diverse hypotheses couldrepresent the data well and therefore give valid predictions without saying anything trueabout the mechanisms behind the data.
The view of Oreskes et al. (1994) need to be expanded. First of all it is true that thevalidation sometimes is confused with verification. But for experienced modellers involvedin model construction and reporting of results it is generally not true. The autonomy of themodeller manifests itself in the final model as decisions are made during the developmentprocess and he/she is well aware of the shortcomings of the model in regards to theirrepresentational power and performance capacities.
The misunderstanding of validation often comes when model results are conveyed toparties outside the scientific practices as policy makers. They can have difficulty in
352 M. Rehman and S.A. Pedersen
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
understanding the purpose and grasping the interplay between model shortcomings andmodel reliability. This can compel the modeler to compromise and develop models inscientifically unrealistic ways, and the relationship between the scientific ‘outsiders’ andthe scientists or researchers becomes politically entangled.
Secondly, very large models as climate models or wind turbine models consist ofsmaller models lumped together and the whole model system is partly validated byverification of the submodels. This use of the word verification is adopted by manymodellers. According to Oreskes et al. (1994), this way of using the word verification is notcorrect and the word can only be applied for closed systems as realised in laboratorysystems. As models can represent both closed or open systems, a broader understanding ofthe concept of verification than ‘establishing truth’ is needed.
If the word verify is looked up in an Oxford dictionary (Fowler and Fowler 1992), themeaning is stated as Establish truth, correctness or validity of by examination and this morebroader definition allows open systems to be verified. Recall that an open system is asystem which continuously interacts with its environment and can be represented in anumber of ways. In such systems it is usually not possible to control all parameters and,consequently, they may show unpredictable and inexplicable features. Despite suchuncertainties these systems can be verified not in an absolute sense but in a more varied orgraded sense. The evidence of the correctness, legitimacy and soundness of the modelrepresenting the system can still be provided hence it can be verified. The issue is not thatopen systems cannot be verified, rather it is to what extent such systems can be verified.This depends on many different factors which will be elaborated in the remainder of thearticle by categorising the validation factors and giving examples.
3. Validation factors
The most important factors influencing the validation process will be elaborated in thissection and examples will be given and the implications for the validation will bepresented.
3.1. Intended purpose
Models can have different purposes and a categorisation of the possibilities of purposes ispresented here.
3.1.1. Prediction
They can be used to predict, meaning that they state something about an observable thathas not yet been observed. There is a futuristic element to this. Such models we callpredictive models.
In physics, a model describing the motion of a simple mathematical pendulum can bederived by applying Newton’s Second Law. That leads to a non-linear second-orderdifferential equation
€� þg
lsin � ¼ 0 ð1Þ
Journal of Experimental & Theoretical Artificial Intelligence 353
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
where g is the acceleration of gravity and l the length of the pendulum. When the
displacement � is small sin� can be replaced by � and we have the equation of the simple
harmonic oscillator
€� ¼ �k� ð2Þ
where k is gl . It captures central features of the behaviour of the pendulum and is important
in the predictive analysis of it. The simple harmonic oscillator is very useful in other areas
of physics and chemistry. For instance, atoms in vibrating molecules undergo harmonic
oscillatory motion and the classical mechanical energy expression can be incorporated in
the Schrodinger 2 equation describing the dynamics of the molecule (microscopic system).
The solution to the Schrodinger equation is given by a wave function � which contains all
the dynamic information about the system it describes (Sakurai 1994, Atkins 1998,
Friedman and Atkins 2001). Predictions of the system can be made after solving the
equation. In many cases first-order approximations – like considering the simple
mathematical pendulum as a harmonic oscillator – are sufficient. Generally in physics, a
lot of predictive models can be derived from basic equations when applied to a physical
body for instance the path of an automobile on an inclined plane can be predicted or the
motion of a rocket and so forth.In the example of pharmacokinetic/pharmacodynamic (PK/PD) (Rowland and Tozer
2009) models which are used to model drug passage through a human body, the models
when developed are used to predict the drug passage for different dosing regimens. For
instance, if the model is based on data from a study where patients have been given 200mg
of a drug every 6 h the final model could be used to simulate what the drug profile would
look like if 400mg were to be given every 12 h.Other types of predictive models are global climate models (GCM) used to predict
what will happen if the level of CO2 or other greenhouse gases is increased by a certain
factor, changes in cosmic radiation occur or other changes in factors affecting the climate
(Henderson-Sellers and McGuffie 2005).
3.1.2. Explanation and mechanism
Exploratory or mechanistic models are models that explain a phenomena or a mechanism in
a system. Let us take a look into the world of physics and chemistry to exemplify these
models. The Bohr model explains matter which consists of atoms, which are built up of
protons, neutrons and electrons. The atom consists of a core or nucleus of protons and
neutrons held together by strong nuclear forces and the electrons move around the nucleus
in circular orbits in the order of 10�10m (Finn and Alonso 1992). Bohr also devised that
the energy of an electron in an atom is quantised3 (Figure 1). The states corresponding to
allowed energies are called stationary states and the state with the lowest possible energy is
the ground state. The electrons can gain and lose energy by jumping from one allowed
orbit to another absorbing or emitting electromagnetic radiation with a frequency
determined by energy difference of the levels. He developed the model and the theory to
give the basis for the nuclear model of the atom and to explain the line spectra4 mostly in
the visible region for hydrogen-like atoms. In this region, the frequency v of the
electromagnetic radiation emitted or absorbed by the atom in a transition between
354 M. Rehman and S.A. Pedersen
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
stationary states corresponding to n1 and n2 is given as the following when the Bohr’stheory is applied 5 (Finn and Alonso 1992):
v ¼ 3:2898� 1015Z2 1
n21�
1
n22
� �Hz ð3Þ
Z is the atomic number. Although this model is too simple and has been modified it gives arather good first explanation of line spectrum.
Then there are models that can explain a mechanism for instance when modelling thekinetics of a drug in PK/PD modelling some models are developed based on hypothesis ofdrug mechanism and the data used for constructing such models is more specific.The model represents a proposed mechanism and in case of good agreement they are usedfor understanding how the drug is working in the body.
Taking the case of climate modelling, the large global models consist of smaller modelslumped together and many of the more specific submodels are explanatory in the sensethat they might include the mechanisms of radiative transfer in the atmosphericlayers, ozone depletion, heat transfer or other phenomena important to the climate(Henderson-Sellers and McGuffie 2005).
3.1.3. Design
As the name suggests Design models are used for the purpose of designing an artefact or aprocess. In the field of aerodynamics, models are often used for designing purposeswhether it be a wing for an airplane or for a wind turbine. In the latter case, mathematicalmodels are used to simulate the effect of accumulated stress over time on the rotor-blades,and the rest of the windmill. This information is used to optimise the material and designof the rotor-blades or other design aspects of the wind turbine. The role of models andsimulation can be very similar when developing airplanes (Moller 2003).
3.1.4. Data representation
This is a rather different kind of purpose of models where they are used to manipulate andreformulate raw data such that the data can be comparable and adequate for developmentand analysis of other models and theories. In many situations it is not possible to use theraw data and interpret it and so a reduction or transformation is needed. Let us take the
Figure 1. The Bohr model.
Journal of Experimental & Theoretical Artificial Intelligence 355
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
case of simple data reduction first. In this case, the amount of data is reduced by taking ameans or smoothing the raw data. This is for instance often necessary when working withclimate models where the amount of observations is enormous. Data corresponding tospecific regions is divided and mean over a particular region can be calculated to minimisethe amount. This is often convenient as calculations run relatively faster but also becausethe computational power needed to handle the full amount of data is simply not available.Another kind of simple data reduction could be the exclusion of obvious outliers whichcould result from inaccuracy of measurements. Omitting data points surely reduces theamount of data in a simplistic way. Methods for simple data reduction we call reductionmethods.
In many cases it is not possible to make adequate data reductions without involvingadvanced scientific hypotheses. We call this complex data reduction. This is, for instance,the case in geodesy when the gravitational field is being measured in a mountain area. Rawgravity measurements will, in such cases, fluctuate drastically and will be of no use unlessthey are reduced to a common level. That is done by isostatic reduction where mountainsare treated as icebergs floating on a plastic surface (Heiskanen and Moritz 1967). Byreducing the data to what they would have been on the surface without the mountains it ispossible to construct a consistent and adequate system of data. The isostatic models play acentral role here.
Another example of complex data reduction is optic observations in astronomy(Kjeldsen 2009). When observing stars through telescopes it is necessary to take intoaccount that light is being influenced by many different physical sources before it reachesthe telescope, so we need to model the impact of these different sources: we need models ofaberration, the Doppler effect, etc. This is done by using adaptive optics where a computeris used to detect the amount of light that is diffracted and a correction is made just beforethe light reaches the telescope detector.
Methods used for complex data reduction purpose we will call data transformationmethods. They depend on complex scientific hypotheses and involve further parametersthat require other forms of data in order to be estimated correctly.
3.2. Mathematical character
In spite of the extensive development of modelling tools it is often difficult to come up withtractable mathematical models. Even in cases where reasonable mathematical models areavailable it is impossible to find analytical solutions. For some problems it is evenuncertain whether analytical solutions do at all exist. An example is the Navier–Stokes’sequations where it is unknown whether solutions do exist in the general case. It is,therefore, important to study the mathematical character of models. Sangwin (2011) haveproposed a very useful categorisation:
(1) Exact models with exact solutions(2) Exact models with approximate solutions(3) Approximate models with exact solutions6
(4) Approximate models with approximate solutions
Models are used to represent some aspects of a phenomenon or system which we want tocontrol, illustrate or understand. Therefore, all models involve simplifications to differentdegrees and the notion of an exact model is an oxymoron according to Sangwin (2011)
356 M. Rehman and S.A. Pedersen
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
We agree that this is the case, however, Sangwin (2011) did not manage to clarify what ismeant by exact when all models are selective in terms of what they represent. The abovecategorisation seems insufficient until it is not specified what is meant by exact andapproximate models. We will attempt to define what is to be understood of being exact orapproximate in order to make the categorisation more clear. Models that represent somefeatures of a phenomenon or system which are directly derived from fundamental theoriesas the theory of quantum mechanics, the theory of classical mechanics, etc. without needfor further adjustments to capture the desired features, is what we mean by exact models.Approximate models are to be understood as models that possibly are developed by fittingto data or derived from theory but in addition need to be adjusted to capture the system orphenomena of interest. This also means that a model in a certain context can serve as anexact model, and in another context it will not suffice and further development is needed.An example could be in quantum mechanics where relativistic effects sometimes areneglected and other times needed depending on the system being modelled.
When trying to arrive at a model for phenomena or systems, in the first category thereare not many examples that come to mind. However, a graphical model of the metronetwork of a big city is an exact representation of the distribution and connection betweenmetro stations and the position of the trains on the lines. But, it does not represent theactual geometry of the system. Such graphical modes may be complex but they give exactrepresentations of trains and the places of individual trains. They are important tools inthe control of the metro system.
In the second type, the accuracy of an approximation can be established but this isrelative to the ‘best’7 approximation. The simple mathematical pendulum considered as aharmonic oscillator is a simple example of this. An example from computational chemistrywill be introduced to elaborate this. In quantum chemistry, a wide range of approximativemethods to solving the Schrodinger equation exist, as it is only possible to solve it exactlyin very few one-electron cases. One method is the Hartree–Fock (HF) method for solvingthe electronic8 Schrodinger equation. It is an ab initio9 approach where a model for theelectronic wave function is chosen and the H�¼E� is solved using as an input, only thevalues of fundamental constants and atomic numbers of the nuclei. The accuracy isdetermined by the model chosen for the wave function. Ab initio methods arecomputationally expensive for large molecules where other approaches are applied. Thewave function for an N-electron system in the HF approach is described by a slaterdeterminant given as
� ¼1ffiffiffiffiffiffiN!p
�1ðx1Þ �2ðx1Þ � � � �Nðx1Þ�1ðx2Þ �2ðx2Þ � � � �Nðx2Þ
..
. ... . .
. ...
�1ðxNÞ �2ðxNÞ � � � �NðxNÞ
0BBB@
1CCCA ð4Þ
Here, the �(x) denotes a spin orbital. H–F equations are solved for the individualspinorbitals and are given as
f ðx1Þ�iðx1Þ ¼ �i�iðx1Þ ð5Þ
where f is the Fock operator defined by the Coulomb operator and the Exchange operator.
Jjðx1Þ ¼
Zdx2j�jðx2Þj
2r�112 ð6Þ
Journal of Experimental & Theoretical Artificial Intelligence 357
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
Kjðx1Þ�iðx1Þ ¼
� Zdx2�
�j ðx2Þr
�112 �iðx2Þ
��jðx1Þ ð7Þ
The former gives the average local potential at point x1 due to charge distribution from the
electron in orbital �j whereas the latter is harder to explain. Because the Coulomb and
Exchange operators are defined in terms of spinorbitals the Fock operator depends on the
spinorbitals from all the other electrons in the system. So, to set up HF equations it is
necessary to know the solutions before hand. This is a common dilemma in electronicstructure calculations and is coped with by adopting an iterative solution and stopping
when the solution is self-consistent. Therefore, it is also known as the self-consistent field
(SCF) theory. In practice, a trial set of spinorbitals is used to formulate the Fock operator
and then the HF equations are solved to obtain a new set of spinorbitals which are used to
construct a revised Fock operator and so on. This is repeated until the defined convergence
criteria is satisfied. Electron correlation is not accounted for in the HF approach and thefollowing method incorporates this effect.
A more sophisticated approximate method is perturbation theory where the
Hamiltonian is divided into several parts (Jensen 1999).
H ¼ H0 þ �H0 þ �2H00 þ � � � ð8Þ
The significance of the parameter � is that it keeps track of the order of the perturbationand enables the identification of all first-order, second-order terms and so on. The
Hamiltonian consists of two main parts a reference (H0) and one or more perturbations
depending on the order. The perturbation can be time-dependent but for simplicity this is
considered to be the time-independent case. It is assumed that the Scrodinger equation for
the reference Hamilton operator is solved so the solutions are known. Methods can differin the expression for the Hamilton operator or the expression for the wave function. As the
perturbation increases from zero to a finite value, the energy and wave function must also
change continuously and both energy,W, and wave function, �, can be written as a Taylor
series expansion in powers of the perturbation parameter �.
W ¼ �0W0 þ �1W1 þ �
2W2 þ � � � ð9Þ
� ¼ �0�0 þ �1�1 þ �
2�2 þ : � � � ð10Þ
One very recognised method is called coupled cluster (CC) which is used in many electron
systems where the electron correlation can be included to differing extents by using a
specific expression for the wave function and depending on how much correlation is
included more or less mathematical terms will be added in the perturbed wave functionexpression.
�CC ¼ eT�0 ð11Þ
Here, �0 is the zero-order wave function which is the solution to the unperturbed Hamilton
operator and T is the cluster operator which will not be elaborated further here. Even
within the CC methods different degrees of accuracy can be achieved depending on the
number of perturbing terms included in the specific CC wave function. Many of the other
methods lacking the correlation can be compared to the results of CC to assess theaccuracy and in some textbooks a kind of hierarchy of the accuracy of the computational
methods is presented (Jensen 1999). The tinkering and optimising of a low-hierarchical
358 M. Rehman and S.A. Pedersen
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
method is possible as the results can be bench marked relative to a more sophisticated
method higher in the hierarchy. The closer the results of the low level methods are to the
results of the high-level method the more improvement has been introduced to the method,
being tested.Different problems require different methods. For molecular geometries HF–SCF can
suffice but for molecular properties more sophisticated methods are required for adequate
results. It is a reference validation that is possible but the reference has to be chosen. In the
example given, a number of CC methods can be chosen as a reference depending on the
computational power available and the complexity of the system of interest.In the third category, the problem is of a representational character. Models only
represent a situation approximately and one cannot be sure that the representation is
adequate and reflects the relevant features of the situation. The validation is simply at the
model level or the mathematical level and a more holistic validation of the modelled
system is not possible. One such example has already been mentioned the PK/PD models
used in the pharmaceutical industry to model how the drug passes through the human
body and what it does to the body. The human body is divided into compartments very
often two or three compartments, one central compartment (often the blood stream) and
one or more peripheral compartments (deeper tissues). In the two-compartment case
where a dose is given intravenously, the model used to represent the body is the following.
There is no doubt that this is a crude division and an approximate model of the human
body. However, the differential equations describing the change in amount of drug in each
compartment have exact solutions (Figure 2). By solving these equations the mathematical
expression for the concentration as a function of time in the body can be derived.
CðtÞ ¼ C1e��1t þ C2e
��2t ð12Þ
C is the concentration, and C1, C2 are coefficients and �1, �2 are exponents which are
determined by fitting the empirical data. This is an exact solution to an approximate
model, that does not include the essential features in the human body.In the fourth possibility there is both the representational issue and the possibility of
relative validation making it a very delicate situation to deal with. Many complex models
Figure 2. Two-compartment representation.
Journal of Experimental & Theoretical Artificial Intelligence 359
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
as climate models and astrophysical models are of this character. A historical example will
illustrate this. The first computer-based atmospheric models appeared in the 1950s based
on Jules Charneys idea of theory pull. It was known at that time that earlier models were
numerically instable.10 Charney and colleagues therefore adopted a progressive approach.
One started with a coarse model which could be computed, and if that gave acceptable
results the model was expanded by adding new physical factors that would remedy
discrepancies. One of the first approximate models was a two-dimensional barotropic
model.11 In 1950, the first successful results based on this model were published (Charney,
Fjortoft, and von Neumann 1950). It was an approximate model, as it only covered the
barotropic vorticity equation12, and it only had approximate solutions. Later atmospheric
models are, of course, significantly more complex both mathematically and computation-
ally but still approximate models with approximate solutions.13
3.3. Time
The time dimension of what is being modelled is of crucial importance to how the model is
validated. For many models the validation can partly consist of a comparison of
predictions with observations, but there are applications where the observation simply is
not possible. Take for instance GCM. The time dimension of the modelled climate change
is typically in the order of 50–100 years and the results cannot be compared to
observations within the lifetime14 of the model. To illustrate the notion of the lifetime of a
model a few examples with differing lifetimes will be elucidated. The validation of climate
models is mostly retrospective in the sense that past climate events, west and east winds or
other meteorological phenomena are expected to be modelled adequately. Due to the
amount of tuning15 in climate models together with a time dimension greater than the
observable span the reliability of the models can be questioned. There are too many
variables that can change before we reach the time for the predictions that rely on the
ability to reproduce past events. The time to observation means that it is only possible to
validate retrospectively. The different submodels of a GCM are validated against current
observations and this is a kind of subvalidation of the large model, but not a validation of
the whole model in itself.In other predictive applications, the time for observation is relatively short meaning
that observation can be made. This entails that a validation in terms of a comparison
between predictions and observations can be made. A model is constructed based on data
whereafter an experiment can be run and the data from that can be compared to the
proposed model. This is the case with the PK/PD models where phase 1 trial outcome is
the basis of model construction and phase 2 data is used to confirm what has been learned
in phase 1 and suggest changes to the original model. This is in the pharmaceutical
industry known to be the learn-confirm-learn cycle. The lifetime for such models is much
longer than for GCM as the system is less open in a sense. There are not as many variables
that change rapidly and influence the results predicted. After approval of a drug the
models are more or less established for their purposes whereas climate models still need to
predict the new situation arising from occurred disruptions in the system originally
modelled. It could for instance be a larger frequency and magnitude of volcanic
activity resulting in a changed composition in the atmosphere of volcanic gases
influencing the climate. It is also known that the fundamental equations in meteorological
360 M. Rehman and S.A. Pedersen
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
models are heavily non-linear and as such are very sensitive to small changes in initial
values.An example of models with very short lifetime is agent-based models Humphreys 2004
used for simulating behaviour, interactions or actions of autonomous agents. This type of
modelling is used to simulate flocking behaviour of birds, social phenomena, etc.
The models are descendants from cellular automata models and the state of the system iscalculated at each time step by simple rules. The models are especially good for simulating
highly complex systems that change rapidly. There are no overarching models of the
system being modelled only rules capturing the actual behaviour. This means that for each
behavioural change a new agent-based modelling approach is needed and since the
modelled systems change rapidly the lifetime of one simulation is very short.In the case of design models, a model is constructed16 to produce data for a design of,
say, the dimensions of a wind turbine. It is then used for simulations of how much force it
can bear. These models are validated by comparison to measurements on real windturbines, and they are adjusted or tuned to fit these measurements. All in all it is also a
validation consisting of a comparison to observations, which can be denoted a
confirmative validation. Recall from Oreskes et al. (1994) that a model at best can be
confirmed by data or observations.
3.4. Validation categories
So far the different validation factors have put forward some types of validation
approaches. To summarise, the following has emerged:
. Confirmative validation
. Subvalidation
. Reference validation
The most common type of validation, we would argue, is the confirmative validation,
where the model is confirmed by observations from empirical knowledge and so the time
dimension has to be within reach to make such validations.Subvalidation is relevant for large models lumped together by smaller models. The
purpose of such a validation is to state the legitimacy of the large model construction
through a confirmative validation of the embedded smaller models. But subvalidation is
not sufficient in itself. Further, practical and theoretical evaluations of large models are
necessary before they can be accepted.Reference validation can be defined as the measure of accuracy of a model
benchmarked to the Potissimus model possibility. This kind of validation is of a
theoretical nature and in such cases simulations can be regarded of as being theoretical
experiments, as the observations are out of reach or not of primary interest.All of the above three approaches exist and as already stated models may be validated
by more than one approach. An important aspect when understanding models is that the
validation is an iterative process where the model is adjusted, tested again, adjusted and so
forth, to get to the final model. The legitimacy of the model could be obtained by model
validation but some models require an extensive amount of data reduction17 and the datareduction has an impact on the representational power of the models as well. In the
interpretation it is crucial to take all such actions into account.
Journal of Experimental & Theoretical Artificial Intelligence 361
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
3.4.1. External and internal validation
Within each of the listed validation categories, a further distinction can be made toexternal and internal validation. External validation is the assessment of how well themodel represents the system of interest. So, it is a matter of studying both predictedoutcome and observed outcome and comparing this with a theoretical understanding ofthe system. The nature of this kind of assessment is macroscopic and requires that it ispossible to measure properties of the system that are predicted.
There is also the internal validation which is linked to the mathematical abilities of themodel. A minimum level of stability is required meaning that the model gives consistentresults and does not fluctuate too much when a small perturbation in starting conditions ismade. Moreover, the ability to converge is also included, as it sometimes happens thatmodels cannot converge in the computational time allocated for the calculation.Consistency of converged solutions is also a necessity. This is a more microscopicassessment of model elements.
Both of these kinds of validations are necessary in the above categorisation.
4. Conclusion
In this article, a classification of models in terms of the intended purpose is provided.Moreover, we argue that the purpose of a model imposes certain requirements to thevalidation. For instance, if the PK/PD models are taken as an example, they play a pivotalrole in drug approval and the risk associated with a spurious model is high due to thepossibility of clinical adverse events. On the other hand, if the simple pendulum model isconsidered as an isolated example the implications of a less correct model may not be assevere. The design model can in the worst case give bad results which lead to a poorconstruction of an artefact. This does not lead to casualties as an artefact is tested before itis applied and an inadequate product will, in most cases, be detected.
We have developed a general categorisation of validation types but to understand thespecific validation approach chosen for a given model it is necessary to contextualise themodel. The purpose, time dimension, available data and the mathematical representationaland computational power of a model determines the univocal case specific validationapproach. However, there are two validation elements that inherently are included in allapproaches and that is the external and internal validation. These two elements ensure therepresentational ability of models together with stability and consistency. Hopefully, wehave succeeded in demonstrating the existing diversity in model validation approaches andclarified some of the crucial factors influencing the validity of models. It is important tounderstand that the epistemology of models is diverse due to a similar diversity in theontological foundations of models and that these are deeply entangled.
Notes
1. Oreskes et al. (1994) defines this as the establishment of truth.2. The time-independent version is given as H�¼E� and the time-dependent version is
H� ¼ i�h ���t (Sakurai 1994, Atkins 1998, Friedman and Atkins 2001)3. This means that it can only have certain values.4. In a spectroscope, radiation of a given frequency appears as a line and this is called the line
spectrum (Kriz, Pavia, and Lampman 2001).5. This is also called the Balmer series.
362 M. Rehman and S.A. Pedersen
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4
6. In this case, it is difficult to know how well the models fit the system or phenomena of interest.7. Closest to the exact solution.8. This is the equation obtained after applying the Born-Oppenheimer approximation where the
motion of the nuclei is neglected because they weigh much more than electrons. The Schrodingerequation for the electrons in the static electric potential arising from the nuclei in a particulararrangement is solved. This gives the potential energy surface from which equilibrium geometryand vibrational frequencies can be obtained for a molecule.
9. Means from scratch and in this case it means that methods to calculate electronic structuresusing only the Schrodinger equation and not using any empirical information.
10. They did not satisfy the Courant condition.11. A model is barotropic when pressure depends on the location only, not on the local height.
Mixing of air and loss of potential energy is not taken into account.12. @�
@t þ �v � r� ¼ 0, where � is the absolute vorticity and v the velocity.13. The exact solutions of the Navier–Stokes equations cannot still be resolved so various kinds of
limit solutions are considered, see (Cullen 2007).14. Lifetime is here defined as the time interval where the scientific community involved considers
the model to capture and include cutting edge research. Thereafter it becomes more or lessobsolete and a new model is developed possibly my modifying the old model.
15. The inclusion and adjustment of parameters to fit the data.16. This is based on knowledge of the requirements.17. As described there exist many different forms of data reduction.
References
Atkins, P.W. (1998), Physical Chemistry (6th ed.), Oxford: Oxford University Press.
Charney, J.G., Fjortoft, R., and von Neumann, J. (1950), ‘Numerical Integration of the BarotropicVorticity Equation’, Tellus, 2, 237–254.
Cullen, Mike (2007), ‘Modelling Atmospheric Flows’, Acta Numerica, 16, 67–154.
Dee, D.P. (1995), A Pragmatic Approach to Model Validation (ch. 1), Washington, D.C.: AmericanGeophysical Union.
Finn, E.J., and Alonso, Marcelo (1992), Physics, Harlow, UK: Addison Wesley Longman.Fowler, F.G., and Fowler, H.W. (1992), The Pocket Oxford Dictionary of Current English (8th ed.),
New York: Oxford University Press.Friedman, R.S., and Atkins, P.W. (2001), Molecular Quantum Mechanics (3rd ed.), New York:
Oxford University Press.
Heiskanen, W.A., and Moritz, H. (1967), Physical Geodesy, San Francisco and London: W.H.Freeman and Company.
Henderson-Sellers, A., and McGuffie, K. (2005), A Climate Modelling Primer (3rd ed.), Chichester,
UK: John-Wiley & Sons.Humphreys, P. (2004), Extending Ourselves, New York: Oxford University Press.Jensen, F. (1999), Introduction to Computational Chemistry, New York: Wiley.
Kjeldsen, H. (2009), ‘Astronomernes Kaempeteleskoper’, KVANT, 2, 7–13.Kriz, G.S., Pavia, D.L., and Lampman, G.M. (2001), Introduction to Spectroscopy, Fort Worth, TX:
Hartcourt College Publishers.Moller, C.B. (2003), ‘Forskere i Modelkonkurrence’, Nyheder fra Vindmolleindustrien, 31, 12–13.
Oreskes, N., Shrader-frechette, K., and Belitz, K. (1994), ‘Verification, Validation and Confirmationof Numerical Models in the Earth Sciences’, Science, 263, 641–646.
Rowland, M., and Tozer, T.N. (2009), Clinical Pharmacokinetics and Pharmacody-namics (4th ed.),
Baltimore, MD: Lippencott Williams & Wilkins.Sakurai, J.J. (1994), Modern Quantum Mechanics (revised edition), New York: Addison-Wesley.Sangwin, C. (2011), ‘Modelling the Journey from Elementary Word Problems to Mathematical
Research’, Notices of the AMS, 58, 1436–1445.
Journal of Experimental & Theoretical Artificial Intelligence 363
Dow
nloa
ded
by [
Lun
d U
nive
rsity
Lib
rari
es]
at 0
3:43
29
Oct
ober
201
4