on the assessment of a bayesian validation methodology for ... · on the assessment of a bayesian...
Post on 10-Apr-2020
6 Views
Preview:
TRANSCRIPT
On the Assessment of a Bayesian Validation Methodology for Data Reduction Models Relevant to
Shock Tube Experiments
M. Panesi, K. Miki, S. Prudhomme, and A. Brandisby
ICES REPORT 11-36
November 2011
The Institute for Computational Engineering and SciencesThe University of Texas at AustinAustin, Texas 78712
Reference: M. Panesi, K. Miki, S. Prudhomme, and A. Brandis, "On the Assessment of a Bayesian Validation Methodology for Data Reduction Models Relevant to Shock Tube Experiments", ICES REPORT 11-36, The Institute forComputational Engineering and Sciences, The University of Texas at Austin, November 2011.
On the Assessment of a Bayesian Validation Methodology for DataReduction Models Relevant to Shock Tube Experiments
M. Panesi∗,a, K. Mikia, S. Prudhommea, and A. Brandisb
aCenter for Predictive Engineering and Computational Sciences (PECOS), Institute for Computational Engineeringand Sciences (ICES), The University of Texas at Austin
bCenter for Turbulence Research, Stanford University
Abstract
Experimental raw data provided by measuring instruments often need to be converted into mean-
ingful physical quantities through data reduction modeling processes in order to be useful for
comparison with outputs of computer simulations. These processes usually employ mathematical
models that have to be properly calibrated and rigorously validated so that their reliability can be
clearly assessed. A validation procedure based on a Bayesian approach is applied here to a data
reduction model used in shock tube experiments. In these experiments, the raw data, given in terms
of photon counts received by an ICCD camera, are post-processed into radiative intensities. Simple
mathematical models describing the nonlinear behavior associated with very short opening times
(gate widths) of the camera are developed, calibrated, and not invalidated, or invalidated, in this
study. The main objective here is to determine the feasibility of the methodology to precisely quan-
tify the uncertainties emanating from the raw data and from the choice of the reduction model. In
this analysis of the methodology, shortcomings, suggested improvements, and future research areas
are also highlighted. Experimental data collected at the Electric Arc Shock Tube (EAST) facility
at the NASA Ames Research Center (ARC) are employed to illustrate the validation procedure.
Key words: Uncertainty quantification, Bayesian analysis, Model calibration, Parameter
identification
1. Introduction
Traditional procedures for validation of models in computational sciences and engineering are
often related to data fitting in the sense that the closer the output quantities computed by mod-
els are to experimental observations, the better the models should be. The Center for Predictive
∗Corresponding author. E-mail address: mpanesi@ices.utexas.edu. Address: ICES, The University of Texas atAustin, 1 University Station, C0200, Austin, TX 78712, USA.
Preprint submitted to CMAME November 5, 2011
Engineering and Computation Sciences (PECOS) has recently proposed a novel methodology al-
lowing for the calibration of model parameters, the validation of models, and the quantification
of uncertainties in specific model predictions. This new approach is currently being evaluated for
simulating and predicting the multi-physics heating environment encountered during atmospheric
entry. However, it is clear that, for such processes, experimental data play a critical role, not only
for the calibration of the model parameters (e.g. reaction rate constants, spectroscopic constants,
etc.), but also for the validation of the model itself. Unfortunately, it is not uncommon that legacy
data, available for example in the literature, lack a complete description of associated uncertainties
and that the data reduction procedures, used to convert the raw data into meaningful physical
quantities, are usually reported with minimal information. It thus becomes virtually impossible
to account for the systematic errors due to experimental manipulations and, more importantly, to
estimate the influence and validity of the data reduction models.
Data collected from experiments are in general indirectly related to physical observations. Raw
data, usually in the form of electric signals (e.g. voltage) from electronic instruments or pixel counts
in digital images, are mapped into quantities such as stress components or radiative intensity by
using data reduction models. These models, as with physico-mathematical models that describe
physical phenomena, provide sometimes only crude approximations of the operating functions of
the instruments. They may also contain model parameters that need to be calibrated and whose
uncertainties have to be quantified. In certain situations, the reduced models need to be validated
as the instruments may be used in regimes outside of the calibration range and uncertainties should
then be correctly propagated to the reduced data. Figure 1 illustrates this concept and shows the
strong interaction between the validation of physical models and the validation of data reduction
models. The left box of the diagram describes the calibration and validation processes for the
data reduction models while the actual calibration and validation processes of the physical models
of interest are described in the right box of the diagram. The output of the first box consists of
experimental reduced data with associated uncertainty that are later utilized in the second box for
the calibration and validation of the physical model parameters.
The present work is devoted to the first stage of this two-step problem. Recently, experimental-
ists at NASA Ames Research Center (ARC) have embarked on an extensive experimental campaign
aiming at characterizing the radiative properties of high-temperature shock-heated gas in the con-
text of the CEV Aerosciences Project (CAP) [10, 8]. Data collected from the Electric Arc Shock
Tube (EAST), using imaging spectrographs, have been utilized for the calibration of kinetic and
2
Data ReductionModel
Reduced DataRaw Data
Data Reduction Modeling Computational Modeling
Validation
Calibration
Event
MathematicalModel
Instrument
Calibration
Validation
Figure 1: The validation of physical models requires the calibration and the validation of the data reduction models.
radiative model parameters as discussed in [13]. Results for the forward predictions of the radiative
intensity are summarized in Fig. 2. The plot shows excellent agreement between the predicted mean
intensity and the experimental observations, but also indicates that uncertainties, characterized by
the 2.5th and 97.5th percentile curves, are very large. At that stage of the investigation, it was un-
clear whether the sources of uncertainties originated from errors in the experimental procedures, in
the data reduction models, or in the physical models. The objectives in the present work are to re-
visit the data reduction models describing the functional elements of one intensified charge-coupled
device (ICCD) camera used in the shock tube experiments at NASA Ames Research Center and
to submit these models to the validation process proposed at PECOS. During this process, a tight
cooperation between modelers and experimentalists was developed in order to better understand
the functionality of the instruments and of the modeling capabilities.
Measuring tools or instruments should ideally be employed when operated in their linear regime.
However, in the case of the EAST experiments, such an assumption cannot be respected as the
acquisition time is extremely short due to minimizing the smearing of the data on the spectrometers
as the flow is traveling at very high speeds (about 10 km/s) in the shock tube. In consequence,
the ICCD camera exhibits a non-linear response for such small gate widths (time during which
the “shutter” of the camera is open) as the time required to open the camera slit is comparable,
or sometimes even greater than, the nominal time the gate is open during the experiment. One
possibility to characterize the non-linear response of the camera is to make a series of measurements
at small gate widths using calibrated sources of radiation. Unfortunately, the intensity of the
lamps used for calibration is considerably smaller than that of the radiative source during actual
3
0 1 2 3 4 5Z (cm)
0
0.05
0.1
0.15
0.2
Inte
grat
ed I
nten
sity
(W
/cm
3 -sr) Posterior 2.5% percentile
Posterior meanPosterior 97.5% percentile EAST exp. data (Shot# 47)
Figure 2: Comparison of numerically predicted integrated intensities and experimental observations as presented
in [13]. The mean intensity (Posterior mean) is in good agreement with the measured data (EAST exp. data (Shot
#47)). However, the 2.5th and 97.5th percentile curves indicate large uncertainties in the computed intensity.
experiments. Therefore, the correction factors accounting for the non-linear response at small gate
widths, although representative of the response of the camera at those calibration conditions, may
become inadequate when applied to the actual experimental data. Furthermore, at small gate
widths, the number of photon counts is drastically reduced when using calibration lamps, which
results in an increased scattering and noise that would be far too large for the actual experimental
data. This issue has been the motivation for constructing a mathematical model of the camera
shutter that can predict the response of the instrument at low gate widths. Simple models describing
the electronic circuit responsible of the gating of the photo-cathode in cameras have thus been
developed in this work following earlier works [26, 1, 19], and have been calibrated and validated
using the validation process suggested in [3, 17, 18] based on a Bayesian statistics framework [4, 6].
We emphasize here that our motivation has been to construct the simplest physical models that
would reasonably quantify the uncertainty in the reduced experimental data.
The paper is organized as follows. Section 2 describes the EAST experiments and experimental
set-up. Possible data reduction models that convert photon counts into radiative intensities are
described in Section 3. The Bayesian framework and the validation methodology used throughout
this investigation are detailed in Section 4. Results are presented and discussed in Section 5 while
concluding remarks are provided in Section 6.
4
2. Brief description of experimental apparatus
The Electric Arc Shock Tube (EAST) facility at NASA Ames Research Center is used to gen-
erate high-enthalpy gas tests for the simulation of high-speed atmospheric entries. The facility is
composed of a long tube and a chamber. In the chamber, also referred to as the driver, one sets
an electromagnetic discharge, giving rise to a sudden increase of the gas temperature and pressure.
This high?pressure gas then bursts the diaphragm that initially separates the driver gas from the
test gas, forming a shock wave that travels at high speed downstream of the tube toward the test
section where spectral measurements are performed. As the shock propagates downstream, the
shock-heated gas radiates energy. The radiative signature contains useful information concerning
the thermo-chemical and radiative properties of the medium, which can be inferred using emission
spectroscopy. For this purpose, the spectrally and spatially resolved shock-layer radiance is ana-
lyzed by taking a snapshot of the shock wave as it passes in front of the optical access window.
The experimental apparatus is composed of the collecting optics, spectrometers, and intensified
cameras. The light radiated by the high-temperature gas in the shock tube is directed toward the
spectrometers by the collecting optics as described in [8]. The incoming light is then decomposed
into different harmonics through the spectrographs. Finally, the spectrally and spatially resolved
radiation is recorded by means of the ICCD cameras.
2.1. Emission spectroscopy
Emission spectroscopy consists of the analysis of radiation emitted by a source through inspec-
tion of the radiative spectral signature. The spectral decomposition of the incident radiation is
detected by images taken by ICCD cameras that record pixel counts in proportion to the inten-
sity of radiation. As the measurement of absolute radiative intensities is the primary quantity of
interest, the cameras need to be calibrated. Calibration is generally performed by measuring the
radiation emitted from a well-characterized source. The choice of the calibration source depends
on the spectral interval to be observed, which in the present investigation ranges from 500 to 900
nm. Furthermore, at the nominal shock speed of 10 km/s, the shock travels 1 cm each microsec-
ond, which requires the usage of high-speed gating cameras to avoid substantial smearing effects
in the measurement. Image intensifier coupled with CCD cameras are thus used. Besides enhanc-
ing the sensitivity in low-light conditions, ICCD cameras can capture extremely small time-scale
phenomena due to the possibility of high-speed gating capabilities.
5
������������������������������������������������������������������������
������������������������������������������������������������������������
PhosphorScreen
Fiber OpticOutputWindow
Photocathode
MicrochannelPlate (MCP)
Photons
Photo−electron
Applied voltage
Applied voltage
Figure 3: Schematic of the intensifier in an ICCD cam-
era.
Wavelength [nm]
Dis
tance [cm
]
Shot B38
Shock Front
Atomic Line
Radiation
Contact
Discontinuity
Figure 4: Image obtained using an ICCD camera in a
shock tube experiment (Shot B38 from EAST).
2.2. Intensified charge-coupled device cameras
The intensifier is composed of 1) a photo-cathode, 2) a multi-channel plate (MCP), and 3) a
phosphor screen (see Fig. 3). Gating of the intensifier is made possible by controlling the voltage
across the photo-cathode. In other words, if the photo-cathode is biased more positively than the
MCP, electrons are repelled and the intensifier is gated off. The dynamic response of the gate
system can be represented by a resistor capacitor (RC) circuit, as discussed in greater detail in the
next section. It is important to observe that even when the photo-cathode is gated off, the current
across the intensifier is not zero. In fact, even in this case a fraction of photons not absorbed by
the photo-cathode may strike the MCP releasing photoelectrons. Furthermore, another source for
the build-up of charges on the CCD comes from photoelectrons released by the photo-cathode due
to thermal excitation. A reduction of the flux of photoelectrons can be achieved by simultaneously
gating the photo-cathode and the MCP.
Figure 4 shows an example of the spectrally resolved radiation output from an EAST exper-
iment. The colors indicate the photon counts recorded at each pixel of the image. The region
between the shock front and the contact discontinuity constitutes the area of interest for the mea-
surements of the radiative properties of the gas. The horizontal axis represents the wavelength
variation in nm while the vertical axis represents the distance in cm. The mapping from the raw
counts to the radiative intensity is obtained by defining the ratio between the counts collected
when using a source of known radiation (i.e. tungsten lamps). The objective in the remainder of
the paper is to develop a model of this ratio, to calibrate the model parameters, and to invalidate
6
(or not) the model for low gate widths.
3. Data reduction modeling
In this section, we derive simple data reduction models to be used to convert the photon
counts into radiative intensity. We focus in particular on modeling the correction factor, given as
a function of the gate width ∆t of the camera and which defines the deviation with respect to a
linear behavior, for small gate widths. We also list some of the hypotheses that we introduce to
construct the models.
3.1. Modeling and hypotheses
The number of photon counts N∆t(i, j) for a given ∆t are collected at each pixel (i, j) of the
image, 1 ≤ i, j ≤ 512, and can easily be mapped into photon counts N∆t(z, λ) at each position z
and wavelength λ, where z varies roughly between 0 and 12 cm and λ between 400 and 950 nm.
We note that ideally, we would like to have a linear relationship between the intensity I(z, λ)
and the photon counts per unit time, i.e.
I(z, λ) = K N∆t(z, λ)
∆t(1)
with K constant. Unfortunately, the ICCD camera does not have a perfectly linear response and
the coefficient K should be corrected as follows:
K = κ× γ(G)× ξe(λ)× ρ(∆t) (2)
where:
1. κ is a new constant coefficient.
2. γ(G) is a correction factor that depends on the amplification gain G of the ICCD camera that
controls the power of the signal. For large gains, we may observe some saturation effects in
the number of counts which would lead to a non-linear response.
3. ξe(λ) is a correction factor that depends on the quantum efficiency of the camera, which is
defined as the ratio between the incoming flux of photons and the output flux of electrons
through the photo-cathode.
4. ρ(∆t) is the correction factor of interest (also referred to as the “non-reciprocity”) that departs
from unity at very low gate widths ∆t.
7
In the following study, the gain is fixed so that the coefficient factor γ(G) can be ignored. Actually,
due to the noise in the number of photon counts from one pixel to the other, we consider an averaged
value of N∆t over the region A = ∆z ×∆λ around a given position z and frequency λ. Therefore,
introducing κ = κ γ(G) ξe(λ), we have
I(z, λ) =1
|A|
∫∫AI(z, λ) dzdλ =
1
|A|
∫∫Aκ ρ(∆t)
N∆t(z, λ)
∆tdzdλ (3)
where I denotes the averaged intensity and |A| the area of A = ∆z ×∆λ. Using the fact that κ is
roughly constant in A, it follows that
I(z, λ) = κρ(∆t)
∆t
(1
|A|
∫∫AN∆t(z, λ) dzdλ
)= κ ρ(∆t)
N∆t(z, λ)
∆t(4)
where N∆t denotes the averaged photon count captured at z and λ during the time period ∆t.
For a large gate width ∆tL, the correction factor ρ should be unity, i.e. ρ(∆tL) ≈ 1. Therefore,
using the same calibration lamp, the same gain G, and considering the same λ and z, we would get
for a small gate width ∆t,
I(z, λ) = κ ρ(∆tL)N∆tL(z, λ)
∆tL= κ ρ(∆t)
N∆t(z, λ)
∆t(5)
that is
ρ(∆t) ≈ N∆tL
∆tL
(N∆t
∆t
)−1
(6)
where we have dropped the dependency on (z, λ). This result actually shows that the non-
reciprocity can be evaluated by defining a model for the number of counts with respect to the
gate width. Note that it could also be determined using the known intensity I(z, λ). However,
for small gate widths and dim radiative sources such as tungsten lamps, the amount of photons
collected by the camera might be too low, which would yield large sources of uncertainties. These
uncertainties would later be incorrectly propagated to the actual measurements as the radiation
from the shock-heated gas is much brighter. The objective is thus to develop a model that can be
calibrated using data obtained at large gate widths and can predict the non-reciprocity at ∆t = 0.1
µs.
Figure 5 gives an interpretation of the fundamental issues associated with the calibration of
the camera. Calibration using the tungsten lamp needs to be performed using a large gate width
(∆t ≥ 10 µs) in order to capture an adequate amount of photon counts emitted from the lamp.
On the other hand, actual experiments are made at much shorter gate widths (∆t ∼ 0.1 µs) to
minimize the spatial smearing due to the very high shock speed. It is thus important to consider
8
Inte
nsity Data Reduction Model
Photon
Count
Gate width
Calibration
ExperimentRegime
Regime
Figure 5: Schematic interpretation of the data reduction model. Calibration of the camera can reliably be performed
at small intensity (tungsten lamp), small counts, and large gate widths. Actual experiments, on the other hand, are
performed with large intensity, large photon counts, and small gate widths.
and validate a data reduction model during calibration of the instrumentation. Figure 6 shows the
total number of photon counts N∆t corresponding to the set of data reported in Table 2. We indeed
observe that the deviation from a purely linear response estimated using ∆t = 10 µs (dashed line)
becomes large as the gate width approaches zero (see inset).
3.2. Models for the non-reciprocity factor
From our previous discussion, and in particular Eq. (6), the correction factor ρ(∆t) can be
estimated by modeling the number of photons N∆t collected by the camera. We assume that this
number should be roughly proportional to the total integrated time the camera remains open and
to the manner the camera opens and closes. Gating of the camera is essentially controlled by the
photo-cathode. Following [26], we propose to model the number of counts based on the linear RC
circuit shown in Fig. 7, composed of one capacitor and two resistors. In this model, the capacitance
C1 takes into account the surface change between the outer and the inner faces of the cathode
and the resistor R1 models the combination of all loads related to the system except that of the
photo-cathode. The voltage Vin(τ), a step function, is introduced as the input signal mimicking
the switch “on” and “off” of the photo-cathode and the output signal is the current I2(τ) through
the resistor R2, or simply the voltage V (τ) = R2I2(τ) (Ohm’s law). Here, τ represents the time
measured from the instant at which the photo-cathode is turned on.
The main hypothesis of the models presented below is that the photon counts are related to
9
Figure 6: Total number of photon counts N∆t collected in calibration experiments with respect to the gate width ∆t
(solid line) and comparison with a purely linear response estimated using ∆tL = 10 µs (dashed line).
the integrated value of the voltage V (τ) over time, i.e.
N∆t ∝∫ ∞
0V (τ) dτ (7)
The tension V (τ) is governed by the differential equation:
dV
dτ+R1 +R2
R1C1R2V =
1
R1C1Vin(τ) (8)
or simplydV
dτ+ α V = α β Vin (9)
where we have used the notation:
α =R1 +R2
R1C1R2, β =
R2
R1 +R2. (10)
Since the input step function Vin(τ) is piecewise constant, i.e.
Vin(τ) =
1, if 0 < τ < ∆t
0, if τ > ∆t(11)
∆t being the gate width, the general solution of the differential equation is given by:
V (τ) = β Vin(τ) + Ce−ατ (12)
10
Resistor
Input
CCondenser
R RResistor
2
in 1
1
VVoltage
Figure 7: A linear RC circuit model for the photo-cathode (see e.g. [26]).
with C a constant that depends on the initial condition. Prescribing V (0) = 0, the solution of the
problem is given by
V (τ) =
β(1− e−ατ
), if 0 ≤ τ ≤ ∆t
β(1− e−α∆t
)e−α(τ−∆t), if τ > ∆t
(13)
In the above definition of the voltage, we assumed that the same RC circuit controls the opening
and closing of the camera. We could imagine that two different circuits are responsible for the two
actions with respective characteristic time 1/α1 and 1/α2. In that case, the solution for the voltage
reads:
V (τ) =
β(1− e−α1τ
), if 0 ≤ τ ≤ ∆t
β(1− e−α1∆t
)e−α2(τ−∆t), if τ > ∆t
(14)
Inserting the solution (14) for the voltage V into (7), the integral is given by:∫ ∞0
V (τ) dτ =
∫ ∆t
0V (τ) dτ +
∫ ∞∆t
V (τ) dτ
=
∫ ∆t
0β(1− e−α1τ
)dτ +
∫ ∞∆t
β(1− e−α1∆t
)e−α2(τ−∆t) dτ
= β(∆t−
(1− e−α1∆t
)/α1
)+ β
(1− e−α1∆t
)/α2
= β[∆t−
(α2 − α1
)(1− e−α1∆t
)/α2α1
](15)
We observe that the second term in the expression for N∆t represents a correction with respect to
the linear regime, denoted as:
Λ(∆t, α1, α2, β) = β(α2 − α1
)(1− e−α1∆t
)/α2α1 (16)
so that
N∆t ∼ β∆t− Λ(∆t, α1, α2, β) (17)
We are now in a position to define several models for the photon counts N∆t:
11
Model M1: In the first model, the simplest, we suppose that the rising and decaying rates are the
same, i.e. α1 = α2 = α. In this case, the correction Λ(∆t, α, α, β) vanishes and the photon
counts is straightforwardly given by:
N∆t = β∆t (18)
We know that this model should fail to provide accurate predictions but will still be considered
to test the reliability of the validation process proposed in the next section.
Model M2: In the second model, we relax the previous assumption and now suppose that the
rising and decaying rates are different. The photon count is thus given by (17). Note that for
∆t = 0, the correction Λ vanishes while for large gate widths, Λ tends to the constant value
β(α2 − α1)/α1α2.
Model M3: In this model, we assume that the input parameter ∆t is not precisely controlled by
the camera. We thus introduce a new parameter δ which represents a correction to the gate
width and which will be referred to as the time delay in the remainder of the sequel. The
model for the photon counts reads:
N∆t = β (∆t+ δ)− β(α2 − α1
)(1− e−α1(∆t+δ)
)/α2α1 (19)
Note that δ can take positive or negative values. In this model, the value of N∆t may be
nonzero as ∆t approaches zero.
Model M4: An alternative to the time delay model is to consider a white-noise type of uncertainty
(e.g., anode dark current due to the thermionic emission [11]) generated by the environment.
We suspect that the noise may become significant at small gate widths and that its magnitude
may vary according to the time period during which experiments are made, which constitutes
an extra source of uncertainty. The noise level is supposed here uniform and equal to ν during
the entire observation period so that the new model is given by:
N∆t = (β + ν)∆t− β(α2 − α1
)(1− e−α1∆t
)/α2α1 (20)
Model M5: In the last model, we suppose that the camera is submitted to a dark noise as in
Model M4 and that there is a time delay as in Model M3. The model reads:
N∆t = (β + ν)(∆t+ δ)− β(α2 − α1
)(1− e−α1(∆t+δ)
)/α2α1 (21)
12
Table 1: Proposed physical models and corresponding model parameters. The symbols 3 or 7 indicate that the given
parameter is or is not part of the model, respectively.
Model α1 α2 β δ ν Photon counts N∆t
M1 7 7 3 7 7 β∆t
M2 3 3 3 7 7 β∆t− Λ(∆t, α1, α2, β)
M3 3 3 3 3 7 β(∆t+ δ)− Λ(∆t+ δ, α1, α2, β)
M4 3 3 3 7 3 (β + ν)∆t− Λ(∆t, α1, α2β)
M5 3 3 3 3 3 (β + ν)(∆t+ δ)− Λ(∆t+ δ, α1, α2β)
Long gate width
Short gate width
Ltt !""
#
different from1
2!
!
Long gate width
Short gate width
Ltt "##
$
Figure 8: Solution of (14) in the cases where α1 = α2 (left) and α1 < α2 (right). In both cases, we show the solutions
for a short gate width ∆t and a long gate width ∆tL. For the short gate width, we observe that not enough time
is left for the camera to fully open. The shaded areas correspond to the value of the integral in (7). Note that the
contributions to the integral from the opening and closing processes become negligible as the gate widths become
large.
Remark 3.1. Additional models could be considered if the electronic system is treated as a more
complicated circuit consisting of resistors, inductors, and capacitors (RLC circuit) [19]. Compari-
son of all different data reduction models will be the subject of future works.
The five models with their parameters are summarized in Table 1. The number of physical
parameters varies between 1 and 5. Finally, we show in Figure 8 examples of the voltage described
by (14).
13
4. Validation process
The contemporary view of predictive computational modeling for physical events presumes that
observational data be acquired for the calibration and validation of models for simple scenarios of
the theory. In addition, one has to consider the predictive model scenario, denoted Sp, for which
one hopes to predict the quantity of interest using a model that survives the validation process. We
briefly describe below the calibration, validation, and prediction processes in an abstract manner,
illustrated in Fig. 9, and based on the work presented in [2, 16, 17, 18]. Other approaches for model
validation in scientific computing have been proposed in [25, 21, 15] and in the references therein.
4.1. Brief description of the proposed validation process
For the purpose of the description, we introduce the following abstract model problem that
consists in finding the solution u such that:
A(θ, S, u(θ, S)) = 0 (22)
where A is an operator representing the model, θ the model parameters, and S some possible
scenario for which the model has been determined. Here, some of the model parameters are assumed
unknown and need to be identified through calibration with respect to given data.
Calibration. Simple calibration scenarios Sc are first considered for which experiments can be run
that provide observables represented by data Dc. One then needs:
1) to introduce the likelihood density function π(Dc|θ) for the calibration scenario based on the
theory represented by (22),
2) to provide the prior probability density function πc(θ) for the parameters and probability
density function π(Dc) for the data,
3) to solve the inverse problem for the posterior pdf σc(θ|Dc) using Bayes’ Theorem (see e.g. [22]),
i.e.
σc(θ|Dc) =πc(θ)π(Dc|θ)
π(Dc)(23)
using sampling methods such as those described in [24, 23, 12, 5, 20]. For simplicity in the
notation, we will write σc(θ) rather than σc(θ|Dc) in what follows.
4) to solve the forward problem (22) for u(σc(θ), Sp), using the prediction scenario Sp, such that
A(σc(θ), Sp, u(σc(θ), Sp)) = 0 (24)
14
(!) (!)
v(D )"
"(D )c
(!) (!)c
v#
c#
"v
"
Estimate
Feedback Control
p
v
c
S
S
S No
Data
Data
Model withRe−CalibratedParameter(s)
Model withCalibrated
Parameter(s)
Validation
v
vScenarioValidation
ScenarioCalibration
Sensitivity/Uncertainty
ScenarioPrediction
usingQoI
Q
Quantification
Yes
Decision
Calibration
c $D ,(Q Q ) <
cQ
Invalid
ConfidenceIncreased
is
Model not
Invalid
Model
Figure 9: The flow of a calibration, validation, and prediction processes. Prior pdf’s of model parameters and pdf
of observational data π(Dc) are provided to calibrate the model for given calibration scenarios Sc. Inverse analysis
yields the posterior pdf σc(θ). This can serve as the prior for the more elaborate validation process which involves new
scenarios Sv and validation data with pdf π(Dv). Inverse analysis yields the posterior σv(θ). The stochastic forward
problems are solved using calibrated and validated data and the corresponding prediction QoI’s are compared. If
they meet a preset tolerance, the model is not invalidated and the calibrated model is used to make the predictions.
Otherwise the model is declared invalid.
and to predict the probability density function πc(Qp) associated with the quantity of interest
Qp estimated using u(σc(θ), Sp).
Validation. Validation scenarios Sv are now selected, usually with the objective of checking one or
more hypotheses of the theory, to produce validation observations θv. One may also hope that these
observations will reflect the ability of the model to deliver acceptable predictions of the quantity of
interest Qp. In the first stage of the validation, the model parameters are re-calibrated by repeating
the same steps as in the calibration process, i.e. one has:
1) to introduce the likelihood function for the validation process, π(Dv|θ),
2) to define a prior pdf for the validation process πv(θ) (one may use the posterior pdf σc(θ))
and a probability density function π(Dv) for the data,
15
3) to solve the inverse problem for the posterior pdf σv(θ) = σv(θ|Dv), i.e.
σv(θ) =πv(θ)π(Dv|θ)
π(Dv)
4) to solve for the solution u(σv(θ), Sp) of the forward problem on the prediction scenario,
A(σv(θ), Sp, u(σv(θ), Sp)) = 0 (25)
and to compute the probability density function πv(Qp) associated with the quantity of in-
terest Qp obtained using u(σv(θ), Sp).
Note that the mathematical model chosen for the prediction can never be validated; it can,
at best, be not invalidated for the specific validation experiments performed. The determination
of a criterion for accepting a model as “not-invalidated” is a subjective decision that requires the
acceptance of a metric to compare the predicted quantities produced by the calibration and the
validation processes, and a tolerance that we establish as an acceptable measure of the predictability
of the model. Thus, our validation process involves comparing πc(Qp) and πv(Qp). Let D denote
a metric and let γtol denote a preset tolerance. We will declare the model as not invalid if
D(πc(Qp), πv(Qp)) < γtol (26)
Several metrics could be thought of to define D in (26). For example, computing the cumulative
probability density functions
cdfc(Qp) =
∫ Qp
−∞πc(Q) dQ and cdfv(Qp) =
∫ Qp
−∞πv(Q) dQ (27)
a comparison of πc(Qp) and πv(Qp) is then afforded by
D(πc(Qp), πv(Qp)) = sup
p∈[0.1,0.9]
∣∣cdf−1c (p)− cdf−1
v (p)∣∣ (28)
Such a metric is illustrated in Fig. 10 and was suggested in [9]. However, the difference between the
cdf’s is evaluated here only in the range p ∈ [0.1, 0.9] in order to exclude the content of the tails due
to possible approximation errors in the estimation of the pdf’s. An immediate consequence is that
the quantity thus computed will be necessarily equal to or smaller than the quantity one would
evaluate using the whole unit interval and that one could be conservative in accepting models.
However, this could be easily offset by the choice of the tolerance in the rejection criterion.
16
p
c vpdf pdf
1
Quantity of Interest
Q00
cdf =cumulativedensityfunction c v
p
Metric distance < tol
cdf cdf
cdf
Q00
1EXAMPLE OF METRIC
Quantity of Interest
Figure 10: Illustration of the metric D defined in (28).
4.2. Application of the validation process to the non-reciprocity models
As mentioned above, calibration and validation processes rely on data. In our problem, data-
sets consist in 512× 512 matrices representing the photon counts as a function of wavelength and
position. An example of a digital image obtained during calibration of the instrumentation is shown
in Fig. 11 and was acquired using a tungsten lamp. Since the time response of the camera is the same
regardless of the position and wavelength considered, we can restrict ourselves to the particular area
of the CCD array where the camera is most sensitive. Referring to (3)–(4), observable quantities
consist of averaged photon counts N∆t over the area shown in Fig. 11 to reduce the effect of noise
in the measurement, the effect of input radiation and wave-dependent quantum efficiency, and
position dependence. We also see in Fig. 11 that the measurements are fairly reproducible.
We show in Fig. 12 (left) the numbers of counts N∆t averaged using either the wavelength
range 600–700 nm or 700–800 nm. We observe that the choice of the wavelength range has little
effect on the observations. We also show in Fig. 12 (right) the value of the correction factor (non-
reciprocity) using data obtained from experiments performed at different dates. The non-linear
behavior is observable only for gate widths smaller than 1.5 µs and the results are in fair agreement
with each other for values larger than 1 µs. However, we also observe severe discrepancies (as high
as 30 percent) for small values of the gate width. These large variations are attributed to the small
number of counts at such gate widths and reinforce here the idea that a model is needed to predict
the correction factor at ∆t = 0.1 µs. We provide in Table 2 one set of data that we shall use for
17
500 600 700 800 900Wavelength [nm]
0
50
100
150
200
Cou
nts
[-]
- 1 Test- 2 Test- 3 Test- 4-11 Test
Spectrally integrated curve
Figure 11: Example of data used in the calibration of the data reduction models. (Left) Digital image obtained when
exposing the camera to a tungsten lamp. (Right) Reproducibility of experiments: the number of counts N∆t is shown
versus the wavelength λ at the distance z indicated on the right image in the case of several calibration shots.
the validation of our data reduction models.
The likelihood density functions are constructed as follows. Let Di be the observable data for
a given gate width ∆ti and let X(∆ti) be the model output (i.e. the number of counts N∆ti) for
the same gate width, i = 1, . . . , ND (ND = NC for the calibration data-set and ND = NV for the
validation data-set). Assume that the data points are statistically independent from each other
and consider a multiplicative error e between Di and X(∆ti) so that the latter is always of the
same sign as that of the former (positive), i.e.
Di = Xi exp(e) or, simply, e = lnDi − lnX(∆ti) (29)
The error e is chosen here to be Gaussian with zero mean and variance σ2 and includes the ex-
perimental and modeling errors. The choice of the error model is based on the assumption that
the errors are proportional to the values of the observations and that the errors at the data points
are all statistically independent of each other. Results of the calibration process may indeed de-
pend on the choice of the error model (see e.g. [14]). Nevertheless, the value of σ2 should provide
here some useful information about the model adequacy. Let σexp and σmod refer to the standard
deviations associated with the experimental errors and the modeling errors, respectively. These
can be viewed as unknown parameters of the error model and can be identified along with the
other model parameters during the calibration process. In this case, the prior pdf’s of the stan-
dard deviation (equivalently the variance) need to be provided: for σexp, it is based on some prior
18
Table 2: Data set used in the calibration and validation of the data reduction models.
Gate width ∆t Values of photon counts N∆t Coefficient of
in µs (× 105) Variation
0.05 0.25773 0.25878 0.25562 0.25573 0.523
0.10 0.44993 0.45040 0.45401 0.44889 0.427
0.20 0.78555 0.78846 0.78787 0.78460 0.202
0.30 1.18851 1.19294 1.18762 1.19091 0.175
0.40 1.65956 1.65786 1.66407 1.65570 0.185
0.50 2.15651 2.16631 2.16610 2.15334 0.266
0.60 2.70125 2.70294 2.69529 2.69464 0.112
0.70 3.30536 3.30940 3.29062 3.30003 0.213
0.80 3.93567 3.94550 3.94070 3.95361 0.167
0.90 4.64449 4.63364 4.61162 4.62549 0.259
1.00 5.34160 5.33976 5.33305 5.34784 0.098
2.00 12.98473 12.99352 12.87610 12.96557 0.360
3.00 20.57746 20.61848 20.48911 20.56103 0.227
4.00 28.20695 28.28952 28.23362 28.27682 0.117
5.00 35.88943 36.03104 35.78025 35.86519 0.251
10.00 74.72930 74.72930 0.000
Table 3: Prior probability density functions for the calibration of the model parameters.
Parameters α1 α2 β δ ν log(σ2)
Uniform pdf’s π [0.3,100.0] [0.3,100.0] [5.0,7.0] [0.0,1.0] [0.0,5.0] [-8.0,-1.0]
19
0 0.5 1 1.5 2Gate Width [µs]
0
0.05
0.1
0.15
0.2
Nor
mal
ized
Int
egra
ted
coun
ts [
nm]
λ = 600-700 nm λ = 700-800 nmTime linear curveFitting Parabola
τ = 0.293 µs
0.01 0.1 1 10Gate Width [µs]
0
0.5
1
1.5
2
2.5
3
Rec
ipro
city
Fac
tor
Pecos - Data (Nov)Parabola -LSEAST - dataPecos - Data (Dec)Pecos - Data (Mar)
Figure 12: (Left) Numbers of counts N∆t versus ∆t, averaged using either the wavelength range 600–700 nm or 700–
800 nm. (Right) Value of the correction factor (non-reciprocity) using data obtained from experiments performed at
different dates.
knowledge of the experimental uncertainty; for σmod, the pdf should be chosen as large as possible.
However, in practice, it is in general difficult to distinguish one from the other, especially if σexp
is small. Consequently, the two components are usually combined together into the total variance
σ2 = σ2exp + σ2
mod. In the present study, the value of σ2 obtained from the calibration process will
provide an indication of the value of σ2mod, i.e. the modeling error, since the same experimental
data will be used for all models. Based on these assumptions, the likelihood function reads
π(D|θ) =1
(√
2πσ)NDexp
[− 1
2σ2
ND∑i=1
(Di − Xi)2
](30)
where Di and Xi stand for lnDi and lnX(∆ti), respectively. In the numerical experiments shown
below, the posterior probability density functions are obtained by sampling the parameter spaces
using the Markov Chain Monte Carlo method proposed in [7]. For the calibration process, we use
up to 1,000,000 samples to sweep the parameter spaces and for the forward problem, we consider
5,000 samples generated from the parameter posterior distributions to evaluate the pdf’s and cdf’s
of the quantity of interest. Note that the whole process takes here less than ten minutes on a recent
serial Linux machine thanks to the simplicity of the models. In addition, extensive statistical
convergence tests using different sample sizes were performed in order to ensure that the pdf’s of
the parameters and of the quantities of interest are accurately approximated.
Depending on the models Mj , j = 1, . . . , 5, the set of model parameters θ will consist of subsets
of (α1, α2, β, δ, ν) and of the variance σ2 for the error. The prior probability density functions for
20
0 1 2 3 4 5 6 7 8 9 10Gate Width [µs]
0
0.2
0.4
0.6
0.8
1
No
rma
lize
d I
nte
gra
ted
co
un
tsManufactured data
0 0.5 1 1.5 20
0.05
0.1
0.15
Figure 13: Manufactured data using Model M5.
1 1.5 2 2.5 3α
1
0
5
10
15
20
Marg
inal P
DF
s
[0.5,10.0]
[0.9,10.0]
(a) Model parameter α1
0.06 0.08 0.1 0.12 0.14δ
0
1
2
3
4
Ma
rgin
al P
DF
s
[0.5,10.0]
[0.9,10.0]
(b) Model parameter δ
Figure 14: Marginal posterior pdf’s of α1 and δ obtained using Model M5 and manufactured data.
the model parameters are in general chosen based on existing information. In this absence of such
information, we choose uniform prior pdf’s for the different parameters as indicated in Table 3.
4.3. Verification of the validation process with manufactured data
Before using the actual experimental data, we propose to check that the validation process
can provide adequate results. WIth that goal in mind, we use one of the mathematical models
presented above, namely M5, to produce observable quantities to which noise is added to generate
the calibration and validation data (see Fig. 13) similar to those from the actual experiments (see
Fig. 6(a)). These data sets will be referred to as “manufactured data” in the following. The nominal
values of the model parameters are chosen as follows: α1 = 2.0, α2 = 50.0, log(β) = 6.0, δ = 0.1
and log(ν) = 5.0. Additional multiplicative noise based on a Gaussian distribution with zero mean
21
0 1 2 3 4 5Reciprocity
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
[0.5,10.0][0.9,10.0]
22%
0 1 2 3 4 5 6 7 8 9 10Reciprocity
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
[0.5,10.0][0.9,10.0]
81%
0 1 2 3 4 5 6 7 8 9 10Reciprocity
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
[0.5,10.0][0.9,10.0]
21%
0 1 2 3 4 5 6 7 8 9 10Reciprocity
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
[0.5,10.0][0.9,10.0]
91%
0 1 2 3 4 5 6 7 8 9 10Reciprocity
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
[0.5,10.0][0.9,10.0]
16%
(a) (b)
(c) (d)
(e)
Figure 15: Marginal posterior cumulative density functions of the non-reciprocity at ∆t = 0.1 using (a) M1, (b) M2,
(c) M3, (d) M4, and (e) M5.
22
and standard deviation σ = 0.01 is considered here. In the insert of Fig. 13, we observe that the
solid line obtained from fitting the data in the linear region diverges from the data computed at
small gate widths due to the non-linear behavior of the model. The posterior pdf’s of parameters
α1 and δ are shown in Fig. 14 as illustration of the calibration process. The prior pdf’s were
chosen uniform, as in Table 3, but with smaller ranges since we know here the nominal values of
the parameters. The dashed lines were obtained using the data in the range [0.9,10.0] while the
solid lines were obtained considering the data in the whole range [0.5,10.0]. We observe that the
nominal values α1 and δ are recovered once a sufficient amount of data is used.
We show in Fig. 15 the cdf’s of the non-reciprocity computed at ∆t = 0.1 using the calibrated
models. Recall that Model M1 is a linear model and as such should be rejected. Although the
posterior cdf’s from the two data sets remain close to each other, we do observe that the variance
of the corresponding pdf’s slightly diverges when more data is used in the calibration process.
Therefore, the model can be rejected based on the latter criterion. In the case of both M2 and M4,
the cdf’s using the two data sets are far apart. These models should be invalidated. Finally, in the
case of M3 and M5, the maximum distance between the cdf’s (quantified as 21% for M3 and 16% for
M5) remains relatively small. We also see that the variance decreases as the amount of data used in
the calibration increases. Therefore, we may conclude that these two models cannot be invalidated.
Such a result is actually expected in the case of M5 since the manufactured data was generated
using that model. However, it is interesting to see that Model M3 also provides acceptable results,
in the sense that the model cannot be invalidated by the proposed methodology. We will observe
similar results in the next section using the actual data. Moreover, this study reveals that the
introduction of the white-noise into the model through parameter ν is not as important as the time
delay δ. We may conclude from this study that it is actually unclear how ∆t is measured with
respect to the opening and closing times of the camera.
5. Numerical results
The main objective in the following experiments is to assess the validation process proposed
in Section 4 and to analyze the various outcomes and possible shortcomings of the process using
actual data. Our view is that this process involves many a priori choices and that the effects of
these choices should be carefully examined before drawing any conclusions. We recall that the
quantity of interest for assessing the validity of the models Mj , j = 1, . . . , 5, is the correction factor
(non-reciprocity) at ∆t = 0.1 µs, which accounts for the nonlinear behavior with respect to the
23
gate width of the camera. The most important choice that is made is maybe the distribution of
the available data into a calibration data set and a validation data set. In this study, we select
for the calibration stage the data points corresponding to ∆ti = 0.9, . . . , 10.0 µs while we use for
the validation stage the data points ∆ti = 0.5, . . . , 10.0 µs, as specified in Table 2. Presently,
it is not clear what the best criterion is for determining whether experiments can be prescribed
to be either calibration or validation data. Determining this criterion is a topic requiring further
research. However, for the purpose of this paper, several variations on the choice of calibration and
validation experiments are used in order to assess the proposed Bayesian validation methodology.
Furthermore, it should also be highlighted that the calibration and validation scenarios are similar
in this study, the only difference being in the choice of the input parameter ∆ti. However, at this
point of the study, we assume it is the only available data, a fact that one is often faced with as
experimental data are usually scarce. Finally, the choice of the prior distributions for the various
model parameters are provided in Table 3 and the form of the likelihood functions is as given
in (30).
5.1. Results of the validation process
In this section, the validation process is exercised exactly as described above using the cumu-
lative density function of the non-reciprocity at ∆t = 0.1 µs to assess the validity of the models
according to the criterion (28). The tolerance is arbitrarily set here to 25 percent (just for the
purpose of the discussion here, i.e. not based on particular decision-making). The estimated cdf’s
for the five models are shown in the middle column of Fig. 16. We also display in the left column
of the same figure the mean and 95 percent confidence interval (CI) of the non-reciprocity with
respect to the gate width. Although each model Mj , j = 1, . . . , 4, can be viewed as a submodel of
M5, we observe that the predictions from the validation process are very different from one model
to the other.
Model M1. In this case, the predicted non-reciprocity as a function of the gate width is simply
constant, as expected, due to the linear nature of the model; the relative distance between the cdf’s
is estimated at 20 percent. Such a result could be viewed as acceptable; however, the issue here is
that the mean value of parameter σ is fairly large, as observed in the right column of Fig. 16 and
that the variance of the posterior pdf clearly increases as the amount of data used in the calibration
of the model increases.
24
0.2 0.4 0.6 0.8 1Gate width (µs)
0
0.5
1
1.5
2
2.5
3R
ecip
roci
ty[0.5,10.0] 95 % CI[0.5,10.0] mean[0.9,10.0] 95% CI[0.9,10.0] mean
0.2 0.4 0.6 0.8 1Gate width (µs)
0
5
10
15
20
Rec
ipro
city
[0.5,10.0] 95 % CI[0.5,10.0] mean[0.9,10.0] 95% CI[0.9,10.0] mean
0 1 2 3 4 5 6 7Reciprocity
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.9,10.0]
65%
-8 -7 -6 -5 -4 -3 -2 -1 0Log( 2)
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.9,10.0]
0.2 0.4 0.6 0.8 1Gate width (µs)
0
5
10
15
20
Rec
ipro
city
[0.5,10.0] 95 % CI[0.5,10.0] mean[0.9,10.0] 95% CI[0.9,10.0] mean
0 1 2 3 4 5 6 7Reciprocity
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.9,10.0]
92%
-8 -7 -6 -5 -4 -3 -2 -1 0Log( 2)
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.9,10.0]
0 0.2 0.4 0.6 0.8 1Gate width (µs)
0
5
10
15
20
Rec
ipro
city
[0.5,10.0] 95 % CI[0.5,10.0] mean[0.9,10.0] 95% CI[0.9,10.0] mean
0 1 2 3 4 5 6 7Reciprocity
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.9,10.0]
98%
-8 -7 -6 -5 -4 -3 -2 -1 0Log( 2)
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.9,10.0]
Figure 16: (Left) Posterior mean and 95% CI for the non-reciprocity, (middle) marginal posterior cumulative density
functions of the non-reciprocity at ∆t = 0.1 µs, and (right) posterior pdf of parameter σ obtained for models M1 to
M5 (top to bottom).
25
0 20 40 60 80 1001,
2
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
1[0.5,10.0]
2[0.5,10.0]
1[0.9,10.0]
2[0.9,10.0]
Figure 17: Marginal posterior cdf’s for α1 and α2 for M2.
Model M2. We observe that the non-reciprocity monotonically increases as the gate width tends to
zero, using either the data set [0.9, 10.0] or [0.5, 10.0]. However, the predicted reciprocities obtained
using the calibrated and re-calibrated models diverge from each other at small ∆t as the former
provides a non-reciprocity that tends to infinity while the latter predicts an almost constant non-
reciprocity as a function of the gate width. The large difference in the two results is mainly due to
the fact that the cdf of α2 significantly changes from using the first data set to using the second one
(see Fig. 17); hence the large discrepancy between the two cdf’s profiles, as seen in Fig. 16. This
could have been expected using a simple expansion analysis around ∆t = 0. In this case, we easily
derive an estimate for N∆t as β∆t α2/α1 which would allow us to conclude that the non-reciprocity
at ∆t = 0 should approximately converge to α2/α1. In other words, when α2 becomes much larger
than α1, the non-reciprocity should tend to ∞.
Model M3. The introduction of the time delay δ yields a new feature in the non-reciprocity, in that
the correction factor now vanishes at ∆t = 0. Nevertheless, we still observe a large difference in the
cdf of the non-reciprocity due maybe to the fact that the mean value of δ becomes far from being
negligible when using the validation data set as shown in Fig. 18. We also show in the figure the
cdf’s of the calibrated parameters α1 and α2. We observe that the mean of α1 is small and that of
α2 is large, suggesting that the camera opens at a slow rate and quickly closes.
Model M4. The results obtained for M4 seem to be a combination of the ones obtained with models
M3 and M2. In Fig. 19, we show the posterior cdf’s of ν for M4, which significantly differ with
respect to the chosen data sets. For small values of ν, M4 behaves more as M2, for large values, it
behaves as M3. However, the distance between the cdf’s of the non-reciprocity obtained using the
26
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1M
argi
nal C
DFs
[0.5,10.0][0.9,10.0]
0 20 40 60 80 1001,
2
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
1[0.5,10.0]
2[0.5,10.0]
1[0.9,10.0]
2[0.9,10.0]
Figure 18: Marginal posterior cdf of δ (left) and cdf’s of α1 and α2 (right) in the case of M3.
0 1 2 3 4 5 6Log(ν)
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
ν[0.5,10.0]ν[0.9,10.0]
0 1 2 3 4 5 6Log(ν)
0
0.2
0.4
0.6
0.8
1
Ma
rgin
al C
DF
s
ν[0.5,10.0]ν[0.9,10.0]
Figure 19: Marginal posterior cumulative density functions of ν in the case of M4 (left) and M5 (right).
calibration and validation data sets is still large for this model.
Model M5. Finally, in the case of M5, we observe slightly better results than those obtained with
the other models, as shown in Fig. 16. However, it is not clear that the addition of the back noise
was necessary to improve the model. In fact, we see that the non-reciprocity as a function of ∆t
has similar profiles in this case as those obtained using model M3. We also show in Fig. 19 the
behavior of ν for M5. We observe in this case that the two cdf’s remain close to each other when
using the calibration or validation sets.
5.2. Discussion
With the criterion D(πc(Qp), πv(Qp)) < γtol and γtol = 25 percent, we would have to conclude
that M1 is the only valid model among the five models. Such a conclusion would be contradictory
with our assumptions, as M1 is a linear model when the non-reciprocity has clearly a nonlinear
27
behavior with respect to the the gate width. Moreover, none of the other models Mi, i = 2, . . . , 5,
satisfies the proposed acceptance criterion as the distance between the cdf’s of the non-reciprocity
is in all cases too large.
At this stage of the analysis, the main question that we should ask is why the validation process
yields such a deceptive conclusion. For model M1, the answer is relatively straightforward. The
modeling error, embodied by the standard deviation σ, is large, meaning that the model is able to
reproduce the calibration data simply by increasing the modeling error. In other words, the misfit
between the observations and the quantities predicted by the model is too large for the model to be
acceptable. In the case of the other models, the modeling error remains small indicating that the
models are able to predict relatively well the non-reciprocity at large gate widths, i.e. ∆t ≥ 0.5.
If we look closely at the values of σ obtained using the data at ∆ti = 0.5, . . . , 10.0 µs for each
model Mi, i = 2, . . . , 5, we observe that these are slightly smaller for M3 and M5 than for M2 and
M4. This clearly indicates that the introduction of the time delay δ into the model reduces the
modeling error, unlike the white-noise ν. This observation is consistent with the results obtained
using the manufactured data. The fact that the difference between the cdf’s is large suggests that
the solution of the inverse problem is still changing a lot when going from the calibration data to
the validation data. The question here is whether the calibration data set includes enough data
points to properly identify the parameters of the models. We observe for instance in Fig. 12 that
the data points corresponding to ∆t = 0.9, . . . , 10 µs only describe the beginning of the nonlinear
behavior of the correction factor. We show in Fig. 20 the evolution of the cdf of the non-reciprocity
when the parameters of models M3 and M5 are calibrated by adding one data point at a time
from the initial calibration set. From the results in the figure, it is clear that the conclusions from
the validation would be different if we would have chosen different data sets such as [0.5, 10.0] and
[0.6, 10.0], or even [0.7, 10.0] in the case of M5. It follows that another issue is to decide how to
split the available data into calibration and validation data sets. It appears that this choice may
be crucial and it would be extremely useful to devise methods that would help confirm that the
choice made is actually an appropriate choice.
The present application of the validation process is a particular case; indeed, the calibration
and validation scenarios are similar and the observed data and final quantity of interest are simply
the non-reciprocity evaluated at different gate widths. Nevertheless, if one considers the proposed
validation process as a general procedure, it should also work for the current validation of the data
reduction model. Examining the results shown above just reinforces the idea that many systematic
28
0 1 2 3 4 5 6 7Reciprocity
0
0.2
0.4
0.6
0.8
1M
argi
nal C
DFs
[0.5,10.0][0.6,10.0][0.7,10.0][0.9,10.0]
0 1 2 3 4 5 6 7Reciprocity
0
0.2
0.4
0.6
0.8
1
Mar
gina
l CD
Fs
[0.5,10.0][0.6,10.0][0.7,10.0][0.9,10.0]
Figure 20: Evolution of the cumulative density functions of the non-reciprocity predicted using M3 (left) and M5
(right) calibrated by adding one data point at a time from the initial calibration set.
sanity checks need to be developed in order to design a robust validation process. This is work in
progress as well as the topic of future research as model validation for scientific computing is by no
means a solved problem.
5.3. Comparisons with actual experimental data
When calibrating data reduction models, it is in general desirable and preferable that calibration
data be collected in the same regimes as those of the actual experiments. In this case, it would be
unnecessary to validate the models since they would already be calibrated for the regime of interest.
However, it may happen, as it is the case here, that the conditions of the actual experiments are to
difficult to reproduce by other means – e.g. the tungsten lamp is much dimer than the radiation
produced in the shock tube. We nevertheless choose here to compare the numerical predictions
of the photon counts using our models with calibration measurements obtained for ∆t ≤ 0.5
µs, although we are aware that the accuracy of the measuring instruments at small gate widths
is questionable (see Fig. 12) and that these data were deliberately left aside in the calibration
process of the models. The results are shown in Fig. 21 where all data points corresponding to
∆t = 0.5, . . . , 10 µs are used for the calibration of the models.
We observe that the data lie within the 97.5 percent CI only in the case of model M1. This was
actually expected as we already know that the model exhibits large uncertainties. Uncertainties
from the other models are in general much smaller and the data points at small gate widths do
not always lie within the 97.5 percent CI. It is nevertheless noticeable that models M3 and M5
predict more accurately the number of photon counts at ∆t = 0.1 µs than models M2 and M4 do.
29
Figure 21: Predicted number of photon counts N∆t using the calibrated models Mi, i = 1, . . . , 5.
30
Time
Figure 22: Comparison of the photon flux obtained from experiments involving a pulsing lamp and that predicted
using model M3.
We may conclude that parameter δ (the time delay) plays a fundamental role in these models, in
other words, that the input parameter ∆t is not necessarily well controlled for small openings of
the camera. However, it is still an open question to decide which of the two models M3 and M5 is
the most appropriate.
Since the tungsten lamp is rather dim and does not allow to understand how the camera opens
and closes, a new calibration setting has been considered during the course of the research project.
The main idea was to use a very small time scale repetitively pulsing lamp to record the photon
flux with respect to time and observe the opening and closing of the spectrometer slit. Preliminary
results from one measurement at ∆t = 0.5 µs are summarized in Fig. 22. This new set of data
constitutes an invaluable source of information: 1) the gate of the camera seems to open slowly and
to quickly shut down; 2) the input parameter ∆t does not seem to be precisely controlled at small
gate widths; 3) there seems to be a long period during which the camera collects photons before
the gate fully opens, a phenomenon that is not directly described by our models. Our plan in the
near future is to improve this new experimental setting and to use the data as a new validation
data set for the validation of our models. By then, it will maybe become necessary to explore new
models of the camera, using for instance the concept of an RLC circuit, based on new evidence
that the current models fail to correctly predict the non-reciprocity at the gate width of interest.
31
6. Conclusions
This paper deals with the validation of data reduction models used in the acquisition of radiative
intensities by cameras during shock tube experiments. The objectives were to show that the
validity of data reduction models should be assessed in a manner similar to that used for the
validation of mathematical models of physical phenomena, to develop simple data reduction models
to convert photon counts into radiative intensities, to apply the validation process described in
Section 4 to these models, and to analyze our confidence level in the models as the result of this
process. A clear advantage of the proposed validation process is that it allows to directly quantify
uncertainties in predicted quantities of interest, namely the correction factor with respect to the
number of counts at small gate widths in this study. We have developed in this work five different
models of various complexity that are supposed to describe the opening and closing of intensified
charge-coupled device cameras and analyzed their capability to predict the correction factor. The
preliminary results for the assessment of this validation methodology have highlighted the need for
a robust criterion to distinguish between calibration and validation experimental data. Therefore,
a definitive conclusion cannot yet be provided to accept or reject the various prediction models.
Furthermore, the assessment has shown that rigorous physical checks of results are required. This
was clearly highlighted with the validation of the simple linear model. Despite knowledge that this
model lacks sufficient physics to offer accurate predictions over the full domain of gate widths, the
model past the set criterion.
This study has allowed us to identify and put forth a number of issues that would need to be
addressed to develop a robust validation process. The current process involves the calibration of the
model parameters using two different data sets. One issue is thus to decide whether the available
data provide sufficient information to properly calibrate the parameters and when the solution of
the calibration problem is sufficiently accurate. Another issue is to assess whether the data is
correctly split into calibration data and validation data, in other words, whether the selection of
scenarios is adequate to decisively compare the cumulative density functions of the quantity of
interest so as to validate or invalidate one of the hypotheses about the models. These theoretical
issues will be the subject of future work.
Acknowledgments. The support of this work by the Department of Energy under Award Number
DE-FC52-08NA28615 is gratefully acknowledged. In addition, the authors would like to thank
Drs. Bogdanoff, Clemens, Cruden, Jagodzinski, and Varghese for their insight during many fruitful
32
discussions. They are also sincerely grateful to Mr. Martinez at NASA for his help with the
experimental calibration procedure. In fact, the modelers were fortunate to be given access to the
raw data, to be allowed to visit the EAST facility, and to interact with the personnel there, as
it constituted an invaluable opportunity, if not a requirement, to develop and validate the data
reduction models.
References
[1] T. Araki, T. Uchida, and S. Minami. A simple photomultiplier gating circuit for the nanosecond
region. Japanese Journal of Applied Physics, 15:2421–2426, 1976.
[2] I. Babuska, F. Nobile, and R. Tempone. A stochastic collocation method for elliptic partial
differential equations with random input data. SIAM Journal on Numerical Analysis, 45:1005–
1034, 2007.
[3] I. Babuska, F. Nobile, and R. Tempone. A systematic approach to model validation based
on Bayesian updates and prediction related rejection criteria. Computer Methods in Applied
Mechanics and Engineering, 197:2517–2539, 2008.
[4] J. L. Beck and L. S. Katafygiotis. Updating models and their uncertainties. i: Bayesian
statistical framework. ASCE Journal of Engineering Mechanics, 124:455–461, 1998.
[5] D. Calvetti and E. Somersalo. Introduction to Bayesian Scientific Computing. Springer, New
York, 2007.
[6] S. H. Cheung and J. L. Beck. New Bayesian updating methodology for model validation and
robust predictions of a target system based on hierarchical subsystem tests. Computer Methods
in Applied Mechanics and Engineering, 2009. Accepted for publication.
[7] S. H. Cheung and J. L. Beck. Calculation of posterior probabilities for Bayesian model class
assessment and averaging from posterior samples based on dynamic system data. Computer-
Aided Civil and Infrastructure Engineering, 25:304–321, 2010.
[8] B. Cruden, R. Martinez, J. Grinstead, and J. Olejniczak. Simultaneous vacuum-ultraviolet
through near-IR absolute radiation measurement with spatiotemporal resolution in an electric
arc shock tube. In Proceedings of 41st AIAA Thermophysics Conference, 2009.
33
[9] S. Ferson, W. L. Oberkampf, and L. Ginzburg. Model validation and predictive capability for
the thermal challenge problem. Computer Methods in Applied Mechanics and Engineering,
197:2408–2430, 2008.
[10] J. H. Grinstead, J. Olejniczak, M. C. Wilder, M. W. Bogdanoff, G. A. Allen, and R. Lilliar.
Shock-heated air radiation measurements at lunar return conditions: Phase I EAST Test
Report. NASA Report EG-CAP-07-142, NASA, 2007.
[11] C. B. Johnson and L. D. Owen. Image tube intensified electronic imaging. In Handbook of
Optics, Vol. 1, pages 21.1–21.32. McGraw-Hill, New York, 1995.
[12] J. Kaipo and E. Somersalo. Statistical and Computational Inverse Problems. Springer, New
York, 2005.
[13] K. Miki, M. Panesi, and E. Prudencio. On the (in)validation of a thermochemical model with
east shock tube radiation measurements. 48th AIAA Aerospace Sciences Meeting Including
the New Horizons Forum and Aerospace Exposition, Orlando, Florida, 2010.
[14] K. Miki, M. Panesi, E. E. Prudencio, and S. Prudhomme. Probabilistic models and uncertainty
quantification for the ionization reaction rate of atomic Nitrogen. Journal of Computational
Physics, 2011. Submitted.
[15] W. L. Oberkampf and C. J. Roy. Verification and Validation in Scientific Computing. Cam-
bridge University Press, 2010.
[16] J. T. Oden, A. Hawkins, and S. Prudhomme. General diffuse-interface theories and an approach
to predictive tumor growth modeling. Mathematical Models and Methods in Applied Sciences,
20(3):477–517, 2010.
[17] J. T. Oden, R. Moser, and O. Ghattas. Quantification of uncertainty in computer predictions
– Part I. SIAM News, 43(9), November 2010.
[18] J. T. Oden, R. Moser, and O. Ghattas. Quantification of uncertainty in computer predictions
– Part II. SIAM News, 43(10), December 2010.
[19] R. Pasqualotto and P. Nielsen. GaAs photomultiplier for LIDAR Thomson scattering. Review
of Scientific Instruments, 74:1671–1674, 2003.
34
[20] E. Prudencio and K. Schulz. The parallel C++ statistical library QUESO: Quantification of
uncertainty for estimation, simulation and optimization. IEEE IPDPS, 2011, Submitted.
[21] P. J. Roache. Fundamentals of Verification and Validation. Hermosa Publishers, 2009.
[22] C. Robert. The Bayesian Choice. Springer Verlag, second edition, 2004.
[23] C. P. Robert and G. Casella. Monte Carlo Statistical Methods. Springer Verlag, second edition,
2005.
[24] A. D. Sokal. Monte Carlo Methods in Statistical Mechanics: Foundations and New Algorithms.
Lectures at the Cargese Summer School on “Functional Integration: Basics and Applications”,
September 1996.
[25] A. Tarantola. Inverse Problem Theory. SIAM, 2005.
[26] T. Woldeselassie. Improved photomultiplier tube for positron emission tomography. Medical
& Biological Engineering & Computing, 27:281–287, 1989.
35
top related