DATA REDUCTION AND ERROR ANALYSIS FOR THE ION PROBE

Class notes for GG710, given by Gary R. Huss, Fall, 2015

This document summarizes the statistical tools that are used to evaluate data collected by the ion microprobe. The presentation here is intended as a practical guide and does not include all of the background and derivations that you will find in a good statistics book. I will try to convey the concepts that are directly relevant to measurements using the ion microprobe. If you are interested in studying statistics in more depth, I recommend “Data Reduction and Error Analysis for the Physical Sciences,” by Bevington and Robinson (McGraw Hill). Random and Systematic Errors

It is not possible to exactly measure the composition of a material by the ion probe. We do not count all of the atoms, so we must infer the composition from the sample of atoms that we do count. There are many reasons for this. First, only about 1 in 1000 to 10,000 of the atoms sputtered from the sample make it into the detector. This is because only a small fraction of the sputtered material comes off the sample as an atomic ion in the right charge state, and only a fraction of those ions make it through the mass spectrometer to the detector. Many atoms come off the sample as molecules, neutral atoms, or ions of charge other than the +1 or -1 that we generally measures, and many of the ions of correct charge are blocked in the mass spectrometer by the slits and apertures we use to tune the mass spectrometer. We assume that the fraction of appropriately charged ions that reach the detector is representative of the atoms in the sample, or that any differences between them, such as that caused by mass-dependent instrumental fractionation, can be understood and corrected for.

Measurement uncertainties are typically called “errors”. In this usage, the error is the difference between an observed value and the “true” value. But because we do not know the true value, it is necessary to determine in a systematic way the likely magnitude of the difference between our measurement and the true value. If reported correctly, the smaller the uncertainty, or error, the better we know the “true” value. I will not discuss “errors” that are caused by mistakes in the experiment or in calculation of the result. These clearly should be avoided, and it is up to the experimenter to devise tappropriate safeguards against these mistakes. Here I discuss the possible and unknown differences between our measured value and the “true” value that are inherent in the measurement process. Errors that are reported with ion probe data (or any other measurement data) give a quantitative assessment of how large this difference is likely to be.

Random errors can be thought of as fluctuations in the observed results from measurement to measurement that require the analyst to make repeated measurements to yield a precise result. Counting errors are random errors that derive solely from the fact that we have not measured every atom in the sample, and the subset of atoms that we do measure may not have exactly the composition of the whole. Counting errors describe how well the composition of the ions arriving in the detector represents the true composition. If we have done everything correctly, counting errors should dominate the uncertainty in our measurements. There are other types of random errors that can be treated in much the same way as counting errors. For example, if several points on a sample or standard are measured, and the degree of sample charging is different for each spot, the variations in instrumental mass fractionation tend to be random around the typical mass fractionation value.

2

Systematic errors can be of several types. There are systematic errors associated with calibration of detectors and in determining the level of mass-dependent fractionation produced in the mass spectrometer, with variations in tuning of the instrument, and with differences in the fraction of each element that is in the correct ionization state after sputtering, among other things. It is necessary for the analyst to consider all of these potential sources of error and evaluate them appropriately. There is little that is more embarrassing and that can affect someone’s scientific career more than to publish an exciting experimental result and then have to retract it later when you discover that the result was wrong, particularly when the mistake is something that should have been identified during the data analysis. The only thing worse is that someone else discovers the error and publishes the correct result.

You have heard the terms “precision” and “accuracy” with regard to experimental data. Counting statistical errors describe the precision of a measurement, which should be thought of as the reproducibility of the measurement. This is the degree to which further measurements will show the same result. On the other hand, the accuracy of a measurement is the degree to which the measurement accurately represents the true state of the measured system. A measurement can be highly precise and can be repeated to a very high degree of precision, and yet be inaccurate because of a systematic error introduced by the experiment itself. A result is considered to be “valid” if it is both accurate and precise. In general, the reported error or uncertainty on a result should reflect the degree to which the experimenter believes the result to be valid. In other words, the error should describe random uncertainties, usually dominated by statistical errors, and an estimate of any systematic uncertainties introduced by the measurement.

In this document, I attempt to provide a comprehensive discussion of statistical and other random errors. Systematic errors will be discussed, but in a much less comprehensive way. Poisson and Gaussian statistics

The primary statistical tools for analyzing random errors in counting experiments are probability distributions. These distributions allow you to estimate the most probable value for a parameter. There are several different probability distributions. The Binomial distribution gives you the probability of getting exactly k successes in a sequence of n independent yes/no experiments, each having a probability p of success.

Pr X = k( ) = n!k! n− k( )!"

#$$

%

&'' p

k 1− p( )n−k (1)

for k = 0, 1, 2, …, n

The Binomial distribution describes coin-toss experiments, for example. The Poisson distribution is more relevant to our problem. It is used for counting

experiments where the data represent the number of items or events observed per unit time. It is important for studying random processes. The Poisson distribution gives the probability that a given number of events (k) occur in a fixed time interval if the events occur at a known average rate (λ) and are independent of the time since the last event. The Poisson distribution is only defined for integer values of k.

Pr X = k( ) = λx

k!e−λ (2)

3

As the number of trials, n, approaches infinity while np remains fixed (n is large, p is small), the Binomial distribution approaches the Poisson distribution with λ = np. Figure 1 shows some examples of Poisson distributions for cases where λ is small. As λ gets larger, the Poisson distribution approaches the Gaussian, or Normal distribution.

Figure 1: Poisson distributions for l = 1, 4, and 10.

The Central Limit Theorem is one of the most profound and important theorems in

mathematics. The Theorem states: Given certain conditions, the arithmetic mean of a sufficiently large number of iterates of an independent random variable, each with a well-defined expectation value and well-defined variance, will be approximately normally distributed regardless of the underlying distribution (wording from Wikipedia). This is important because each of our measurement cycles constitutes an “iterate of an independent random variable”, each of which samples the parent population in a random way. The distribution of the means of these iterates forms a normal distribution, the mean of which approximates the value of the parent population. Mathematically, this Normal distribution can be described as: The average of n mutually independent, identically distributed random variables (with finite measured values of the mean µ0 and variance σ0

2) approaches the distribution:

f x,µ,σ( ) = 12πσ 2

e− x−µ( )2

2σ 2 (3)

with µ = µ0 and σ2 = σ02/n

Figure 2 shows some examples of Normal or Gaussian distributions. This distribution is

the limiting form of the Poisson distribution as the event rate λ becomes large. It is the limiting form of the Binomial distribution as the number of events n becomes large. It is the convergent distribution of the Central Limit Theorem. And it is the most likely distribution for our experiments.

4

Figure 2: Examples of Normal or Gaussian distributions.

Properties of Probability Distributions

The shape of a probability distribution can be described by four statistical moments. The first moment is the Expectation Value (µ = E[f]), or mean. The second moment is the Variance, given by σ2 = E[(f-µ)2]. The variance describes the width of the distribution. The third moment is Skewness, given by ϒ1 = E[((f-µ)/σ3]. Skewness describes the degree of asymmetry of the distribution. The fourth moment is Kurtosis, given by ϒ2 = E[((f-µ)/σ4]. Kurtosis describes the “pointiness” of the distribution. A Gaussian distribution has zero skewness and zero excess kurtosis. It is described entirely by the mean (µ) and the variance (σ2).

Figure 3 shows graphically the relationship of three parameters used to describe a Normal or Gaussian distribution. The mean, µ, is the peak of the distribution. The width of the curve is determined by the value of σ , such that for x = µ + σ , the height of the curve is reduced to e-1/2 of its value at the peak (Figure 1). The full-width at half maximium (Γ), also called the half-width, is given by: Γ = 2.354σ (5)

5

Figure 1: Gaussian probability distribution illustrating the relationship of µ, σ, and Γ to the probability envelope. The probability that the measurement falls within one standard deviation (1σ) of the mean is ~68%. A measurement will fall with two standard deviations (2σ) of the mean ~95% of the time.

The standard form of the Gaussian equation is obtained by defining the dimensionless

variable z = (x-µ)/σ . This frees the distribution from any particular value of µ to give:

€

PG z( )dz =12πexp − z

2

2$

% &

'

( ) dz (6)

This equation permits a single computer routine or table of values to give the Gaussian probability function PG(x;υ ,σ) for all values of the parameters µ and σ by changing the variable and scaling the function by 1/σ to preserve the normalization. Parameter Estimation

Ion probe data should be considered to be measurements that sample a large parent population. We wish to use our data to estimate the characteristics of the parent population, and to estimate how well we know those characteristics (the uncertainties). We will assume we are sampling from a Gaussian parent population with mean µ0 and standard deviation σ0. We do not know the value of µ0 and we wish to estimate it from our data. Let us define our estimate of the population mean, µ0, to be m. We hypothesis that our data follow a normal distribution with parameters µ and σ (equation 3). If we perform a set of n measurements, all with the same standard deviation, how do we estimate the most accurate µ?

We will use the Method of Maximum Likelihood. The probability of observing all n measurements is the joint proability of x1, x2, . . ., xn. This is simply the product of all of Pi’s (equation 3):

Pall = Pi =i=1

n

∏ 12πσ 2

"

#$

%

&'

n

exp −12

xi −µ( )2

σ 2i=1

n

∑"

#$$

%

&'' (7)

We seek the m that maximizes this probability, assuming constant σ. The method of maximum likelihood says that this quantity is maximized at µ = µ0. To maximize Pall, we just maximize the argument of the exponential. To do this, we set the derivative of this argument, with respect to µ, to zero.

ddµ

−12

xi −µσ

"

#$

%

&'2

i=1

n

∑"

#$$

%

&''= 0 (8a)

ddµ

−12

xi −µσ

"

#$

%

&'2

i=1

n

∑"

#$$

%

&''= −

12

ddui=1

n

∑ xi −µσ

"

#$

%

&'2

=xi −µσ

"

#$

%

&'

i=1

n

∑ = 0 (8b)

6

since we assumed s is constant:

µ =1n

xii=1

n

∑ = x (9)

The most likely value of the population mean is just the average of the data!

What is the uncertainty on our estimation of µ? The variance of a normal distribution is σ2. How can we calculate the variance? Consider x, a function of u, v, . . . [x = f(u,v, . . .)]. The variance of x can be expressed as a function of the variance of u, v, . . . and partial derivatives by expanding f in a Taylor series:

σ x2 ≈σ u

2 ∂x∂u#

$%

&

'(2

+σ v2 ∂x∂v#

$%

&

'(2

+...+ 2σ uv2 ∂x∂u#

$%

&

'(∂x∂v#

$%

&

'(+... (10a)

so

σ µ2 = σ xi

2 ∂µ∂xi

"

#$

%

&'

2"

#$$

%

&''

i=1

n

∑ (10b)

∂µ∂xi

=∂∂xi

1n

xii=1

n

∑#

$%

&

'(=1n

(11)

With all σxi = σ, this reduces to:

σ µ2 = σ 2 1

n!

"#$

%&2!

"##

$

%&&=

σ 2

ni=1

n

∑ (12)

This is the uncertainty on our estimation of the population mean, the parameter we computed from our data. But what is σ? We don’t know this a priori. We can construct an estimator of the population standard deviation from our sample: the sample standard deviation, s:

σ ≈ s = 1n−1

xi − x( )i=1

n

∑2

(13)

Substituting into equation 12 and taking the square root gives the uncertainty in our estimation of µ, the mean of the population:

σ µ ≅sn

(14)

Our estimate of the variance of the population is given the variance of our data:

s2 = 1n−1

xi − x( )i=1

n

∑2

(15)

7

The uncertainty in our estimate of the variance (given here without derivation) is:

σs2=σ 4 2

n−1+κn

"

#$

%

&' (16)

where κ is the kurtosis, which is zero for a Gaussian distribution. In summary, for a normal distribution consisting of measurements with a constant

standard deviation, we have calculated an estimator of the mean of the population (Equation 9) and for the uncertainty in the estimation of the mean (Equation 14). We have calculated the standard deviation of our data and from that estimated the variance of the population (Equation 15). One can also estimate the uncertainty in our estimate of the variance (Equation 16). Now I will describe how to use these tools to understand ion probe data.

Figure 4: Example of a data set and the distribution inferred from it, giving

estimates of the mean and standard deviation of the parent population. Propagation of errors

So far, we have discussed the probability distribution produced by several measurements of a single quantity. However, we often must deal with several measured quantities, such as a measured count rate, a background correction, and an instrumental fractionation correction, each with its own error. To calculate an isotope ratio, we must combine the uncertainties in the measured, background-corrected count rates for numerator and denominator into an error for the ratio. These various errors can be combined using some standard formulae. Derivations of these formulae can be found in Bevington and Robinson or another statistics book. They are based on the error propagation equation, which, for the situation where x = f(α ,β ,. . .) gives the variance, σ2 for x as:

σ x2 ≈σα

2 dxdα"

#$

%

&'2

+σβ2 dxdβ"

#$

%

&'

2

+...+ 2σαβ2 dx

dα"

#$

%

&'dxdβ"

#$

%

&'+... (17)

If the errors on α and β are uncorrelated, the cross terms in this equation vanish in the limit of a large random selection of observations and the error equation reduces to:

8

σ x2 ≈σα

2 dxdα"

#$

%

&'2

+σβ2 dxdβ"

#$

%

&'

2

+... (18)

with similar terms for additional variables. We will return to the case where errors are correlated later in this discussion. Below I give the specific formulas for propagating uncorrelated errors in several standard cases. Sums and differences

If the result, x, is related to a measured value α ± σα by the relationship

€

x =α + b (19)

where b is a constant, then partial derivative ∂x/∂α = 1 and the uncertainty in x is just:

€

σ x =σα (20)

The relative uncertainty, also known as the fractional error, is given by

€

σ x

x=σα

x=

σα

α + b (21)

If x is the weighted sum of α ± σα and β ± σβ such that

€

x = bα + cβ (22)

then the partial derivatives are constants

€

∂x∂α

$

% &

'

( ) = b

€

∂x∂β

$

% &

'

( ) = c (23)

The variance and standard deviation are:

€

σ x2 = b2σα

2 + c 2σβ2 + 2bcσαβ

2

€

σ = b2σα2 + c 2σβ

2 + 2bcσαβ2 (24)

For uncorrelated errors, cross terms in equation (17) vanish and

€

σ = b2σα2 + c 2σβ

2 (25) Multiplication and division

Equation (17) also applies for multiplication and division. For multiplication, where

€

x = bαβ (26)

9

the partial derivatives are

€

∂x∂α

$

% &

'

( ) = bβ

€

∂x∂β

$

% &

'

( ) = bα (27)

and the variance of x is

€

σ x2 = bβσα( )2 + bασβ( )

2+ 2b2αβσαβ

2 (28)

A more useful expression is given by the equation for relative variance, which can be derived from equation (18) by dividing both sides by x2

€

σ x2

x 2=σα2

α 2 +σβ2

β 2+ 2

σαβ2

αβ (29)

The standard deviation for x:

€

σ x = x σα2

α 2 +σβ2

β 2+ 2

σαβ2

αβ (30)

As with equation 25, the cross term becomes zero in the case where the variances in α and β are uncorrelated (covariance = 0). The standard deviation becomes:

σ x = xσα2

α 2 +σβ2

β 2 (31)

In our application, we will be calculating isotope ratios and elemental abundance ratios.

Uncertainties in the numerator and denominator must be combined to give the uncertainty in the ratio. Thus, for division, if r is the weighted ratio of α and β:

€

r =bαβ

(32)

The partial derivatives are

€

∂r∂α

$

% &

'

( ) =

bβ

€

∂r∂β

$

% &

'

( ) = −

bαβ 2

(33)

and the relative variance for r and the standard deviation are given by

€

σ r2

r2=σα2

α 2 +σβ2

β 2− 2

σαβ2

αβ

€

σ r = r σα2

α 2 +σβ2

β 2− 2

σαβ2

αβ (34)

10

where the last term becomes zero when the errors are uncorrelated.

In summary, when doing addition or subtraction, the errors are determined by calculating the sum of the squares of the absolute errors and then taking the square root. When you are doing multiplication or division, the errors are determined by calculating the sum of the squares of the relative errors, taking the square root, and then multiplying the result by the value of the product or quotient.

Application to ion probe measurements

How does the above formalism apply to counted ions, the output of the ion microprobe? When the mean number of counted ions is N during a given time t, it can be said to be the result of repeated N*M tests for which the probability of counting one ion is 1/M. In an ion counting experiment, the uncertainty of a measurement is equal to the square root of the number of counts measured (

€

σ = N ). Suppose that we want to calculate the uncertainty in a ratio R defined as:

€

R =S1S2

(35)

where S1 and S2 are count rates expressed in counts per second and are derived from two numbers of counts, N1 and N2, collected over time intervals T1 and T2 such that S1 = N1/T1 and S2 = N2/T2 (36) The counting-statistical uncertainty in the ratio can be expressed as the ratio times the quadratic sum of the relative uncertainties in the numerator and denominator as determined by counting statistics (equation 23):

€

σR = R ×N1( )

2

N1( )2+

N2( )2

N2( )2= R ×

T1S1( )2

T1S1( )2+

T2S2( )2

T2S2( )2 (37)

The following examples illustrate the use of Gaussian statistics to determine uncertainties

in data from the ion microprobe.

Example 1: We measure carbon isotopes, 12C and 13C. The measured counts and uncertainties are: 12C = 125390 ± 354 and 13C = 1420 ± 38. The ratio, its variance, and its 1σ standard deviation are:

€

13C12C

=1420125390

= 0.011325

σ r2 = r2 σα

2

α 2 +σβ2

β 2!

"##

$

%&&= 0.11325

2 382

14202+

3542

1253902!

"#

$

%&= 0.00000929

11

€

σ = 0.011325 × 382

14202+

3542

1253902$

% &

'

( )

*

+ , ,

-

. / /

= 0.000305

Example 2: We measure carbon isotopes, 12C and 13C. The measured count rates are: 12C = 125390 cps and 13C = 1420 cps and the counting times are T12 = 10 and T13 = 100 seconds. The ratio is calculated using count rates, the same as in Example 1:

€

13C12C

=1420125390

= 0.011325

But for the uncertainty, we must consider the total number of counts measured for each isotope, not simply the count rate. Total counts for 12C = 125390 × 10 = 1253900 and for 13C = 1420 × 100 = 142000. The variance and 1σ standard deviation are:

σ r2 = r2 σα

2

α 2 +σβ2

β 2!

"##

$

%&&= 0.11325

2 3772

1420002+11202

12539002!

"#

$

%& = 0.000000101

σ = 0.011325× 3772

1420002+11202

12539002"

#$

%

&'

(

)**

+

,--= 0.0000317

Because the 13C counts provide the main contribution to the error, increasing the total number of 13C counts by a factor of 100 relative to Example 1 results in a factor of 10 improvement in the uncertainty for the ratio. Example 3: Suppose we are using an electron multiplier that has a background of 0.5 counts per second. We determined this background by measuring it for 10 seconds, where we got 5 total counts. The background is then (5/10) ± (sqrt(5)/10) = 0.5 ± 0.22 counts per second. We measure a signal for four seconds and find that the count rate is 23.0 counts per second. The uncertainty in our measurement is sqrt(23*4)/4, so the measurement is 23.0 ± 2.4 counts per second. What is the count rate and uncertainty after correcting for the background?

Count rate = 23 – 0.5 = 22.50

€

σ = 2.42 + 0.222( ) = 2.41

Note that the uncertainty in the background correction, which has an absolute magnitude less than 10% of the uncertainty in the measurement, plays almost no role in the final uncertainty. Example 4: Suppose that we have a measured count rate of 1.2 counts per second, measured for 10 seconds (1.2 ± 0.35), and a background of 0.5 counts per second measured for 5 seconds (0.50±0.31).

12

Count rate = 1.20 – 0.50 = 0.70

€

σ = 0.352 + 0.312( ) = 0.47 In this case, the two uncertainties contribute roughly equally to the total uncertainty.

Problems with low count rates Low count rates can cause problems in isotope ratio measurements if counting times are not sufficiently long. Consider a case where the average count rate is 0.3 counts per second. If a counting time of 1 second per cycle is used, only one out of ~3 cycles will record a count, on average. With enough cycles, one can calculate the average count rate reasonably accurately. But suppose I want to calculate ratios for every cycle using these data. In the 0.3 cps mass is in the numerator, the ratio will be zero two-thirds of the time. If it is in the denominator, the ratio becomes undefined two-thirds of the time. Neither of these cases is handled well by the techniques described above. The undefined ratios result in a completely unreasonable ratio that is easy to spot. However, the zeros are equally bad and more insidious when calculating a ratio, because one is typically making ratios with one mass that has an asymmetrical distribution about the mean (Fig. 5) and a another that has a symmetrical distribution (Fig. 6). The result is that the ratio appears reasonable but is too high. Thus, it is necessary to choose counting times such that every cycle has at least one count, so all ratios are defined and non-zero. Alternatively, one can “recompute” the counting interval by combining cycles into blocks such that each block has at least one count. We will return to the issue of low counts later in the course.

Figure 5: Histogram of counts in a cosmic ray detector. The experimenter recorded the number of counts in their detector for a series of 100 2-second measurements. The Poisson distribution, shown as a smooth curve, is an estimate of the parent distribution based on the measured mean, 1.69 counts per 2-second interval. Note that the distribution is not symmetrical because one cannot measure negative counts.

13

Figure 6: The same experiment was performed as that to generate Fig. 5 except that 60 measurements of 15 seconds each were made. The Poisson distribution is again shown as a smooth curve. The mean in this case was 11.48 counts per 15-second interval and the distribution is symmetrical.

In the two cases shown in the Figures above, the mean count rate and standard deviation are the same. Poisson statistics correctly handles the asymmetrical distribution of counts per measurement interval and the final results of the two experiments are statistically equivalent (µ = 0.85, σ = 0.65 for the first experiment; µ = 0.77, σ = 0.23 for the second). The problem only becomes evident if ratios are calculated using a distribution like in Figure 5.

Combining several measurements into a single result

An ion microprobe is best described as an isotope-ratio mass spectrometer. The data are collected as a series of ratios, with the most abundant isotope typically serving as the denominator. This method of collecting data permits us to recognize and correct for changes in signal strength with time and to identify noise spikes or other problems with the data. However, the final result we are after is a single value for each isotope ratio and its uncertainty.

Suppose we make 100 measurements of an isotope ratio and calculate the statistical uncertainty for each ratio as described in the previous section. How can we combine these data into a single measurement? The distribution of these measurements will be Gaussian (assuming that there are no non-statistical uncertainties), characterized by a mean µ and standard deviation σ µ. If the 100 ratios, xi, are all based on a similar number of counts, and thus have the same uncertainty, σ i, then the mean is given by

€

µ = x ≡ 1N

xi∑ (38)

14

and the uncertainty is given by

€

σ µ = σ i2 ∂µ∂xi

$

% &

'

( )

2*

+ , ,

-

. / /

∑ (39)

Because the uncertainties on the individual ratios are all equal in this case, the partial derivatives in equation (39) are

€

∂µ∂xi

=∂∂xi

1N

xi∑$

% &

'

( ) =

1N

(40)

Combining equations (39) and (40), we find

€

σ µ = σ i2 1N#

$ %

&

' ( 2)

* +

,

- . ∑ =

σ i

N (41)

When calculated this way, σ µ is referred to as the standard deviation of the mean or the standard error. If the scatter of the individual ratio measurements is due entirely to statistical effects, the uncertainty calculated from equation (41) will be the same as that calculated from the total counts from all of the ratios treated as a single measurement. We use this equivalency as a test of how good our data is. The output from both the instrument software and our off-line data reduction program gives the ratio of the uncertainty calculated from the spread of the individual measurements for each ratio divided by that expected uncertainty from the total number of counts collected. For good ion probe measurements, this ratio is very close to unity.

Often we find it necessary to combine data from several measurements that were collected with different counting rates or with different numbers of ratios. The uncertainties of the measurements are thus different. I will not attempt to derive the equations here (you can see the derivation in Bevington and Robinson). I will just present and discuss them. The equation for the mean of several values with different uncertainties is:

€

µ =xi σ i

2( )∑1 σ i

2( )∑ (42)

In this equation, each data point xi in the sum is weighted inversely by its own variance. So a measurement with a large error will contribute less to the mean than a measurement with a small error. The uncertainty in the mean can be determined from the by evaluating

€

∂µ ∂xi from equation (42) for the mean µ :

€

∂µ∂xi

=∂∂xi

xi σ i2( )∑

1 σ i2( )∑

=1 σ i

2

1 σ i2∑

(43)

Substituting this into equation (39) gives:

15

€

σ µ =1 σ i

2

1 σ i2( )∑[ ]

2∑ =11 σ i

2( )∑ (44)

The error-weighted mean and its associated standard deviation of the mean (standard error) are used extensively in ion-probe mass spectrometry and we have and Excel macro programmed to make them easy to calculate. When to use the standard deviation of the mean and when to use the standard deviation

The last section showed that the standard deviation of the mean or standard error of a set of measurements is given by the standard deviation of the measurements divided by the square-root of the number of measurements. This is true if the errors of the measurements are a truly random distribution from a single population. Increasing the number of measurements of the isotope ratios in a sample will rapidly improve the precision to which we know the mean of the measurements. However, the standard error is not always the correct statistic to use when analyzing ion probe data. Remember that the standard error asks the statistical question: How well do I know the mean of the group of measurements that I have made and how closely is it likely to represent the true value for the system being measured? In contrast, the standard deviation asks the question: What is the statistical scatter of measurements around the mean value and how close is my next measurement likely to be to the mean value? This uncertainty does not decrease with an increasing number of measurements.

As an example, consider a set of ten measurements xi of an isotope ratio. Our estimate of the mean of these measurements has a value µx, and the uncertainty associated with that mean describing how close it is to the true value of the system is

€

σ µ 10 . However, the likelihood that the next measurement will fall within one sigma of the mean, µx, is given by the standard deviation, σ µ.

It is not sufficient to keep measuring a sample until the standard deviation of the mean becomes arbitrarily small. At some point something other than statistics will control the error on the measurement. Consider the case of a typical ion probe measurement. In order to correct for instrumental effects, we measure standards. To determine the magnitude of the instrumental effect precisely, we measure the standards many times. If the variation of the standards is truly random, then we can determine the value of, for example, the mass-dependent instrumental fractionation, f, very precisely. The standard error of our set of standard measurements (

€

σ f N f ) tells us how well we know the instrumental fractionation of the instrument, and the standard deviation of the measurements (σ f) tells us the likelihood that the next measurement will be within one sigma of the measured instrumental fractionation.

We then measure a sample of unknown composition many times and obtain and value, r, for an isotope ratio and an uncertainty (

€

σ r Nr ). The more we measure, the better determined our mean value, r, becomes. This ratio must be corrected for the mass-dependent instrumental fractionation and the uncertainties of our measurement and of the correction must be combined appropriately. The corrected value and its uncertainty are:

16

€

R = r − f

€

σR =σ r

Nr

#

$ % %

&

' ( (

2

+σ f

N f

#

$ % %

&

' ( (

2

(45)

But suppose that we cannot measure the unknown sample many times and are dependent

on a single measurement to determine the true value. What is the uncertainty in our determination? The uncertainty in the measured ratio, σ r, must be combined with the uncertainty for the mass fraction, as in the previous example. However, in this case, because the sample was only measured once, the uncertainty of the mass fractionation associated with our measurement is the uncertainty associated with an individual measurement of the standard. You don’t know if this measurement is close to the mean or far from the mean and you cannot evaluate it independently because the true ratio is also unknown. So the measurement error must be combined with the measurement error for a single standard measurement, the standard deviation, σ f. We have to ask the statistical question: How likely is the fractionation for this measurement to be within one sigma of the fractionation value, f, that we have determined for the standard? In this case, it doesn’t matter how well we may have determined f through independent measurements (except that any systematic error is reduced). The statistical error is the quadratic sum of the measurement error and the standard deviation of the measurements on the standard

€

σR = σ r2 +σ f

2 (46) If we must apply the uncertainty associated with a single measurement of the

instrumental mass fractionation as the uncertainty on the mass fractionation correction, why collect a large number of measurements of the standards? The standard deviation does not decrease with the number of measurements, but the correction does become better determined. We are interested in more than just the precision of our result. We are also interested in the accuracy. If we were to use a single measurement of the standard, it might be an outlier in a distribution obtained by a large number of measurements. Using this value to correct for instrumental fractionation would add a systematic error to all of the measurements equivalent to the differences between the true value of the instrumental fractionation and the value that we inferred from our single measurement. The better we know the value of the instrumental fractionation, the smaller are any potential systematic errors that may be introduced by the correction.

In summary, the more measurements of a sample or standard you make, the better determined the mean value is. The standard error will improve with additional measurements by a factor of

€

N . The statistical uncertainty associated with a correction of a single measurement for instrumental mass fractionation, detector background, detector sensitivity, deadtime, etc is given by the standard deviation of the measurements that were made to determine the value of the correction. The level of the potential systematic error associated with the correction is given by the standard error calculated from the same set of measurements.

The χ2 statistic: How well do the data follow Gaussian statistics?

In the discussions above, we have assumed that the data can be fit by a Gaussian (or Poisson) distribution and thus can be described by the statistics based on that distribution. However, in the real world, our data often have variations caused by something other than

17

random statistical variation. Sometimes it is clear that this is the case and sometimes it is not. It would be nice to have a statistic that could help us to recognize non-Gaussian distributions. That statistic is called chi-squared (χ2).

Assume that we make N measurements xi of the quantity x. There are j possible values for xi ranging from 1 to n. We define the frequency of observations for each possible value xj as h(xj). If the probability for observing the value xj in any random measurements is denoted by P(xj), then the expected number of such observations is y(xj) = NP(xj), where N is the total number of measurements. Figures 7 and 8 show the same six-bin histogram, drawn from a Gaussian parent distribution with a mean µ = 5.0 and standard deviation σ = 1, corresponding to 100 total measurements. The parent distribution is illustrated by the solid Gaussian curve on each histogram.

For each measured value xj, there is a standard deviation σ j(h) associated with the uncertainty in the observed frequency h(xj). This is not the same as the uncertainties associated with the spread of the individual measurements xi about their mean µ , but rather describes the spread of the measurement of each of the frequencies h(xj) about its mean µ j. If we were to repeat the experiment many times to determine the distribution of frequency measurements at each value of xj, we should find each parent distribution to be Poisson with mean µj = y(xi) and variance

€

σ j2(y) = y(x j ) . Thus, for each value of xj, there is a distribution curve, Pj(yk), that

describes the probability of obtaining the value of the frequency hk(xj) in the kth trial for each value of j that is characterized by σ j(h). These distributions are illustrated in Figures 7 and 8 as dotted Poisson curves at each value of xj. In Figure 7, the Poisson curves are centered at the observed frequencies h(xj) with standard deviations

€

σ j h( ) = h x j( ) . In principle, we should

center the Poisson curves at the frequencies µj = y(xj) with standard deviation

€

σ j h( ) = µ j of the parent population, as in Figure 8. However, in an actual experiment, we generally would not know these parameters.

Figure 7: Histogram drawn form a Gaussian distribution mean µ = 5.0 and standard deviation σ = 1, corresponding to 100 total measurements. The parent

18

distribution y(xj) = NP(xj) is illustrated by the large Gaussian curve. The smaller dotted curves represent the Poisson distribution of events in each bin, based on the sample data.

Figure 8: The same histogram as shown in Figure 7 with dotted curves representing the Poisson distribution of events in each bin, based on the parent distribution.

With the preceding definitions for n, N, xj, H(xj), P(xj), and σ j(h), the definition for χ2 is

€

χ 2 ≡h x j( ) − NP x j( )[ ]

2

σ j h( )2j=1

n

∑ (47)

In most experiments, we do not know the values of σ j(h) because we make only one set of measurements f(xj). Fortunately, these uncertainties can be estimated from the data directly without measuring them explicitly. If we consider the data in Figure 8, we observe that for each value of xj, we have extracted a proportionate random sample of the parent population for that value. The fluctuations in the observed frequencies h(xj) come from the statistical probabilities of making random selections of finite numbers of items and are distributed according to the Poisson distribution with y(xj) as mean.

For the Poisson distribution, the variance σj(h)2 is equal to the mean y(xj) of the distribution, and thus we can estimate σ j(h) from the data to be

€

σ j h( ) = NP x j( ) = h x j( ) . Equation 47 becomes:

19

€

χ 2 ≡h x j( ) − NP x j( )[ ]

2

NP x j( )j=1

n

∑ ≈h x j( ) − NP x j( )[ ]

2

h x j( )j=1

n

∑ (48)

As defined in equations (47) and (48), χ2 is a statistic that characterizes how much the

observed frequency distributions differ from the expected frequencies based on Poisson statistics. If the frequencies were to agree exactly with the predicted frequencies, then we should find χ2 = 0. This is not a likely outcome. The numerator of equation 47 is a measure of the spread of the observations; the denominator is a measure of the expected spread. We might imagine that for good agreement, the average spread of the data would correspond to the expected spread, and thus we should get a contribution of about one from each frequency, or χ2 ≈ n for the entire distribution. This is almost correct. The true expectation for the value of χ2 is

€

χ 2 = ν = n − nc (49) where ν is the number of degrees of freedom and is equal to the number n of sample frequencies minus the number nc of constraints or parameters that have been calculated from the data to describe the probability function NP(xj). For our example, even if NP(xj) is chosen completely independently of the distribution h(xj), there is still the normalizing factor N corresponding to the total number of events in the distribution, so that the expectation value of χ2 must at best be

€

χ 2 = n −1 (50)

In order to estimate the probability that our calculated values of χ2 are consistent with our expected distribution of the data, we must know how χ2 is distributed. If our value of χ2 corresponds to a reasonably high probability, then we can have confidence in our assumed distribution.

It is convenient to define the reduced chi-squared as

€

χν2 ≡

χ 2

ν (40)

which should give a value of 1. Values of

€

χν2 much larger than 1 result from large deviations

from the assumed distribution and may indicate poor measurements, incorrect assignment of uncertainties, or an incorrect choice of probability function. Very small

€

χν2 are also

unacceptable and may imply misunderstanding of the experiment. To quantitatively evaluate the probability of observing a value of

€

χν2 equal to or greater than the value we calculated, we must

turn to a table that gives values of reduced chi-squared corresponding to the probability Px(χ2;ν) exceeding χ2 versus the number of degrees of freedom. Details of how to use such a table are given in Bevington and Robinson. In practice, we typically do not go to the trouble to quantitatively evaluate the goodness of fit described by chi-square. Instead, we use it as a general indicator of whether or not our data can be considered to obey a Gaussian distribution or whether other sources of scatter are affecting the data. Reduced chi-squared is calculated by several of our routines and you should look at the result. If reduced chi-squared is significantly

20

different from one, you should look for the problems in your data that cause such a result and think hard about how to reduce, interpret, and report your data.