bayesian statistical methods for parton analyses
DESCRIPTION
Bayesian statistical methods for parton analyses. Glen Cowan Royal Holloway, University of London [email protected] www.pp.rhul.ac.uk/~cowan DIS 2006 Tsukuba 22 April, 2006. In collaboration with Clare Quarman. Glen Cowan. DIS2006, Tsukuba, 22 April, 2006. Outline. - PowerPoint PPT PresentationTRANSCRIPT
1
Bayesian statistical methods for parton analyses
Glen Cowan
Glen CowanRoyal Holloway, University of London
[email protected]/~cowan
DIS 2006Tsukuba22 April, 2006
DIS2006, Tsukuba, 22 April, 2006
In collaboration with Clare Quarman
2
Outline
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
I. Data analysis difficulties for prediction of LHC observables
Some problems with frequentist statistical methods
II. Bayesian statistics
Quick review of basic formalism and tools
Application to:incompatible data sets,model (theoretical) uncertainties.
III. Prospects for LHC predictions
3
Some uncertainties in predicted cross sections
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
I. PDFs based on fits to data with:
imperfectly understood systematics,not all data compatible.
II. Perturbative prediction only to limited order
PDF evolution & cross sections to NLO, NNLO...
III. Modelling of nonperturbative physics
parametrization of PDF at low Q2, details of flavour composition, ...
4
LHC game plan
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Understanding uncertainties in predicted cross sections is arecognized Crucial Issue for LHC analyses, e.g.
extra dimensions, parton substructure, sin2 W
For LHC observables we have
uncertaintiesin PDFs
uncertainties inparton cross sections
5
PDF fit (symbolic)
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Given measurements:
and (usually) covariances:
Predicted value:
control variable PDF parameters, s, etc. bias
Often take:
Minimize
Equivalent to maximizing L() » e2/2, i.e., least squares same as maximum likelihood using a Gaussian likelihood function.
expectation value
6
Uncertainties from PDF fits
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
If we have incompatible data or an incorrect model, then minimized 2 will be high, but this does not automatically result in larger estimates of the PDF parameter errors.
Frequentist statistics provides a rule to obtain standard deviation of estimators (1 statistical errors):
2 = 2min + 1
but in PDF fits this results in unrealistically small uncertainties.
Try e.g. 2 = 2min + 50, 75, 100?
The problem lies in the application of a rule for statistical errors to a situation dominated by systematics & model uncertainties.
→ Try Bayesian statistical approach
7Glen Cowan
The Bayesian approachIn Bayesian statistics we can associate a probability witha hypothesis, e.g., a parameter value .
Interpret probability of as ‘degree of belief’ (subjective).
Need to start with ‘prior pdf’ (), this reflects degree of belief about before doing the experiment.
Our experiment has data x, → likelihood function L(x|).
Bayes’ theorem tells how our beliefs should be updated inlight of the data x:
Posterior pdf p(|x) contains all our knowledge about .
DIS2006, Tsukuba, 22 April, 2006
Rev. Thomas Bayes
8
A possible Bayesian analysis
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
and use Bayes’ theorem:
To get desired probability for , integrate (marginalize) over b:
→ Posterior is Gaussian with mode same as least squares estimator, same as from 2 = 2
min + 1. (Back where we started!)
Take
Joint probabilityfor all parameters
9
Marginalizing with Markov Chain Monte Carlo
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
In a Bayesian analysis we usually need to integrate over some(or all) of the parameters, e.g.,
Probability densityfor prediction of observable ()
Integrals often high dimension, usually cannot be done in closed form or with acceptance-rejection Monte Carlo.
Markov Chain Monte Carlo (MCMC) has revolutionized Bayesiancomputation. (Google words: Metropolis-Hastings, MCMC)
Produces a correlated sequence of points in the sampled space.Correlations here not fatal, but stat. error larger than naive √n.
10
Low Q2 PDF: Unknown coefficients of higher order, higher twist terms, ...Experimental biases, ...
Systematic uncertainty and nuisance parameters
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
In general we can describe the data better by including more parameters in the model (nuisance parameters), e.g.,
But, more parameters → correlations → bigger errors.
Bayesian approach: include more parameters along with prior probabilities that reflect how widely they can vary.
Difficult (impossible) to agree on priors but remember ‘if-then’nature of result. Usefulness to community comes from sensitivity analysis:
Vary prior, see what effect this has on posterior.
11
The Full Bayesian Machine
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
A full Bayesian PDF analysis could involve:
the usual two dozen PDF parameters,a bias parameter for each systematic,more parameters to quantify model uncertainties,...
as well as a meaningful assignment of priors
consultation with experimenters/theorists
and finally an integration over the entire parameter space toextract the posterior probability for a parameter of interest, e.g., a predicted cross section:
ongoing effort, primary difficulties with MCMC
12
The error on the error
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Some systematic errors are well determined
Error from finite Monte Carlo sample
Some are less obvious
Do analysis in n ‘equally valid’ ways andextract systematic error from ‘spread’ in results.
Some are educated guesses
Guess possible size of missing terms in perturbation series;
vary renormalization scale
Can we incorporate the ‘error on the error’?
(cf. G. D’Agostini 1999; Dose & von der Linden 1999)
13
Motivating a non-Gaussian prior b(b)
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Suppose now the experiment is characterized by
where si is an (unreported) factor by which the systematic error is over/under-estimated.
Assume correct error for a Gaussian b(b) would be siisys, so
Width of s(si) reflects‘error on the error’.
14
Error-on-error function s(s)
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
A simple unimodal probability density for 0 < s < 1 with adjustable mean and variance is the Gamma distribution:
Want e.g. expectation value of 1 and adjustable standard deviation s , i.e.,
mean = b/avariance = b/a2
In fact if we took s (s) » inverse Gamma, we could integrate b(b)in closed form (cf. D’Agostini, Dose, von Linden). But Gamma seems more natural & numerical treatment not too painful.
s(s
)
s
15
Prior for bias b(b) now has longer tails
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Gaussian (s = 0) P(|b| > 4sys) = 6.3 £ 10-5
s = 0.5 P(|b| > 4sys) = 0.65%
b(b
)
b
16
A simple test
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Suppose fit effectively averages four measurements.
Take sys = stat = 0.1, uncorrelated.
Case #1: data appear compatible Posterior p(|y):
Usually summarize posterior p(|y) with mode and standard deviation:
experiment
mea
sure
me
nt
p(
|y)
17
Simple test with inconsistent data
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Case #2: there is an outlier
→ Bayesian fit less sensitive to outlier.
→ Error now connected to goodness-of-fit.
Posterior p(|y):
experiment
mea
sure
me
nt
p(|
y)
18
Goodness-of-fit vs. size of error
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
In LS fit, value of minimized 2 does not affect sizeof error on fitted parameter.
In Bayesian analysis with non-Gaussian prior for systematics,a high 2 corresponds to a larger error (and vice versa).
2000 repetitions ofexperiment, s = 0.5,here no actual bias.
post
erio
r
2
from least squares
19
Is this workable for PDF fits?
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Straightforward to generalize to include correlations
Prior on correlation coefficients ~ ()(Myth: = 1 is “conservative”)
Can separate out different systematic for same measurement
Some will have small s, others larger.
Remember the “if-then” nature of a Bayesian result:
We can (should) vary priors and see what effect this has on the conclusions.
20
Uncertainty from parametrization of PDFs
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Try e.g. (MRST)
(CTEQ)or
The form should be flexible enough to describe the data;frequentist analysis has to decide how many parameters are justified.
In a Bayesian analysis we can insert as many parameters as wewant, but constrain them with priors.
Suppose e.g. based on a theoretical bias for things not too bumpy,that a certain parametrization ‘should hold to 2%’.
How to translate this into a set of prior probabilites?
21
Residual function
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Try e.g. ‘residual function’
where r(x) is something very flexible, e.g., superposition of
Bernstein polynomials, coefficients i:
mathworld.wolfram.com
Assign priors for the i centred around 0, width chosento reflect the uncertainty in xf(x) (e.g. a couple of percent).
→ Ongoing effort.
22
Wrapping up
Glen Cowan RHUL HEP seminar, 22 March, 2006
A discovery at the LHC may depend crucially on assessing theuncertainty in a predicted cross section.
Systematic uncertainties difficult to treat in frequentist statistics,often wind in with ad hoc recipes.
Bayesian approach tries to encapsulate the uncertainties in priorprobabilities for an enlarged set of model parameters.
Bayes’ theorem says where these parameters should lie in light of the data.
Marginalize to give probability of parameter of interest(new tool: MCMC).
Very much still an ongoing effort!
23
Extra slides
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
24
Some references
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
S.I. Alekhin, Extraction of parton distributions and from DIS data with a Bayesian treatment of systematic errors, Eur. Phys. J. C 10, 395-403
W. Giele, S. Keller and D. Kosower, Parton Distribution Function Uncertainties, hep-ph/0104052
G. D'Agostini, Sceptical combination of experimental results: General considerations and application to ’/, CERN-EP/99-139, hep-ex/9910036
V. Dose and W. von der Linden, Outlier tolerant parameter estimation, in V. Dose et al. (eds.), Proc. of he XVIII International Workshop on Maximum Entropy and Bayesian Methods, Garching, 1998 (Kluwer Academic Publishers, Dordrecht, 1999; preprint in www.ipp.mpg/de/OP/Datenanalyse/Publications/Papers/dose99a.ps)
25
Marginalizing the posterior probability density
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
Bayes’ theorem gives the joint probability for all the parameters.Crucial difference compared to freqentist analysis is ability tomarginalize over the nuisance parameters, e.g.,
Do e.g. with MC: sample full (, b) space and look at distribution only of those parameters of interest.
In the end we are interested not in probability of butof some observable () (e.g. a cross section), i.e.,
Similarly do with MC: sample (, b) and look at distributionof ().
26Glen Cowan
Digression: marginalization with MCMCBayesian computations involve integrals like
often high dimensionality and impossible in closed form,also impossible with ‘normal’ acceptance-rejection Monte Carlo.
Markov Chain Monte Carlo (MCMC) has revolutionizedBayesian computation.
Google for ‘MCMC’, ‘Metropolis’, ‘Bayesian computation’, ...
MCMC generates correlated sequence of random numbers:cannot use for many applications, e.g., detector MC;effective stat. error greater than √n .
Basic idea: sample multidimensional look, e.g., only at distribution of parameters of interest.
RHUL HEP seminar, 22 March, 2006
27Glen Cowan
MCMC basics: Metropolis-Hastings algorithmGoal: given an n-dimensional pdf
generate a sequence of points
1) Start at some point
2) Generate
Proposal densitye.g. Gaussian centredabout
3) Form Hastings test ratio
4) Generate
5) If
else
move to proposed point
old point repeated
6) Iterate
RHUL HEP seminar, 22 March, 2006
28Glen Cowan
Metropolis-Hastings (continued)This rule produces a correlated sequence of points (note how each new point depends on the previous one).
For our purposes this correlation is not fatal, but statisticalerrors larger than naive
The proposal density can be (almost) anything, but chooseso as to minimize autocorrelation. Often take proposaldensity symmetric:
Test ratio is (Metropolis-Hastings):
I.e. if the proposed step is to a point of higher , take it;
if not, only take the step with probability
If proposed step rejected, hop in place.
RHUL HEP seminar, 22 March, 2006
29Glen Cowan
Metropolis-Hastings caveatsActually one can only prove that the sequence of points followsthe desired pdf in the limit where it runs forever.
There may be a “burn-in” period where the sequence doesnot initially follow
Unfortunately there are few useful theorems to tell us when thesequence has converged.
Look at trace plots, autocorrelation.
Check result with different proposal density.
If you think it’s converged, try it again with 10 times more points.
RHUL HEP seminar, 22 March, 2006
30
Although numerical values of answer here same as in frequentistcase, interpretation is different (sometimes unimportant?)
Glen Cowan
Example: posterior pdf from MCMCSample the posterior pdf from previous example with MCMC:
Summarize pdf of parameter ofinterest with, e.g., mean, median,standard deviation, etc.
RHUL HEP seminar, 22 March, 2006
31
Bayesian approach to model uncertainty
Glen Cowan DIS2006, Tsukuba, 22 April, 2006
x
y (m
easu
rem
ent)
model:
truth:
Model can be made to approximate the truth better by includingmore free parameters.
systematic uncertainty↔
nuisance parameters
In a frequentist analysis, the correlations between the fittedparameters will result in large errors for the parameters of interest.
In Bayesian approach, constrain nuisance parameters with priors.