Byron Roe 1

Some Current Statistical Considerations in Particle Physics

Byron P. Roe

Department of Physics

University of Michigan

Ann Arbor, MI 48109

Byron Roe 2

Outline

• Preliminaries

• Nuisance Variables

• Modern Classification Methods (especially boosting and related methods).

Byron Roe 3

Preliminaries

• Try to determine a parameter given a measurement x

• For each draw line so probability x is within limits is 90%

• The probability of a result falling in region is 90%

• Given an x, then for 90% of experimentsis in that region. This is the Neyman Construction

Byron Roe 4

Frequentist and Bayesian

• This construction is “frequentist”; no probability is assigned to a physical quantity

• Bayesian point of view: probability refers to state of knowledge of parameter and can have a probability

• Formerly a “war” between the two views.• People starting to realize each side has

some merits and uses; war abating

Byron Roe 5

Ambiguities

• At a given, 90% of the time x will fall in region, but do you want 5% on each side or 8% on lower and 2% on upper?

• Useful ordering principle introduced into physics by Feldman and Cousins: Choose the region to have the largest values of R=likelihood thisgiven x / best likelihood of any physical given x

• Always gives a region; goes automatically from limits to regions

Byron Roe 6

But is it new?

• Feldman and Cousins soon realized this was a standard statistical technique described in a text (Kendall and Stuart)

• Physicists have, in the past, often ignored statistical literature to the detriment of both physicists and statisticians

• In recent years, helped by conferences on statistics in physics since 2000, there has been more and more cooperation

Byron Roe 7

Nuisance Parameters

• Experiments may depend on background, efficiency… which are not the targets of the experiment, but are needed to get to the physical parameter

• These are called nuisance parameters

• The expectation values may be well known or have an appreciable uncertainty.

Byron Roe 8

Problem with Feldman Cousins

• Karmen experiment in 1999 reported results. They were checking LSND expt.

• Background was known to be 2.83+/-0.13 events.

• They observed 0 events and set a limit on lambda using FC at 1.1 at 90% CL

• Common sense: if 0 signal, 2.3 is 90% CL• FC ordering is P given data, BUT 90% CL

is overall P, not P given data

Byron Roe 9

Attempts to Improve Estimate

• With a statistician, Michael Woodroofe, I tried different methods

• Suppose try Bayesian method, taking a prior (initial) probability for uniform.

• Obtain credible limit (Bayesian equivalent of CL)• Go back to frequentist view and look at coverage• Quite close to frequentist and with small

modification, very close in almost all regions, but gives 2.5 for Karmen limit close to desired 2.3

Byron Roe 10

Nuisance Variables with Significant Uncertainty

• Can draw a 90% CL region for joint probability of and b (nuisance par.)

• Project ontoaxis and take extreme values for CL

• Safe, but often grossly over-covers

Byron Roe 11

Nuisance Parameters 2

• 1. Integrate over nuisance parameter b using measured probability of b. Introduces Bayesian concept for b. Tends to over-cover and there are claims of under-coverage

• 2. Suppose max. likelihood solution has values , B. Suppose, given a, the maximum likelihood solution for b is b. Consider:

R = likelihood(x|,b)/ likelihood(x|,B) (Essentially FC ratio)

Let G,b = prob,b(R>C) = CL Approximate: G,b approx Gb

Byron Roe 12

Nuisance Parameters 3

• Use this and make a full Neyman construction. Good coverage for a number of examples, OR…

• Assume -2ln R is approximately a2distribution. (It is asymptotically.) This is method of MINOS in MINUIT. Coverage good in some recent cases. Clipping required for nuisance parameters far from expected values.

Byron Roe 13

Data Classification

• Given a set of events to separate into signal and background and some partial knowledge in the form of set of particle identification (PID) variables.

• Make a series of cuts on PID variables—often inefficient.

• Neural net. Invented by John Hopfield as a method the brain might use to learn.

• Newer methods—boosting,…

Byron Roe 14

Neural Nets and Modern Methods

• Use a training sample of events for which you know which are signal and which are background.

• Practice an algorithm on this set, updating it and trying to find best discrimination.

• Need second unbiased set to test result on, the test sample.

• If the test set was used to determine parameters or stopping point of algorithm, need a third set, verification sample

• Results here for testing samples. Verification samples in our tests gave essentially same results.

Byron Roe 15

Neural Network Structure

Combine the features in a non-linear way to a “hidden layer” and then to a “final layer”

Use a training set to find the best wik to distinguish signal and background

Byron Roe 16

Intuition

• Neural nets and most modern methods use PID variables in complicated non-linear ways. Intuition is somewhat difficult

• However, they are often much more efficient than cuts and are used more and more.

• I will not discuss neural nets further, but will discuss modern methods—boosting,etc.

Byron Roe 17

Boosted Decision Trees

• What is a decision tree?

• What is “boosting the decision trees”?

• Two algorithms for boosting.

Byron Roe 18

Decision Tree

• Go through all PID variables and find best variable and value to split events.

• For each of the two subsets repeat the process

• Proceeding in this way a tree is built.

• Ending nodes are called leaves.

Background/Signal

Byron Roe 19

Select Signal and Background Leaves

• Assume an equal weight of signal and background training events.

• If more than ½ of the weight of a leaf corresponds to signal, it is a signal leaf; otherwise it is a background leaf.

• Signal events on a background leaf or background events on a signal leaf are misclassified.

Byron Roe 20

Criterion for “Best” Split

• Purity, P, is the fraction of the weight of a leaf due to signal events.

• Gini: Note that gini is 0 for all signal or all background.

• The criterion is to minimize ginileft + giniright of the two children from a parent node

Byron Roe 21

Criterion for Next Branch to Split

• Pick the branch to maximize the change in gini.

Criterion = giniparent – giniright-child –ginileft-child

Byron Roe 22

Decision Trees

• This is a decision tree

• They have been known for some time, but often are unstable; a small change in the training sample can produce a large difference.

Byron Roe 23

Boosting the Decision Tree

• Give the training events misclassified under this procedure a higher weight.

• Continuing build perhaps 1000 trees and do a weighted average of the results (1 if signal leaf, -1 if background leaf).

Byron Roe 24

Two Commonly used Algorithms for changing weights

• 1. AdaBoost

• 2. Epsilon boost (shrinkage)

Byron Roe 25

Definitions

• Xi= set of particle ID variables for event i

• Yi= 1 if event i is signal, -1 if background

• Tm(xi) = 1 if event i lands on a signal leaf of tree m and -1 if the event lands on a background leaf.

Byron Roe 26

AdaBoost

• Define err_m = weight wrong/total weight

1 _log

_m

err m

err m

exp( )i i mw w Increase weight for misidentified events

Byron Roe 27

Scoring events with AdaBoost

• Renormalize weights

• Score by summing over trees1

/N

i i ii

w w w

1

( ) ( )treeN

m mm

T x T x

Byron Roe 28

Boost (shrinkage)

• After tree m, change weight of misclassified events, typical ~0.01 (0.03). For misclassfied events:

• Renormalize weights

• Score by summing over trees1

/N

i i ii

w w w

1

( ) ( )treeN

mm

T x T x

exp(2 )i iw w

Byron Roe 29

Unwgted, Wgted Misclassified Event Rate vs No. Trees

Byron Roe 30

Comparison of methods

-boost changes weights a little at a time

• Let y=1 for signal, -1 for bkrd, T=score summed over trees

• AdaBoost can be shown to try to optimize each change of weights. exp(-yT) is minimized;

• The optimum value is

T=½ log odds probability that y is 1 given x

Byron Roe 31

Tests of Boosting Parameters

• 45 Leaves seemed to work well for our application• 1000 Trees was sufficient (or over-sufficient).• AdaBoost with about 0.5 and -Boost withabout 0.03

worked well, although small changes made little difference.

• For other applications these numbers may need adjustment

• For MiniBooNE need around 50-100 variables for best results. Too many variables degrades performance.

• Relative ratio = const.*(fraction bkrd kept)/ (fraction signal kept).

Smaller is better!

Byron Roe 32

Effects of Number of Leaves and Number of Trees

Smaller is better! R = c X frac. sig/frac. bkrd.

Byron Roe 33

Number of feature variables in boosting

• In recent trials we have used 182 variables. Boosting worked well.

• However, by looking at the frequency with which each variable was used as a splitting variable, it was possible to reduce the number to 86 without loss of sensitivity. Several methods for choosing variables were tried, but this worked as well as any

• After using the frequency of use as a splitting variable, some further improvement may be obtained by looking at the correlations between variables.

Byron Roe 34

Effect of Number of PID Variables

Byron Roe 35

Comparison of Boosting and ANN

• Relative ratio here is ANN bkrd kept/Boosting bkrd kept. Greater than one implies boosting wins!

• A. All types of background events. Red is 21 and black is 52 training var.

• B. Bkrd isevents. Red is 22 and black is 52 training variables

Percent nue CCQE kept

Byron Roe 36

Robustness

• For either boosting or ANN, it is important to know how robust the method is, i.e. will small changes in the model produce large changes in output.

• In MiniBooNE this is handled by generating many sets of events with parameters varied by about 1 and checking on the differences. This is not complete, but, so far, the selections look quite robust for boosting.

Byron Roe 37

How did the sensitivities change with a new optical model?

• In Nov. 04, a new, much changed optical model of the detector was introduced for making MC events

• The reconstruction tunings needed to be changed to optimize fits for this model

• Using the SAME PID variables as for the old model:

• For a fixed background contamination of events fraction of signal kept dropped by 8.3% for boosting and dropped by 21.4% for ANN

Byron Roe 38

For ANN

• For ANN one needs to set temperature, hidden layer size, learning rate… There are lots of parameters to tune.

• For ANN if one a. Multiplies a variable by a constant, var(17) 2.var(17) b. Switches two variables var(17)var(18) c. Puts a variable in twiceThe result is very likely to change.

Byron Roe 39

For Boosting

• Only a few parameters and once set have been stable for all calculations within our experiment.

• Let y=f(x) such that if x1>x2 then y1>y2, then the results are identical as they depend only on the ordering of values.

• Putting variables in twice or changing the order of variables has no effect.

Byron Roe 40

Tests of Boosting Variants

• None clearly better than AdaBoost or EpsilonBoost

• I will not go over most, except Random Forests• For Random Forest, one uses only a random

fraction of the events (WITH replacement) per tree and only a random fraction of the variables per node. NO boosting is used—just many trees. Each tree should go to completion—every node very small or pure signal or background

• Our UM programs weren’t designed well for this many leaves and better results (Narsky) have been obtained—but not better than boosting.

Byron Roe 41

Byron Roe 42

Can Convergence Speed be Improved?

• Removing correlations between variables helps.

• Random Forest WHEN combined with boosting.

• Softening the step function scoring: y=(2*purity-1); score = sign(y)*sqrt(|y|).

Byron Roe 43

Soft Scoring and Step Function

Byron Roe 44

Performance of AdaBoost with Step Function and Soft Scoring Function

Byron Roe 45

Conclusions for Nuisance Variables

• Likelihood ratio methods seem very useful as an organizing principle with or without nuisance variables

• Some problems in extreme cases where data is much smaller than is expected

• Several tools for handling nuisance variables were described. The method using approx. likelihood to construct Neyman region seems to have good performance.

Byron Roe 46

References for Nuisance Variables 1

• J. Neyman, Phil. Trans. Royal Soc., London A333 (1937).• G.J. Feldman and R.D. Cousins, Phys. Rev. D57,3873 (1998).• A. Stuart, K. Ord, and S. Arnold, Kendall’s Advanced Theory of

Statistics, Vol 2A, 6th ed., (London: 1999).• R. Eitel and B. Zeitnitz, hep-ex/9809007.• The LSND collaboration, C. Athanassopoulos et. al. Phys. Rev. Lett.

75, 2650(1995); Phys. Rev. Lett. 77, 3082(1996); Phys. Rev. C54, 2685 (1996); Phys. Rev. D64, 112008 (2001).

• B. P. Roe and M.B. Woodroofe, Phys. Rev. D60, 053009 (1999).• B.P. Roe and M.B. Woodroofe, Phys. Rev. D63, 013009 (2001).

Byron Roe 47

References for Nuisance Variables 2

• R. Cousins and V.L. Highland, Nucl. Instrum. Meth. A 320, 331 (1992).

• J. Conrad, O. Botner, A. Hallgren and C. Perez de los Heros, Phys. Rev. D67, 118101 (2003).

• R.D. Cousins, to appear in Proceedings of PHYSTAT2005: Statistical Problems in Particle Physics, Astrophysics, and Cosmology (2005).

• K.S. Cranmer, Proceedings of PHYSTAT2003: Statistical Problems in Particle Physics, Astrophysics and Cosmology, 261 (2003).

• G, Punzi, to appear in Proceedings of PHYSTAT2005.• F. James and M. Roos, Nucl. Phys. B172, 475 (1980).• W.A. Rolke and A.M. Lopez, Nucl. Intrum. Meth. A458, 745 (2001).• W.A. Rolke, A.M. Lopez and J. Conrad, Nucl. Instrum. Meth. A551,

493 (2005).

Byron Roe 48

Conclusions For Classification• Boosting is very robust. Given a sufficient number of

leaves and trees AdaBoost or EpsilonBoost reaches an optimum level, which is not bettered by any variant tried.

• Boosting was better than ANN in our tests by 1.2-1.8.• There are ways (such as the smooth scoring function) to

increase convergence speed in some cases.• Several techniques can be used for weeding variables.

Examining the frequency with which a given variable is used works reasonably well.

• Downloads in FORTRAN or C++ available at:

http://www.gallatin.physics.lsa.umich.edu/~roe/

Byron Roe 49

References for Boosting• R.E. Schapire ``The strength of weak learnability.’’ Machine Learning 5 (2), 197-227

(1990). First suggested the boosting approach for 3 trees taking a majority vote• Y. Freund, ``Boosting a weak learning algorithm by majority’’, Information and

Computation 121 (2), 256-285 (1995) Introduced using many trees• Y. Freund and R.E. Schapire, ``Experiments with an new boosting algorithm, Machine

Learning: Proceedings of the Thirteenth International Conference, Morgan Kauffman, SanFrancisco, pp.148-156 (1996). Introduced AdaBoost

• J. Friedman, “Recent Advances in Predictive (Machine) Learning”, Proceedings of PHYSTAT2003: Statistical Problems in Particle Physics, Astrophyswics and Cosmology, 196 (2003).

• J. Friedman, T. Hastie, and R. Tibshirani, ``Additive logistic regression: a statistical view of boosting’’, Annals of Statistics 28 (2), 337-407 (2000). Showed that AdaBoost could be looked at as successive approximations to a maximum likelihood solution.

• T. Hastie, R. Tibshirani, and J. Friedman, ``The Elements of Statistical Learning’’ Springer (2001). Good reference for decision trees and boosting.

• B.P. Roe et. al., “Boosted decision trees as an alternative to artificial neural networks for particle identification”, NIM A543, pp. 577-584 (2005).

• Hai-Jun Yang, Byron P. Roe, and Ji Zhu, “Studies of Boosted Decision Trees for MiniBooNE Particle Identification”, Physics/0508045, NIM A555, 370-385 (2005).

Byron Roe 50

Adaboost Output for Training and Test Samples

Byron Roe 51

The MiniBooNE Collaboration

Byron Roe 52

40’ D tank, mineral oil, surrounded by about 1280 photomultipliers. Both Cher. and scintillation light. Geometrical shape and timing distinguishes events

Byron Roe 53

Numerical Results from sfitter (a second reconstruction program)

• Extensive attempt to find best variables for ANN and for boosting starting from about 3000 candidates

• Train against pi0 and related backgrounds—22 ANN variables and 50 boosting variables

• For the region near 50% of signal kept, the ratio of ANN to boosting background was about 1.2

Byron Roe 54

Post-Fitting

• Post-Fitting is an attempt to reweight the trees when summing tree scores after all the trees are made

• Two attempts produced only a very modest (few %), if any, gain.

Byron Roe 55