existing tools in hep and particle astrophysics · 2010. 10. 8. · xldb4 – slac oct 7, 2010...

XLDB4 – SLAC Oct 7, 2010

R.Dubois Existing Tools in HEP and Particle Astro 1/20

Existing Tools in HEP and Particle Astrophysics

Richard [email protected]



OutlineOutline

• Introduction: Fermi as example user

• Analysis Toolkits: Root and FTOOLS (et al)

• Shared Catalogues

• Data Access

• Code Infrastructure



Fermi as a Case StudyFermi as a Case Study

e+

e–

γ

“HEP” detector




e+

e–

γ

Classic event display



Fermi as a Case StudyFermi as a Case StudyFermi as a Case StudyFermi as a Case Study

Big difference from a particle physics experiment!




e+

e–

γ

Telescope: (E, direction, time)



Reusing Software: Beg, Borrow and StealReusing Software: Beg, Borrow and Steal

• “HEP”– I/O format – Root– Event-based C++ processing framework – Gaudi

• Many experiments make abstract layer to I/O and implement using ROOT I/O system

– Code management tool – CMT (but moving to SCons)– Detector simulation, geometry navigation – GEANT4

• “Astro”– I/O format - FITS– High level analysis framework & toolkit: FTOOLS– Image manipulation – ds9– Catalogue sharing – Virtual Observatory (VO) etc

• High Performance Data Access– xrootd



ROOTROOT

• “replacement” for 1980-90’s PAW• has achieved world domination in HEP• my first version was 0.96 in 1996• adopted by LHC experiments• large fraction of HEP data resides in ROOT!

http://root.cern.ch/



ROOT feature setROOT feature set

• “a floor wax and a dessert topping” © SNL– Meant to be all inclusive – if ROOT doesn’t have it, you don’t need

it• Features:

– I/O format and manipulation – object I/O – a big draw!– CINT – C++ interpreter– C++ framework– Plots, displays, graphics– GUIs– Geometry– Significant tools

• Minuit (fitting)• FFTs• Neural nets• etc etc etc



ROOT Eye CandyROOT Eye Candy



Pros of ROOTPros of ROOT

• Pro– Most of the HEP world has to use it – lots of experience out there

• A large number of dedicated users• Support team is very responsive

– I/O format works well to describe arbitrarily complex event descriptions

• schema evolution and inclusion of STL objects work well– Works on all platforms– Many powerful 3rd party packages included– CINT mostly makes it easy to look at simple datastructures

• ntuples are very common format for end analysis– pyROOT combines power/robustness of python as scripting

language with compiled ROOT classes– You can do pretty much anything with it



Cons of ROOTCons of ROOT

• ROOT is now 15 years old• Details of ROOT I/O format are not published and hard to reverse

engineer– Discourages other tools from talking to ROOT

• Analysis– It is probably impossible to create a 100% compatible C++

interpreter, so there are bugs in CINT– ROOT is “for real men” – you can build anything you need in

terms of gui’s etc• Essentially only very basic gui’s are provided

– It’s a lot of work to create a publication quality plot, and somewhat arcane (ie you never remember if you’re not doing it all the time)

– No built-in interface to other data formats (except ASCII)• eg FITS!

– Is C++ really an analysis language for mortals?• Framework

– You must surrender to the borg



AstroAstro tools: FTOOLS, ds9 etctools: FTOOLS, ds9 etc• FTOOLS

– FITS older than ROOT – early 80’s• Slow to change (good and bad!)• Mandated by NASA for all their x- and γ-ray missions to allow

indefinite support– One can fetch EGRET data with stock FTOOLS and analyse 19 yr old data!– Maintained by HEASARC, with strict adherence rules

– Data generally is simple ntuples, tables or images– Collection of utilities layered on top of FITS files and “.par” files– Atomic operations with FITS files in/out and behaviour controlled by ASCII

parameter files (or from command line or prompting)– Astro folks string together lists of FTOOLS to form processing pipelines

• Efficient I/O functions all supplied in FTOOLS: format well documented!

• Write custom FTOOLS to handle the functions needed for their particular telescope

• Fermi wrote tools for event selection, exposure calculation, likelihood fitting – all meeting the FTOOLS spec

– Designed for single person executing on a single box• Lots of ways to shoot yourself in a batch environment

http://heasarc.nasa.gov/docs/software/ftools/ftools_menu.html



ds9ds9

http://hea-www.harvard.edu/RD/ds9/

• astronomical imaging and data visualization application

• supports FITS images and binary tables, multiple frame buffers, region manipulation, and many scale algorithms and colormaps.

• provides for easy communication with external analysis tasks and is highly configurable and extensible via XPA and SAMP

• support for Solaris, Linux, MacOSX, and Windows

• supports multiple frame buffers, mosaic images, tiling, blinking, geometric markers, colormap manipulation, scaling, arbitrary zoom, rotation, pan, and a variety of coordinate systems



Sharing Source CataloguesSharing Source Catalogues

• Astrophysics is a multi-wavelength business– You can’t understand a source just

from one wavelength band– Need access to source identification

and data from other telescopes to tie up to yours

• Long history of catalogue search engines– Allows for cross-ID– NED, SIMBAD are classic examples

• NASA is requiring its archive centers to interoperate well

• Now Virtual Observatories are under investigation– I have no experience with them yet

Markarian 421 MWL campaign



High Performance Data AccessHigh Performance Data Access

• Fermi’s requirements– Fermi just passed 1 PB of disk on 35 servers (mostly Sun

thumpers/thors) – not so much anymore• clustered file system

– A data reprocessing is our biggest stress activity and fetches 50 TB/day from disk

• File system that is resistant to high loads– Exploit very capable tape silo to let us tune what we keep

on disk• HSM with tape for quick, transparent retrieval of file

– Access to files outside ROOT (eg access FITS files from FTOOLS)

• Export as FUSE filesystem• We adopted xrootd, largely due to its success with BABAR and

increasing use at LHC



xrootdxrootd in a nutshellin a nutshellquery

redirectorFile servers

STK tapesilo

etc

• Distributes files across disks• Maximizes throughput• Minimizes manual disk management• Automates archiving datasets to (and restoring from) tape• Provides more reliability and scalability than NFS• Remote access via proxies• Supports access control based on Fermi collaborator list

http://xrootd.slac.stanford.edu/



xrootdxrootd ScorecardScorecard

• Fermi usage was not like BABAR usage– Many more small files; more intense usage– Took a year of tuning before launch to get good

performance• Tools for performing some functions not so well fleshed out

yet– eg distributing files, file protection scheme, marking files as

purgable, with a tape copy• It appears we have been hit by a Solaris bug in the last year

wherein heavily loaded servers drop off the network!– We think it is not xrootd’s fault…

• But overall, xrootd has been a significant improvement in our intensive access to data over a collection of file servers



Similar Data Access BeastiesSimilar Data Access Beasties

• dCache– Developed by DESY, FermiLab and Nordic Grid

• Castor– Developed by CERN

• Xrootd– Developed by SLAC

http://www.dcache.org/

http://castor.web.cern.ch/castor/

http://xrootd.slac.stanford.edu/



SummarySummary

• HEP faced the need to handle– Complex detectors, particle interactions and underlying

physics topologies; intensive access to data– Developed tools like ROOT and xrootd

• Astro’s needs were to handle– Images and telescope data, and ensure that analysis would

be possible for decades to come– Developed FITS and FTOOLS

• ROOT has taken over the HEP world, but is getting creaky

existing tools in hep and particle astrophysics · 2010. 10. 8. · xldb4 – slac oct 7, 2010...

Documents