aida – jas · aida grid-land speaking of grid… massimiliano turri, slac infn-lnf, 7 july 2003...
TRANSCRIPT
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDA – JAS
Massimiliano Turri, SLAC
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDA
Abstract Interfaces for Data Analysishttp://aida.freehep.org
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Outline
IntroductionFeatures - AdvantagesHistory – Organization - DevelopersStatusInterfaces OverviewExampleTools and UsersConclusions
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
“The goal of the AIDA project is to define abstractinterfaces for common physics analysis objects, such ashistograms, ntuples, fitters. The adoption of theseinterfaces makes it easier for developers and users toselect and use different tools without having to learnnew interfaces or change their code. In addition it ispossible to exchange data (objects) between AIDAcompliant applications through a standard XML format.”
Introduction
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Interfaces … start with “I”
Define only pure virtual methods for analysisobjects
n AIDA defines behavior, analysis tool providesimplementation
Object factories
n independence of user code from AIDA implementationn easy to switch implementations
w IAnalysisFactory af = IAnalysisFactory.create();
Features
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Use same code with any AIDA-compliant analysis tool.
Used for experiment’s code, e.g. reconstruction orindividual user’s analysis
n User needs to learn only one set of interfaces.
Interoperability of analysis tools.
A
I
D
A
User code(e.g. GEANT4)
Analysis tool 1
Analysis tool 2
Advantages
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Distributed data analysis:n GRID enabled AIDA implementation.n No need for user to change code.
Many places within the GRID where abstract interfacescould be adopted.
n AIDA working group may be a good model to follow
GRIDNode
AnalysisObjects
AIDA
GRIDNode
AnalysisObjects
AIDAGRID-land
Speaking of GRID…
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Initial idea formed at HepVis-99 workshop atOrsay.Informal AIDA discussions at CERN in 2000AIDA workshops:n January 2001 - Paris/Orsayn April 2001 - Boston (preceding HepVis 2001)n June 2002 – CERNn July 2003 - CERN
Informal meetingsn e.g. during Geant4 meetings and V.C.
History
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Open source project
No formal collaboration/author listn currently an informal CERN-LAL-SLAC effort
Interfaces are designed by discussion and (eventual)consensus
n Takes some time, but result is well though out and robust
Past and present contributors:n Guy Barrand, Pavel Binko, Grzegorz Chwajol, Mark
Dönszelmann, Wolfgang Hoschek, Tony Johnson, EmmanuelMedernach, Dino Ferrero Merlino, Lorenzo Moneta, JakubMoscicki, Ioannis Papadopoulos, Andreas Pfeiffer, Max Sang,Victor Serbo, Max Turri
Organization and Developers
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
AIDA Version 3.0 (Oct 2002) second end-user release.n AIDA 3.2 coming out soonn AIDA 2.2 first end-user release
Interfaces available in C++ and Javan automatically generated with AID (http://java.freehep.org)
w easy to add more languages
http://aida.freehep.orgn General information, relevant linksn Tutorial, users’ guide, examplesn Downloads and web-browsable source code
Status
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Interfaces Overview
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Histogramsn fixed and variable width binning n filling n access to overall statistics (entries, mean, rms) n bin information (center, entries, height, error) n arithmetic (add, multiply, divide)n projectionsn slices
IHistogram
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsClouds n unbinned histograms
n scatter plots n auto-conversion to histograms
ICloud
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfiles n fill 1D and 2D profile plots
n access to bin statistics (height, mean , rms)n access to overall statistics (entries, mean rms)n scaling
IProfile
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfilesData Point Sets n sets of n-dimensional points with errors
n add, remove, get pointsn scaling of values and errorsn full mathematical arithmetic (+-*/)n simple error propagation
IDataPointSet
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfilesData Point SetsNtuples
n Evaluatorsn Filters
n fill and retrieve datan support folder-like structuren projections to histograms, clouds and profiles of “evaluated” quantities with “filtering”n chaining
ITuple
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfilesData Point SetsNtuples
n Evaluatorsn Filters
Plottingn create plot area, regionsn control styles (title, text, markers, lines etc.)
IPlotter
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfilesData Point SetsNtuples
n Evaluatorsn Filters
PlottingFitter
n fitting to all the data storage types IFitResult result = fitter.fit(data, function)n change fit method (c2,max. Likelihood, etc)n change optimizer (Minuit, etc)n control parameters (bounds, fix, step, etc.)n set constraintsn create scans and contoursn use function’s analytical gradient
IFitter
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfilesData Point SetsNtuples
n Evaluatorsn Filters
PlottingFitterFunctions
n create scripted or built-in functionsn access/change parameter’s valuesn evaluate function and its gradientn support of PDFs (normalized functions over a range)
IFunction
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
HistogramsCloudsProfilesData Point SetsNtuples
n Evaluatorsn Filters
PlottingFitterFunctionsIO
n storage for analysis objectsn XML interchange format standard (.aida files)nunix-like : ls, cp, mv, cd ….
ITree
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
IAnalysisFactory aF = IAnalysisFactory.create();ITree tree = aF.createTreeFactory().create();IHistogramFactory histF = aF.createHistogramFactory( tree );IFitFactory fitF = aF.createFitFactory();IFitter fitter = fitF.createFitter("Chi2");IPlotter plotter = aF.createPlotterFactory().create("Plot"); IHistogram1D gaussHist = histF.createHistogram1D("gaussHist",
"Gaussian Histogram",100,-5,5);
Random r = Random();for (i=0; i
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Three implementations are available within:n Anaphe/Lizard (C++)
w http://anaphe.web.cern.ch/anaphen Open Scientist (C++)
w http://www.lal.in2p3.fr/OpenScientistn JAIDA/JAS (Java) + AIDA-JNI 3.0 (C++)
w http://java.freehep.org/lib/freehep/doc/aida
Usersn GEANT4: Analysis Advanced examples
n BaBar online (via JAS)
n GAUDI/ATHENA users
n Linear Collider users
n LCG is considering the adoption of AIDA in their ArchitecturalBlueprintw Physicist Interfaces (PI) group is working to create an AIDA
implementation based on ROOT classes
Tools and Users
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Future
Remote Access to AIDA:n Access to remote AIDA objects (through CORBA, RMI,
SOAP, …) from another AIDA implementation.n Access to AIDA objects through WEB Browser (with
limited functionality)PAW
Root
BaBar Online
Oracle
AIDA
AIDA Tool 1
AIDA Tool 2
-Remote AIDA Tree
- WEB AccessRemote
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
Future
Achieve component level interoperabilityn e.g.: use ANAPHE histograms, fit with JAS3 fitter and
plot with OpenScientistn at last week’s workshop we focused on developer
interfacesn first prototype due before October’s workshop
AIDA 4.0 end of 2003
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
AIDAAbstract Interfaces for Data Analysis
AIDA has a comprehensive set of interfaces for dataanalysis
Three implementations are currently available
Currently a few users but potential to become popular inHEP community
AIDA needs feedback
Conclusions
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JAS
Java Analysis Studiohttp://jas.freehep.org
Mark Dönszelmann, Tony Johnson, Joseph Perl, Victor Serbo, Max Turri
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Outline
Introduction to JAS
JAS3 – a new JAS
Tour of major components of JAS3n AIDA
w Root and PAW file readersn Tuple Explorern Java compiler and loadern Scripting Languages
w Pnutsn Record loopn Spreadsheet
JAS3 Architecture
Status
Conclusions
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Introduction to JAS
JAS is a :
n general purposen open-sourcen almost fully in java
data analysis tool with the following features:
n Plotting of 1d, 2d and 3d histograms, XY plots, scatterplots etc.
n High quality print output.
n Fitting (binned or unbinned) using an extensible set of optimizersincluding Minuit.
n Export of plots in a variety of formats including PS, EPS, PDF, SVG,GIF, PNG.
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Introduction to JAS
n Easy to learn GUI for performing common analysis tasks (plotting,ntuple-analysis, fitting etc.).
n More complex analysis can be performed using a variety of scriptinglanguages (pnuts, jython, etc.), or by writing Java analysis modules.
n Able to read data in a variety of formats including: aida, hbook, root,SQL databases, text-files.
n AIDA compliant analysis system.
n Built-in editor and compiler.
n Simple spreadsheet capabilities.
n Logbook for recording analysis tasks, and optionally publishing resultsto the web.
n Highly modular structurew allows addition, replacement or removal of modules to customize application
for a particular problem domain.
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS3 - a new JAS
First version of JAS2 released in 2000n Has worked well, but some limitations
w Too heavily based on event loopn No all analysis tasks involve looping over events
w No support for scriptingw Limited fitting capabilitiesw Limited N-Tuple analysis capabilities
Often used for online monitoring, often used outside HEPn Not really designed for this use
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS3 - a new JAS
Opportunitiesn Realized a lot of overlap with WIRED project
w Created FreeHEP Java library ton Pull out (re-factor) common parts of JAS, WIREDn Implement new common requirements for JAS and WIRED
n Vector Graphics, Application Framework, Utilities
n Formed collaboration with developers of similar tools to createAIDA package
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS3 Components
Tour designed togive you anoverview of thecapabilities ofJAS3, you can trythem out foryourself with thebuilt-in Tutorialand/or examples
Welcome Page, gives initialinfo and links to example
scripts and programsMemorymonitor
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Opening Files
Use filemenu
Drag fromexplorer
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Graphical Interface to AIDA
Histograms,Clouds, Tuplesall presented in
AIDA tree
.aida files,.hbook files,.root files allpresented asAIDA objects
Drag items ontopage, or use
(popup) menus
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Printing
Can sendindividualplots orfull pagedirect toprinter
Or save as PS,EPS, PDF, SWF,SVG, PNG, GIF…
Orcopy/pasteinto Word,
PowerPointetc.
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Java Editor, Compiler and Loader
Built-in Javacompiler
Treeshowsloaded
programs
Built-ineditor forwriting
analysiscode
JAS3 allows any Java program to be loaded. This example“main routine” is taken directly from the AIDA manual
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Scripting
Can alsowrite and run
scripts
Console allowsdirect interaction
with scriptinglanguage
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Pnuts Language
Currently support Pnuts scripting languagen Complete and well documented
w http://javacenter.sun.co.jp/pnuts/doc/guide.html
n Fast (although not as fast as compiled Java)n Syntax very similar to Javan Can easily call compiled Java classes from scripts
w best of both worlds
Plan to support other languages in futuren In particular Python
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Record Sources
Opening record (orevent) based files
causes the run controltoolbar to appear
Job controls.Also supports random
access and “tagged” datasets (mainly for event
displays)
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Histogram Profile
ScatterPlot
XY Data(More appropriate forsmaller data sets)
Tuple Explorer - PlotsWorks with any tuple, read fromfile or dynamically created
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Tuple Explorer – Define Columns
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Tuple Explorer - Cuts
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Tuple Explorer - Tabulate
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
To be used with record loop
Tuple Explorer – Record Source
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS3 Spreadsheet
Simple spreadsheet pluginn for
w Displaying resultsw Calculationsw Simple Plots
n Supports reading/writingw .csv filesw Excel files
n Cut/Paste with Excel etcn Coming Soon…
w Scripting interfacew GUI for building plotsw User defined functions
n Java, scripting
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Miscellaneous Features
Save/Restoreconfiguration
UserPreferences
PluginManager
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS3 Architecture
Design based on Application Shell, into which many (optional)modules can be plugged
n All of the features we have discussed previously are implementedby one (or more) plugin modulesw Loose coupling between modules allows for modules to be removed
without breaking remainder of programw Extra modules (e.g. LCD specific modules) can be easily added
n Highly customizable for different application domainsw HEP/Astrophysics/Otherw DST analysis/Online Monitoring/GRID analysisw Experiment/User specific modules
n Modules can be updated independently of shellw Possible to release bug fixes fast
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Status
Currently released JAS3 version 0.7.2n AIDA functionality is quite solidn Compiler, Loader, Record Loop all quite recently added,
w Certainly still some rough edges
Documentation is still quite limitedn Talks, tutorials, built-in example scripts and programs
accessible from simple built-in web-browser
If you are used to JAS2 you will find some functionality notyet ported to JAS3n Remote (client/server) access to data (this is being worked on)n 3D Lego/Surface plots
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS 2 – GRID interface (Tech-X)
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS and the GRID
We plan to add client-server/distributedcapabilities to JAS3 similar (but better) thanthose in JAS2n Will be based on (remote) AIDAn Want to use Grid standards where they exist
w Work with others (PPDG-CS11,???) to define standardswhere they do not exist
n Tech-X have phase II SBIR approved and will workclosely with us
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
Help us make it better
We have recently purchasedand are now seriously using acommercial bug trackingsystem.n Used not only for bugs, but
also for tracking tasks, ideasetc.
n Please report any problems youfind, and make suggestions forchanges or improvements youwould like to see.w http://bugs.freehep.org
-
Massimiliano Turri, SLAC INFN-LNF, 7 July 2003
JASJava Analysis Studio
JAS3 Links, More Info
JAS – Java Analysis Studio - http://jas.freehep.org
JAS3 – http://jas.freehep.org/jas3
JAIDA – http://java.freehep.org/jaida/
AIDA – http://aida.freehep.org
FreeHEP - http://www.freehep.org
FreeHEP Java Libraries - http://java.freehep.org
WIRED – http://wired.freehep.org