gaudi framework tutorial, 2004 8 interfacing gaudi with python

Gaudi Framework Tutorial, 2004

88

Interfacing Gaudi with PythonInterfacing Gaudi with Python

8-2 Gaudi Framework Tutorial, 2004

ObjectivesObjectives

After completing this lesson, you should After completing this lesson, you should be able to:be able to:

• Understand how Gaudi is interfaced to Python

• Convince you that Python is a very good programming language for Rapid Prototyping

• Write simple Python scripts to access LHCb data

After completing this lesson, you should After completing this lesson, you should be able to:be able to:

• Understand how Gaudi is interfaced to Python

• Convince you that Python is a very good programming language for Rapid Prototyping

• Write simple Python scripts to access LHCb data


Python LanguagePython LanguagePython is an Open Source programming language Python is an Open Source programming language created by Guido van Rossum. While it has been created by Guido van Rossum. While it has been available since 1990, it has recently seen a dramatic available since 1990, it has recently seen a dramatic increase in popularity in a wide variety of domainsincrease in popularity in a wide variety of domains

Python is simple, elegant, powerful and reliablePython is simple, elegant, powerful and reliable

Features:Features:

• Variables & Arithmetic expressions

• String manipulations

• Conditionals (if/else statements), Loops (for/while)

• Functions, Lists, Dictionaries, Classes (Objects), Exceptions , Modules

Python is an Open Source programming language Python is an Open Source programming language created by Guido van Rossum. While it has been created by Guido van Rossum. While it has been available since 1990, it has recently seen a dramatic available since 1990, it has recently seen a dramatic increase in popularity in a wide variety of domainsincrease in popularity in a wide variety of domains

Python is simple, elegant, powerful and reliablePython is simple, elegant, powerful and reliable

Features:Features:

• Variables & Arithmetic expressions

• String manipulations

• Conditionals (if/else statements), Loops (for/while)

• Functions, Lists, Dictionaries, Classes (Objects), Exceptions , Modules


Why Python?Why Python?Free, Open-Source, MaintainableFree, Open-Source, Maintainable

Cross-Platform, PortableCross-Platform, Portable

General-Purpose, High-level, Object-OrientedGeneral-Purpose, High-level, Object-Oriented

InterpretedInterpreted

• However it is quite fast (byte code idea from Java)

Dynamically typed, IntrospectiveDynamically typed, Introspective

• Not need to declare any variable

Simple syntax, RobustSimple syntax, Robust

• Emphasis by the author to minimize typing

Variety of ShellsVariety of Shells

Powerful built-in types and modulesPowerful built-in types and modules

Ideal for Scripting and PrototypingIdeal for Scripting and Prototyping

Free, Open-Source, MaintainableFree, Open-Source, Maintainable

Cross-Platform, PortableCross-Platform, Portable

General-Purpose, High-level, Object-OrientedGeneral-Purpose, High-level, Object-Oriented

InterpretedInterpreted

• However it is quite fast (byte code idea from Java)

Dynamically typed, IntrospectiveDynamically typed, Introspective

• Not need to declare any variable

Simple syntax, RobustSimple syntax, Robust

• Emphasis by the author to minimize typing

Variety of ShellsVariety of Shells

Powerful built-in types and modulesPowerful built-in types and modules

Ideal for Scripting and PrototypingIdeal for Scripting and Prototyping


Python as a “Glue”Python as a “Glue”Extension ModulesExtension Modules

• Very easy to interface to C++ classes (C-API)

CollaborativeCollaborative

• Extension modules can be plugged and used together

Extension ModulesExtension Modules

• Very easy to interface to C++ classes (C-API)

CollaborativeCollaborative

• Extension modules can be plugged and used together

GUI

Python

mathmathshell

gaudipython

DatabaseEDG APIGUI

Very rich set of Python standard modules

Several GUI toolkits

XML

Very rich set specialized generic modules

GaudiFramework

PyROOT

RootClasses

PVSS

JPE

JavaClasses

LHC modules

Gateways to

other

frameworks


Python Binding TechniquesPython Binding Techniques

Python provides a very complete C-APIPython provides a very complete C-API• By hand bindings are possible but even for

medium-sized projects, it becomes cumbersome and unmaintainable.

Developing Python bindingsDeveloping Python bindings• Several existing options: SWIG, Boost.Python, SIP

PyLCGDict: Python binding to the LCG DictionaryPyLCGDict: Python binding to the LCG Dictionary• It allows the user to interact with any C++ class for

which the LCG Dictionary has been generated• With this module, there is no need to generate

specialized Python bindings or wrapper code to interact with C++ classes

Python provides a very complete C-APIPython provides a very complete C-API• By hand bindings are possible but even for

medium-sized projects, it becomes cumbersome and unmaintainable.

Developing Python bindingsDeveloping Python bindings• Several existing options: SWIG, Boost.Python, SIP

PyLCGDict: Python binding to the LCG DictionaryPyLCGDict: Python binding to the LCG Dictionary• It allows the user to interact with any C++ class for

which the LCG Dictionary has been generated• With this module, there is no need to generate

specialized Python bindings or wrapper code to interact with C++ classes


PyLCGDict: Mode d’emploi PyLCGDict: Mode d’emploi

From class definitions (.h From class definitions (.h files) a “dictionary” library is files) a “dictionary” library is producedproduced

• Description of the class

• “stub” functions to class methods

Absolutely non-intrusiveAbsolutely non-intrusive

The PyLCGDict module does The PyLCGDict module does the adaptation between the adaptation between Python objects and C++ Python objects and C++ objects in a generic wayobjects in a generic way

• It works for any dictionary

From class definitions (.h From class definitions (.h files) a “dictionary” library is files) a “dictionary” library is producedproduced

• Description of the class

• “stub” functions to class methods

Absolutely non-intrusiveAbsolutely non-intrusive

The PyLCGDict module does The PyLCGDict module does the adaptation between the adaptation between Python objects and C++ Python objects and C++ objects in a generic wayobjects in a generic way

• It works for any dictionary

MyClass.hMyDict.so

select.xml

lcgdict

Pythoninterpreter

loadMyClass.so

MyScript.py

PyLCGDict


PyLCGDict: Supported FeaturesPyLCGDict: Supported FeaturesConversion of C++ and Python primitive typesConversion of C++ and Python primitive types

C++ classesC++ classes• Mapped to Python classes and loaded on demand.

Templated classes supported.

C++ namespacesC++ namespaces• Mapped to python scopes. The "::" separator is replaced

by the python "." separator

Class methodsClass methods• Static and non static class methods are supported.

Default arguments.• Method arguments are passed by value or by reference• The return values are converted into Python types and

new Python classes are created if required. Dynamic type returned if possible.

• Method overloading works by dispatching sequentially to the available methods with the same name until a match with the provided arguments is successful.

Conversion of C++ and Python primitive typesConversion of C++ and Python primitive types

C++ classesC++ classes• Mapped to Python classes and loaded on demand.

Templated classes supported.

C++ namespacesC++ namespaces• Mapped to python scopes. The "::" separator is replaced

by the python "." separator

Class methodsClass methods• Static and non static class methods are supported.

Default arguments.• Method arguments are passed by value or by reference• The return values are converted into Python types and

new Python classes are created if required. Dynamic type returned if possible.

• Method overloading works by dispatching sequentially to the available methods with the same name until a match with the provided arguments is successful.


PyLCGDict: Supported FeaturesPyLCGDict: Supported Features

Class data membersClass data members• Public data members are accessible as Python

properties

Emulation of Python containersEmulation of Python containers• Container C++ classes (std::vector, std::list, std::map

like) are given the behavior of the Python collections to be use in iterations and slicing operations.

Operator overloadingOperator overloading• Standard C++ operators are mapped to the

corresponding Python overloading operators

Class data membersClass data members• Public data members are accessible as Python

properties

Emulation of Python containersEmulation of Python containers• Container C++ classes (std::vector, std::list, std::map

like) are given the behavior of the Python collections to be use in iterations and slicing operations.

Operator overloadingOperator overloading• Standard C++ operators are mapped to the

corresponding Python overloading operators


PyLCGDict: Example (1)PyLCGDict: Example (1)#include <iostream>namespace Example { class MyClass { public: MyClass() : m_value(0) {} MyClass(const MyClass& m ) : m_value(m.m_value) {} ~MyClass() {} int doSomething(const std::string& something ) { std::cout << “I am doing something with “ << something << std::endl; return something.size(); } int value() { return m_value; } void setValue(int v) { m_value = v; } private: int m_value; public: float fprop; std::string sprop; };}

MyClass.h


PyLCGDict: Example (2)PyLCGDict: Example (2)> pythonPython 2.2.2 (#1, Feb 8 2003, 12:11:31) [GCC 3.2] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import PyLCGDict>>> PyLCGDict.loadDict('MyClassDict')>>> m = Example.MyClass()>>> m.doSomething('care')I am doing something with care4>>> m.setValue(99)>>> print m.value()99>>> n = Example.MyClass(m) # copy constructor>>> print n.value()99>>> n.fprop = 1.2>>> n.sprop = 'String property'>>> print n.sprop, n.fpropString property 1.2


Another Example: ROOTAnother Example: ROOT

>>> from ROOT import gRandom, TCanvas, TH1F>>> c1 = TCanvas('c1','Example',200,10,700,500)>>> hpx = TH1F('hpx','px',100,-4,4)>>> for i in xrange(25000):... px = gRandom.Gaus()... hpx.Fill(px)...>>> hpx.Draw()>>> c1.Update()

The ROOT.py loads the ROOTDict.so and initializes the ROOT system


GaudiPythonGaudiPython

Enabling the interaction of Gaudi components from Enabling the interaction of Gaudi components from Python interpretedPython interpreted

• Configuration, Interactivity, etc.

Starting from Gaudi v14r1, GaudiPython has been Starting from Gaudi v14r1, GaudiPython has been re-implemented using PyLCGDictre-implemented using PyLCGDict

• Generated dictionaries for most common Gaudi “Interfaces” and “Base classes” (~80 classes)

• Not need to generate dictionaries for all classes (in particular the implementations)

Enabling the interaction of Gaudi components from Enabling the interaction of Gaudi components from Python interpretedPython interpreted

• Configuration, Interactivity, etc.

Starting from Gaudi v14r1, GaudiPython has been Starting from Gaudi v14r1, GaudiPython has been re-implemented using PyLCGDictre-implemented using PyLCGDict

• Generated dictionaries for most common Gaudi “Interfaces” and “Base classes” (~80 classes)

• Not need to generate dictionaries for all classes (in particular the implementations)



The end-user module “gaudimodule.py” The end-user module “gaudimodule.py” hides some of the technicalities and adds hides some of the technicalities and adds some handy functionalitysome handy functionality

• Very easy to extern/modify/adapt since is written in Python

• Basically backward compatible with previous version

The end-user module “gaudimodule.py” The end-user module “gaudimodule.py” hides some of the technicalities and adds hides some of the technicalities and adds some handy functionalitysome handy functionality

• Very easy to extern/modify/adapt since is written in Python

• Basically backward compatible with previous version


Using GaudiPythonUsing GaudiPython

Add in the requirements file:Add in the requirements file:Add in the requirements file:Add in the requirements file:

use GaudiPython v*


Starting MethodsStarting Methods

Starting a Gaudi application from Python ShellStarting a Gaudi application from Python ShellStarting a Gaudi application from Python ShellStarting a Gaudi application from Python Shell

>>> import gaudimodule>>> gaudi = gaudimodule.AppMgr()>>> gaudi.run(100) ...>>> gaudi.exit()

Starting a Python interpreted from a Gaudi applicationStarting a Python interpreted from a Gaudi application

ApplicationMgr.DLLs += { "GaudiPython“ };ApplicationMgr.Runable = "PythonScriptingSvc";PythonScriptingSvc.StartupScript = "../options/AnalysisTest.py";

Optional initial script


ConfigurationConfiguration

Instead of using “JobOptions” files we could use Instead of using “JobOptions” files we could use Python to completely configure Gaudi applicationsPython to completely configure Gaudi applications

• ATLAS is going in this direction

““Properties” are mapped to native Python types (int, Properties” are mapped to native Python types (int, float, str, list, etc)float, str, list, etc)

Instead of using “JobOptions” files we could use Instead of using “JobOptions” files we could use Python to completely configure Gaudi applicationsPython to completely configure Gaudi applications

• ATLAS is going in this direction

““Properties” are mapped to native Python types (int, Properties” are mapped to native Python types (int, float, str, list, etc)float, str, list, etc)

>>> gaudi = gaudimodule.AppMgr()>>> gaudi.EvtMax = 100>>> gaudi.TopAlg = [‘Alg1’, ‘Alg2’]>>> alg1 = gaudi.algorithm(‘Alg1’)>>> alg1.EnergyCut = 10.7>>> msgsvc = gaudi.service('MessageSvc')>>> msgsvc.OutputLevel = 3 ...


Interacting with the Event Interacting with the Event

Since we have the complete set of Dictionaries Since we have the complete set of Dictionaries we can interact with any Event classwe can interact with any Event class

References to Event objects are obtained by References to Event objects are obtained by asking the Event data service ( AppMgr.evtSvc() ) asking the Event data service ( AppMgr.evtSvc() )

• There are currently some small limitations (i.e. handling directly KeyedContainer<T>, etc.)

Since we have the complete set of Dictionaries Since we have the complete set of Dictionaries we can interact with any Event classwe can interact with any Event class

References to Event objects are obtained by References to Event objects are obtained by asking the Event data service ( AppMgr.evtSvc() ) asking the Event data service ( AppMgr.evtSvc() )

• There are currently some small limitations (i.e. handling directly KeyedContainer<T>, etc.)

>>> evt = gaudi.evtSvc()>>> header = evt['Header']>>> print ‘Run number = ‘, header.runNum()>>> mcparts = evt['MC/Particles']>>> for p in mcparts.containedObjects() :... print p.momentum().px()


Interacting with HistogramsInteracting with Histograms

It is possible to book and fill histograms using It is possible to book and fill histograms using the Histogram data servicethe Histogram data service

• Same interface as in C++

The complete AIDA Histogram interfaces are The complete AIDA Histogram interfaces are available to Pythonavailable to Python

It is possible to book and fill histograms using It is possible to book and fill histograms using the Histogram data servicethe Histogram data service

• Same interface as in C++

The complete AIDA Histogram interfaces are The complete AIDA Histogram interfaces are available to Pythonavailable to Python

>>> import random>>> his = gaudi.histsvc()>>> h1 = his.book('h1',‘Histogram title', 40, 0, 1000)>>> for v in range(1000) :... h.fill(random.uniform(0,1000))>>> print h1.mean()


Complete Example: Read a DST fileComplete Example: Read a DST fileimport gaudimodulegaudi = gaudimodule.AppMgr(outputlevel=5, joboptions='$EVENTSYSROOT/options/PoolDicts.opts')

sel = gaudi.evtsel()sel.open(['PFN:rfio:/castor/cern.ch/lhcb/DC04/00000541_00000001_10.dst'])

evt = gaudi.evtsvc()his = gaudi.histsvc()

h1 = his.book('h1','# of MCParticles', 40, 0, 5000)

class DumpAlg(gaudimodule.PyAlgorithm): def execute(self): evh = evt['Header'] mcps = evt['MC/Particles'] print 'event # = ',evh.evtNum() h1.fill(mcps.size())

gaudi.addAlgorithm(DumpAlg())gaudi.run(10)print h1, h1.contents()

Ex/PythonExample

Ex/PythonExample


Complete Example: Read a DST fileComplete Example: Read a DST file

EventSelector SUCCESS Reading Event record 0. Record number within stream 0: 0event # = 1event # = 2event # = 3event # = 4event # = 5event # = 6event # = 7event # = 8event # = 9event # = 10Histogram 1D "# of MCParticles" 40 bins [0.000000,5000.000000] [5, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]


Building on GaudiPythonBuilding on GaudiPython

It is fairly easy to extern the functionality It is fairly easy to extern the functionality provided by GaudiPythonprovided by GaudiPython

There are already several “products” which are There are already several “products” which are build on top of GaudiPythonbuild on top of GaudiPython

BenderBender

• Physics Analysis on Python

PanoramixPanoramix

• Detector and Event Visualization

It is fairly easy to extern the functionality It is fairly easy to extern the functionality provided by GaudiPythonprovided by GaudiPython

There are already several “products” which are There are already several “products” which are build on top of GaudiPythonbuild on top of GaudiPython

BenderBender

• Physics Analysis on Python

PanoramixPanoramix

• Detector and Event Visualization


SummarySummaryThis is an attempt to convince you that Python is a very This is an attempt to convince you that Python is a very good programming language for good programming language for Rapid PrototypingRapid Prototyping

• Efficient way to test your Physics Analysis ideas

Interfacing C++ classes to Python is fairly easyInterfacing C++ classes to Python is fairly easy

• Several well established technologies exists (Boost.Python, etc.)

• PyLCGDict is another way of doing it

• PyLCGDict exploits the LCG dictionaries that are already available due to object persistency (POOL)


• Is the package that enables the interaction of Gaudi from Python

• Latest version of GaudiPython is based on PyLCGDict

Examples of the available functionality has been shownExamples of the available functionality has been shown

This is an attempt to convince you that Python is a very This is an attempt to convince you that Python is a very good programming language for good programming language for Rapid PrototypingRapid Prototyping

• Efficient way to test your Physics Analysis ideas

Interfacing C++ classes to Python is fairly easyInterfacing C++ classes to Python is fairly easy

• Several well established technologies exists (Boost.Python, etc.)

• PyLCGDict is another way of doing it

• PyLCGDict exploits the LCG dictionaries that are already available due to object persistency (POOL)


• Is the package that enables the interaction of Gaudi from Python

• Latest version of GaudiPython is based on PyLCGDict

Examples of the available functionality has been shownExamples of the available functionality has been shown

gaudi framework tutorial, 2004 8 interfacing gaudi with python

Documents