26 april 2005steve armstrong1 high level trigger software tutorial steve armstrong brookhaven...

26 April 2005 Steve Armstrong 1

High Level Trigger Software Tutorial

Steve ArmstrongBrookhaven National Laboratory

26 April 2005ATLAS Trigger/DAQ Week at CERN


OVERVIEW OF THIS TUTORIAL

• Welcome! • Main Goal: Convert newcomers or currently inactive HLT members into active software developers – hope to break the prevalent addiction to avoiding trigger software development!

• Minimal knowledge of Unix/Linux, C++, shell scripts, cmt, Athena, ATLAS, paw or ROOT is required;

• DISCLAIMER: this is first tutorial of this type given, so please give me feedback on what is good/bad/clear/not clear about it!

• ATLAS Software is amazingly capricious – be prepared for this, have patience;

• Never be afraid to ask questions: frequently things are genuinely wrong and need to be fixed by experts;

• Impossible to document every step and allow for every contingency, even for simple examples.

• Be aware of good web-based resources:• http://hepunx.rl.ac.uk/atlasuk/simulation/level2/doc/PesaCore/• https://uimon.cern.ch/twiki/bin/view/Atlas/ReleaseRecipes• https://uimon.cern.ch/twiki/bin/view/Atlas/RunningReconstruction

http://hepunx.rl.ac.uk/atlasuk/simulation/level2/doc/PesaCore/

https://uimon.cern.ch/twiki/bin/view/Atlas/ReleaseRecipes

https://uimon.cern.ch/twiki/bin/view/Atlas/RunningReconstruction

https://uimon.cern.ch/twiki/bin/view/Atlas/RunningReconstruction


TUTORIAL CONVENTIONS

X.Y IS A STEP X.SUB-STEP Y IN THE TUTORIAL.

red text are commands you type at shell prompt

blue text is something that should be printed out from Linux/Athena

underlined and italicized text is not literal - replace it with something like your afs username

purple text is something that needs to be changed/checked


STEP 0. WHAT YOU SHOULD HAVE BEFORE YOU START

0.1 We will work on CERN LXPLUS machines running Scientific Linux SLC 3.0.3. Therefore you must have:

0.1.1 An active LXPLUS account;0.1.2 A means to ssh to LXPLUS from your laptop/computer.

0.2 Please create a new session on LXPLUS:

ssh -l yourusername lxplus.cern.ch

0.3 Please use a common ATLAS shell (e.g., tcsh, bash, zsh). Check your shell by doing:

ps

PID TTY TIME CMD5939 pts/26 00:00:00 tcsh10861 pts/26 00:00:00 ps

This tells you which shell you are using.


1.1 Do you have a scratch0 or maxidisk area with some space on it?

1.1.1 If yes, you should see/do:

ls

mail public scratch0 util www

cd scratch0fs listquota

Volume Name Quota Used %Used Partitions.atlas.sarmstro.0 1500000 634754 42% 88%

1.1.2 If no, you should use temporary “/tmp” space:

cd /tmpmkdir yourusernamecd /tmp/yourusername

STEP 1. CREATE A DIRECTORY TO DEVELOP SOFTWARE

Your scratch0 area exists! If not, goto 1.1.2

Your quota in kBytes. Make sure (Quota – Used) > 100000

If you have to do this, please, in subsequent transparencies, replace

“~/scratch0” with “/tmp/yourusername”


STEP 1. CREATE A DIRECTORY TO DEVELOP SOFTWARE(continued)

1.2 Make a directory in your scratch area and change into it:

cd ~/scratch0 ; mkdir Tutorial ; cd Tutorial

Here, I have chosen the directory name “Tutorial”. You can choose another name, just be careful to replace “Tutorial” in the subsequent slides with the name you

have chosen.


STEP 2. CONFIGURE CMT TO USE RELEASE 10.0.1

2.3 Edit the cmt requirements file to look something like this:

set CMTSITE CERN \ cmt_atlas "ATLAS"

macro ATLAS_DIST_AREA "/afs/cern.ch/atlas/software/dist"

## Select releases as they are available#macro ATLAS_RELEASE "10.0.1"

#otherwise use nightly#macro ATLAS_RELEASE "/nightlies/rel/atlrel_3"

use AtlasSettings v* $(ATLAS_DIST_AREA)#use HLTSettings HLTSettings-* $(HLT_DIST_AREA)/releases/$(HLT_RELEASE)

## Prepend your test area(s) according to your choices#path_remove CMTPATH /afs/cern.ch/user/s/sarmstro/scratch0/Tutorialpath_prepend CMTPATH /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial

2.1 In your Tutorial area, make a “cmtdir” and change into it:cd ~/scratch0/Tutorial ; mkdir cmtdir; cd cmtdir

2.2 Get a copy of a cmt requirements file:cp ~sarmstro/public/requirements ./

Make sure this says “10.0.1”

Change these TWO lines to make it agree with the full

path of YOUR Tutorial area


STEP 3. SETUP CMT AND DOUBLE-CHECK THINGS

3.2 Make sure your cmt is working and configured properly:

cmt config------------------------------------------Configuring environment for standalone package.CMT version v1r16p20040901.System is i686-slc3-gcc323-dbg------------------------------------------Creating setup scripts.Creating cleanup scripts.

3.1 Make sure you are in your Tutorial cmtdir and Issue CMT setup command:

cd ~/scratch0/Tutorial/cmtdirsource /afs/cern.ch/sw/contrib/CMT/v1r16/mgr/setup.[c]sh

3.3 Execute cmt setup script:source setup.[c]sh -tag=opt

3.4 Make sure the cmt path is sensible:echo $CMTPATH/afs/cern.ch/user/s/sarmstro/scratch0/Tutorial:/afs/cern.ch/atlas/software/

dist/10.0.1:/afs/cern.ch/atlas/offline/external/Gaudi/0.14.6.14pool201:/afs/cern.ch/atlas/offline/external/LCGCMT/LCGCMT_33

“dbg” means we are going to compile a debug version

of the code which is significantly slower than

“opt” or optimized version


4.2 For demonstration purposes, let us also explicitly check out an HLT algorithm, T2CALO. Normally this would be done only if you were modifying it.

cmt co -r TrigT2Calo-00-03-03 Trigger/TrigAlgorithms/TrigT2Calo

4.1 Change directory to the top of your Tutorial area and check out a specific tag of the TriggerRelease package

cd ~/scratch0/Tutorialcmt co -r TriggerRelease-00-02-22 Trigger/TriggerRelease

STEP 4. CHECK OUT TRIGGER SPECIFIC PACKAGES

This tag, 00-02-22, is more recent than the one in Release 10.0.1 – it

contains some minor fixes

Again, we are using a specific tag version. If we were really modifying T2CALO in a development context, we would most

certainly work with the “head” cvs version and in careful coordination with the other developers of this algorithm.


STEP 5. MODIFY PACKAGE CMT REQUIREMENTS FILES

5.1 Within each Athena package, there is a cmt directory containing a requirements file for that package. We need to modify the one for TriggerRelease to let it know we have explicitly checked out T2CALO and may be modifying it:

cd ~/scratch0/Tutorial/Trigger/TriggerRelease/TriggerRelease-00-02-22/cmtemacs -nw requirements

5.2 Within the TriggerRelease requirements file there are many “use” statements that are commented with a “#”. Look for the one for T2CALO and uncomment it (i.e., delete the “#”)

...#use TrigAlgExample TrigAlgExample-* Trigger/TrigAlgorithms#use TrigIDSCAN TrigIDSCAN-* Trigger/TrigAlgorithms#use TrigMoore TrigMoore-* Trigger/TrigAlgorithms#use TrigSiTrack TrigSiTrack-* Trigger/TrigAlgorithms#use TrigSiTreeAlg TrigSiTreeAlg-* Trigger/TrigAlgorithmsuse TrigT2Calo TrigT2Calo-* Trigger/TrigAlgorithms#use TrigTRTLUT TrigTRTLUT-* Trigger/TrigAlgorithms#use TrigTRTxK TrigTRTxK-* Trigger/TrigAlgorithms...


STEP 6. CONFIGURE AND COMPILE PACKAGES

6.1 From the cmt directory of TriggerRelease, setup things and then issue a cmt “broadcast” command to build packages

cd ~/scratch0/Tutorial/Trigger/TriggerRelease/TriggerRelease-00-02-22/cmtcmt broadcast cmt configsource setup.[c]shcmt broadcast gmake

and wait a few minutes...


STEP 7. PREPARE THE RUN DIRECTORY

7.1 Create a run directory and cd to it:

mkdir ~/scratch0/Tutorial/runcd ~/scratch0/Tutorial/run

7.2 Setup run area:

source /afs/cern.ch/atlas/software/dist/10.0.1/Control/AthenaRunTime/*/cmt/setup.[c]sh

7.3 Setup RecExCommon:

RecExCommon_links.[c]sh

7.4 Are there files in your run directory?

ls

dc1.001003.simul.redig.00001.test.h130_2e2mu_extract.zebra PDGTABLE.MeV ZEBRA_HEEMM.P ZEBRA_ZEE_LUMI10.Pdc2_mixing_all_phys_0003621_file100002.data PoolFileCatalog.xml ZEBRA_HEEMM_TDR.P ZEBRA_ZEE.PFastShowerHistograms PoolFileCatalog.xml.BAK ZEBRA_JET_25.P jobOptions.py RawEvent_hllll_g3_extract.re ZEBRA.PmyTopOptions.py ZEBRA_HEEMM_EXTRACT.P ZEBRA_ZEE_LUMI02.P


STEP 8. COPY HLT SOFTWARE CONTROL FILES

8.1 Make sure you are still in the run directory:

cd ~/scratch0/Tutorial/run

8.2 Use the “get_files” command to get files:

get_files -copy -xmls LVL1triggermenu.xml LVL1triggerthresholds.xml Finding LVL1triggermenu.xmlFinding LVL1triggerthresholds.xmlCopying file LVL1triggermenu.xml from /afs/cern.ch/user/s/sarmstro/scratch0/TutorialCopying file LVL1triggerthresholds.xml from /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial

get_files -copy -xmls HLTsequence.xml HLTsignature.xml Finding HLTsequence.xmlFinding HLTsignature.xmlCopying file HLTsequence.xml from /afs/cern.ch/user/s/sarmstro/scratch0/TutorialCopying file HLTsignature.xml from /afs/cern.ch/user/s/sarmstro/scratch0/Tutorial


STEP 9. EXAMINE/MODIFY HLT SEQUENCE FILE

9.1 Make sure you are in run directory and edit HLTsequence.xml file:

cd ~/scratch0/Tutorial/runemacs -nw HLTsequence.xml

9.2 Modify your HLTsequence.xml file to look like this:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<!DOCTYPE SEQUENCE_LIST SYSTEM "sequencelist.dtd"><SEQUENCE_LIST>

<SEQUENCE level="L2" input="EMTAUROI"algorithm="T2Calo/T2Calo/g4TrigIdScan::IdScanMain/IdScan_IdScanMain/1TrigSiTrack/SiTrack/SiTrackTrigEgammaNtuple/TrigEgammaNtuple/1"output="LVL2out" />

</SEQUENCE_LIST>

L2 and EMTAUROI means that this sequence is for Level-2 will be executed upon receipt of a Level-1

accept for EMTAU triggers

This specified the exact sequence of HLT

algorithms that will be run everytime we have an RoI

from Level-1


STEP 10. GET AND MODIFY JOBOPTIONS



10.2 Copy Trigger jobOptions using the “get_files” command:

get_files -copy Trigger_topOptions_rome.py

10.3 Examine jobOptions:

emacs -nw Trigger_topOptions_rome.py

Locate and make note of the following lines:

doTrigger = True

EvtMax = 2

PoolRDOInput = [ "LFN:ele25_Rome_RDO_extract.pool" ]

doCBNT = True

This tells Athena that we are using Trigger software.

This tells Athena how many events to process. Change to a low number (2) at first to make sure job runs!

This is our MC file containing single electrons of 25 GeV. You

can change it to your own favorite file, but you must make

sure it is known in Pool file catalog in your run directory.

This tells Athena that we want a combined Ntuple (CBNT) output.

Note: these days, jobOptions in Athena are in python, which is powerful, but yet

another language to learn.


STEP 10. GET AND MODIFY JOBOPTIONS (continued)

10.4 For this tutorial, we want to run the Level-2 Algorithm T2CALO. Let us make sure the jobOptions is reading from our HLTsequence file. So we modify jobOptions:

emacs -nw Trigger_topOptions_rome.py

Comment the following lines:# JobO with TrigCaloRec#TriggerConfig.sequenceListFileLocation = "HLTsequenceLVL2_EF.xml"#TriggerConfig.signatureListFileLocation = "HLTsignatureLVL2_EF.xml"

Uncomment the following lines:# JobO without TrigCaloRecTriggerConfig.sequenceListFileLocation = "HLTsequence.xml"TriggerConfig.signatureListFileLocation = "HLTsignature.xml"

10.5 Are you uncomfortable with using ROOT to look at Ntuples? If so, you may insert the following line in your jobOptions:

useROOTNtuple=False

but make sure you put it in only AFTER the line

include ("RecExCommon/RecExCommon_flags.py")

so you can use PAW instead of ROOT to look at ntuples.


10.6 Insert the default data file into the pool catalog:

pool_insertFileToCatalog ${ATLASTESTDATA}/ele25_digi_Rome_10ev_test.pool.rootFCregisterLFN -p ${ATLASTESTDATA}/ele25_digi_Rome_10ev_test.pool.root -l \ele25_Rome_RDO_extract.pool

STEP 10. GET AND MODIFY JOBOPTIONS (and put data file intoPOOL catalog) (continued)


STEP 11. ENHANCE MEMORY AND RUN ATHENA!



11.3 Run athena with the Trigger jobOptions:

athena.py -bs Trigger_topOptions_rome.py >& out1.log &

11.2 Enhance shell virtual memory:

(for tcsh): limit vmemoryuse 1300000(for zsh): limit addressspace 1300000(for bash): ulimit -Sv 1300000

11.4 Monitor progress:

tail -f out1.log

The “-bs” flag tells Athena to print out ALL jobOptions used (i.e., the included jobOptions from included jobOptions...) to configure your job. This will result in a

HUGE but very instructive log file. We do it here to demonstrate how highly configured and configurable Athena is. Frequently, problems running an Athena

job are due to mistakes in jobOptions rather than C++ source code.


STEP 12. UNDERSTAND OUTPUT FILE

12.1 As noted before, we are running athena with “-bs” option to explicitly show all jobOptions being used. Hence, in the log file, you will have literally thousands of lines looking like:

Athena INFO including file "AthenaCommon/Compat.py"Athena INFO including file "AthenaCommon/Bootstrap.py" -+- 0 #************************************************************** -+- 1 # -+- 2 # Athena bootstrap options file -+- 3 # -+- 4 #============================================================== -+- 5 -+- 6 #-------------------------------------------------------------- -+- 7 # Atlas specific event loop and output stream -+- 8 #-------------------------------------------------------------- -+- 9 A theApp.EventLoop = "AthenaEventLoopMgr" -+- 10 A theApp.OutStreamType = "AthenaOutputStream“

Line number of current jobOptions file being

included.

Important! Only lines with “A” (for Active) are used – without “A” means it is a comment or part of a conditional statement

which was not satisfied.

12.2 After lots of lines (about 8000) you should see lines showing what was defined by the HLTsequence.xml file:

SequenceTable INFO *** Sequence table for step 1 ***Sequence INFO - Level: L2 Input(s): EMTAUROI Algo(s):

T2Calo/T2Calo/g4 TrigIdScan::IdScanMain/IdScan_IdScanMain/1

TrigSiTrack/SiTrack/SiTrack TrigEgammaNtuple/TrigEgammaNtuple/1 Output:

LVL2out


STEP 12. UNDERSTAND OUTPUT FILE (continued)

12.3 Event processing starts at about line 8300 of the log file with the lines:

AthenaEventLoopMgr INFO ===>>> start of run 2885 <<<===AthenaEventLoopMgr INFO ===>>> start of event 1 <<<===

12.4 If everything worked OK, you should see at the end of your output file:

NTupleSvc INFO NTuples saved successfullyApplicationMgr INFO Application Manager Finalized successfullyApplicationMgr INFO Application Manager Terminated successfully


STEP 13. LOOK AT RESULTS!

13.1 A successful athena job produces a CBNT ntuple output file as specified by the default jobOptions (you can change if you want):

RootNtupleOutput = ntuple.rootHbookNtupleOutput = ntuple.hbook

13.2 The content of the LVL2 Calorimeter Ntuple block is defined in the package Trigger/TrigAnalysis/TrigNtCalo. You can have a look at it here:

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAnalysis/TrigNtCalo/src/TrigNtCaloBlock.cxx

13.3 Let’s make a quick plot of Electromagnetic Cluster Energy, T2CaEmE using paw or root on the ntuple file…

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAnalysis/TrigNtCalo/src/TrigNtCaloBlock.cxx


STEP 14. LEARN MORE ABOUT AN HLT ALGORITHM!

14.1 Examine the code of T2CALO using the LXR browser:

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/

14.2 Note the 4 main “tools” that actually do the work of T2CALO:

CaloSamp2Fex.cxx which calculates cluster position and energy in EM Sampling 2:http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloSamp2Fex.cxx

CaloSamp1Fex.cxx which calculates strip profile and energy in EM Sampling 1:http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloSamp1Fex.cxx

CaloEmEnFex.cxx which calculates energy in EM Presampler and Sample 3:http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloEmEnFex.cxx

CaloHadEnFex.cxx which calculates energy in Tile and HEC:http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloHadEnFex.cxx

14.3 Examine the T2CALO jobOptions which are read in and control T2CALO operation, found in the T2CALO ./share area:

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/share/jobOfragment_TrigT2Calo.py

and make special note of the line that defines which tools and their order of execution:

T2Calo_g4_L2.T2CaloTools=["CaloSamp2Fex","T2CaloL1Sim","CaloSamp1Fex","CaloEmEnFex","CaloHadEnFex"]

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloSamp2Fex.cxx

http://atlassw1.phy.bnl.gov/lxr/source/atlas/Trigger/TrigAlgorithms/TrigT2Calo/src/CaloSamp1Fex.cxx

26 april 2005steve armstrong1 high level trigger software tutorial steve armstrong brookhaven...

Documents

atlas software

atlas atlas macro atlas

cern slide

steve armstrong4 step

steve armstrong6 step

steve armstrong7 step

tutorial welcome

common atlas shell