software development for the compass experiment

Software development for the COMPASS

experiment

Martin Bodlák1 Vladimír Jarý1∗ Igor Konorov2

Alexander Mann2 Josef Nový1 Stephan Paul2

Miroslav Virius1

1Faculty of Nuclear Sciences and Physical EngineeringCzech Technical University in Prague

2Physik-DepartmentTechnische Universität München

Conference “Tvorba softwaru 2012”24th May 2012, Ostrava

∗[email protected]

Vladimí Jarý et al. Software development for the COMPASS experiment

Overview

1 IntroductionCOMPASS experiment

2 Current DAQ systemArchitecture of the systemDATE package

3 Control and monitoring software for a new DAQ systemMotivation and requirementsOverview of the hardware architectureLayers of the DAQ softwareImplementation detailsPerformance tests

4 Conclusion and outlook


COMPASS experiment

COMPASS: Common muon and proton apparatus forstructure and spectroscopyexperiment with a fixed target situated on the Super ProtonSynchrotron particle accelerator at CERN, [1]scientific program approved in 1997 by CERN

experiments with hadron beam (glueballs, Primakoffscattering, charmed hadrons,. . . )experiments with muon beam (gluon contribution to thenucleon spin, transverse spin structure of nucleons,. . . )multiple types of polarized target

data taking started in 2002plans at least until 2016 as COMPASS-II

3 programs: GPDs, Drell-Yan, Primakoff scattering

international project: cca 250 physicists from 11 countriesand 29 institutions


COMPASS spectrometer

polarized target on the left, length approximately 50 m

COMPASS spectrometer, image taken from [1]

spectrometer consists of detectors:1 measurement of deposited energy (calorimeters)2 particle identification (RICH, muon filters)3 particle tracking (wire chambers)


Terminology

event: collection of data describing flight and interactionsof particle through the spectrometerroles of the data acquisition system (DAQ):

1 reads data produced by detectors (readout)2 assembles full events from fragment (event building)3 sends events into a permanent storage (data logging)4 enables configuration, control, and monitoring (run control)5 preprocesses and filters data (e.g. track reconstruction,

online filter)

trigger : selects physically interesting events (or refusesnoninteresting events) in a high rate environment withminimal latencytrigger efficiency : ε = Ngood(selected)/Ngood(produced) < 1DAQ deadtime: D = timesystem is busy/timetotal

when system is busy, it cannot accept any other triggerwhich leads to loss of data


Overview of the TDAQ system

Structure of the trigger and data acquisition system according to [4]


Current DAQ architecture

influenced by the cycle of the SPS particle accelerator:12 s of accelaration, 4.8 s of extraction (spill/burst)key aspects: multiple layers, parelelism, buffering

1 detector (frontend) electronics:preamplify, digitize data250000 data channels

2 concentrator modules (CATCH, GeSiCA):perform readout (triggered by the Trigger Control System)append subevent header

3 readout buffers: buffering subevents in spillbuffer PCI cardsmakes use of the SPS cycle to reduce data rate to 1/3 of theonspill rate, roughly stable data rate on the output(derandomization)

4 event builders:assemble full events from subeventssend full events to the permanent storage, storemetainformation about events into the Oracle DBadditional tasks (online filter, data quality monitoring)


Current DAQ software

based on the ALICE DATE package[2]DATE distinguishes two kinds of processors:

1 local data concentrators (LDCs)perform readout of subevents, correspond to readout buffers

2 global data collectors (GDCs)perform event building, correspond to event builders

requirements on the nodes:1 all nodes must be x86 compatible2 all nodes must be powered by GNU/Linux OS3 all nodes must be connected to the network supporting the

TCP/IP stackflexible system (fixed targer mode × collider mode)scalable system (full scale LHC experiment × smalllaboratory system with one processor)performance:

40 GB/s readout2.5 GB/s event building1.25 GB/s storage


Functionality

DATE provides:1 readout, data flow2 event building3 run control4 interactive configuration (based on the MySQL database)5 event monitoring (COOOL)6 data quality monitoring (MurphyTV )7 information reporting (infoLogger, infoBrowser )8 online filter (Cinderella)9 load balancing (EDM, optional)

10 log book11 . . .


Problems with existing DAQ system

Motivation260 TB recorded during the 2002 Run, 508 TB during the2004 Run, more than 2 PB during the 2010 Runincreasing number of detectors and detector channels,trigger rate⇒ increasing data ratesaging of the hardware⇒ increasing failure rate of hardwarePCI technology deprecated

Main idea of the new systemreplace ROBs and EVBs by custom FPGA-based HWhardware based data flow control and event buildingsmaller number of components, higher reliability


Overview of the hardware architecture

frontend electronics and concentrator modules unchangedreadout buffers and event builders replaced with customhardware:

Field Programmable Gate Array (FPGA) technologyFPGA card designed as a module for AdvancedTelecommunications Computing Architecture (ATCA) carriercard, in total 8 carrier cards:

6 for data multiplexing2 for event buildingeach carrier card equipped with 4 FPGA modulesdifferent functionality, same firmware

FPGA card equipped with 4 GB of RAM, 16 serial links(bandwidth 3.25 GB/s)softcore processor on cards for powering control andmonitoring software, communication based on Ethernet

ROBs and EVBs will be used for computing farm


Hardware architecture


Requirements analysis

Requirements:distributed system, communication based on TCP/IPcompatibility with Detector Control Systemcompatibility with software for physical analysisremote control and monitoringmultiple user rolesreal time not required

Decisions:use the DIM library for communicationdo not use the DATE packagepossibly reuse some DATE components (COOOL,MurphyTV )keep data format unchanged


Software architecture

Roles participating in the control and monitoring software


Roles participating in the software

1 Master processcontrols slave processesreceives commands from GUIauthenticate and authorize usersreads and writes configuration to online database

2 Slave processesmonitor and control the hardwarereceive configuration information and commands from themaster process

3 GUIreceives information about health of the system from themaster processsends commands to the master process that distributesthese commands to the slave processes

4 Message logger: collects messages produced by otherprocesses and stores them into database

5 Message browser: displays the messages produced byother process


Implementation details

communication between nodes based on the DIM libraryimplementation in Qt framework

slave processes implemented in C++ language, without Qt

scripting in Python (e.g. starting of the slave processes)MySQL database (compatibility with Detector ControlSystem and DATE)complex system⇒ describe behavior of the master andthe slave processes by state machines


State machines

State machine describing behavior of the master process


DIM Library[3]

developed for the DELPHI experiment at CERNasynchronous one-to-many communication inheterogenous network environment [3]based on the TCP/IPinterfaces to C, C++, Python, Java languagescommunication between servers (publishers) and clients(subscribers) through DIM Name Server (DNS)types of messages:

services updated at regular intervalsservices updated on demandcommands


DIM Name Server

Position of the DIM Name Server


Evaluation of the system

Test scenario:number of nodes: 2 - 16message size: 100 B - 500 kBCOMPASS internal network during winter shutdown (GigabitEthernet)standard x86 compatible hardware (event builders)

Tests performed:performance

is system able to update information about status ofhardware every 100 ms?

stability


Results of the performance tests

Transfer speed as a function of size of the message


Results of the stability tests

Stability of the software in time


Summary and outlook

1 Analysis of the existing data acquisition systembased on the DATE packagescalability issues, deprecated technologies (PCI bus)

2 Development of control and monitoring software for newDAQ architecture

analysis of requirements on softwaredescription of the hardware architecturedefinition of roles and behavior of the systemimplementationperformance tests

3 Goals:to test system on the real hardwareto have fully functional system in 2013to deploy the system in 2014


The bibliography

P. Abbon et al.: The COMPASS experiment at CERN, In:Nucl. Instrum. Methods Phys. Res., A 577, 3 (2007) pp.455–518. See also the COMPASS homepage athttp://wwwcompass.cern.ch

T. Anticic et al. (ALICE DAQ Project): ALICE DAQ and ECSUser’s Guide, CERN EDMS 616039, January 2006

C. Gaspar: Distributed Information Management System[online]. 2011. Available at: http://dim.web.cern.ch

W. Vandeli: Introduction to Data Acquisition, In: InternationSchool of Trigger and Data Acquisition, Roma, February2011

Acknowledgement

This work has been supported by the MŠMT grants LA08015and SGS 11/167.


http://wwwcompass.cern.ch

http://dim.web.cern.ch

software development for the compass experiment

Documents