roc-based daq: latest developments and perspectives

DABC

ROC-based DAQ: latest developments and perspectives

Jörn Adamczewski-Musch, Hans G. Essel, Sergey LinevGSI, Experiment Electronics: Data Processing group

• Optics• ROClib version 2• Preparation for beamtime in June 2010• BNet demonstrator• Throttling

13.04.2010 S.Linev ROC-based DAQ: status and perspectives 2

DABCROClib - status end 2009

• Only nXYTER frontend was existing and supported• Connection between ROC and PC via Ethernet• ROC GUI was disabled• rocutil was the only tool for configuration of the ROC• DABC DAQ application for readout of several ROCs• go4monitor for simple online/offline analysis of data, taken during beamtime

• Worked reliable during beamtime 2009


DABCOptic developments

• Three kind of traffic in one media:– data– control– DLM

• Required extra hardware:– AVNet board

• Required extra software (included in ROClib):– PCI driver (root account)– mprace library– abbdaemon application

• Lot of tests last several month:– stable work in fixed configuration, bis 250 MB/s datarate– problems with simultaneous usage of both ports– already supported in ROClib, some improvements required


DABCNew functionality in ROClib v2.0

• Goal for v 2.0 – optics as mainstream, Ethernet as fallback solution

• New functionality in ROC FPGA code:– support of new frontend (FEET)– programmable commands lists– new message format (little endian)– reorganization of control address space (breaks compatibility)

• On the PC side:– support of the new FPGA features– device/transport classes for optic– new roc::Message format – support both old and new formats– roc::Iterator class for access data from different data sources– rocGui for ROC/nXYTER/FEET configurations– go4monitor can be connected directly to the ROC (without DAQ app)


DABCNew roc::Message and roc::Iterator classes

• roc::Message is replacement for old nxyter::Data class• Supports both old (v1.x) and new (v2.x) message formats

• roc::Iterator class allows to get data:– from ROC via Ethernet (via roc::Board class)– from ROC via Optic (also via roc::Board class)– from LMD files (replacement for fileapi)

roc::Board* brd = roc::Board::Connect(“optic://abb0”);brd->startDaq();roc::Iterator iter(brd);int cnt = 0;while (iter.next() && (cnt++<1000)) { iter.msg().printData();}brd->stopDat();roc::Board::Close(brd);


DABCrocGui

• Qt4-based

• Only for configurations– DAQ with go4monitor

• Can configure:– ROC itself– nXYTER frontend– FEET frontend

• Works with optic and Ethernet

• Able to produce scriptsfor rocutil

• Modular design, easy to extend


DABCMigration from v1.x to 2.0

• Version 2.0 is not backward compatible with 1.x

• Old LMD files are supported in v2.0

• Final version 2.0 will be announced together with exact instruction for upgrade

• Exact sequence to upgrade firmware/software on the ROC:– checkout new ROClib in separate location;– with old rocupload program update first firmware (not forget jumpers):

• rocupload cbmtest01 –fw ~/roclib2/firmware/Release/FEET02000004_ETHERNET02000002.bit

– with old rocupload program update PowerPC software:• rocupload cbmtest01 –sw ~/roclib2/sw-host/release/image.bin

– reboot ROC– compile new ROClib and try to use it with the ROC

• rocGui cbmtest01

– For more info see README.txt and cbm-wiki.gsi.de


DABCBeamtime - plans end 2008

Datacombiner

board

SFPSFP

SFP

SFP

SFP

SFP

FEB4nx SiCBM01B2

ROC

EthA

BSYNC-S

AUX

FEB4nx SiCBM01B2

ROC

EthA

BSYNC-S

AUX

FEB4nx SiCBM01B2

ROC

SFPA

B AUX

FEB1nxGen

BEAMAUX...

ROC

SFPA

B AUX

ROC

SFPA

BSYNC-S

AUX

FEB4nxSiDem1

ROC

SFPA

BSYNC-S

AUX

FEB4nxSiDem1

ROC

SFPA

BSYNC-S

AUX

FEB4nxSiDem1

ROC

SFPA

B AUX

FEB4nxSiDem1CBM

STSDemo 1

ROC

SFPA

BSYNC-S

AUX

FEB4nxSiDem1

ROC

SFPA

BSYNC-S

AUX

FEB4nxSiDem1

ROC

SFPA

BSYNC-S

AUX

FEB4nxSiDem1

ROC

SFPA

B AUX

FEB4nxSiDem1CBM

STSDemo 1

Datacombiner

board

SFPSFP

SFP

SFP

SFP

SFP

Datacombiner

board

SFPSFP

SFP

SFP

SFP

SFP

Eth

MBS

DABC node1

SFP/PCIe

ReadoutEvent building

FilteringFile store

IB/PCIe

DABC node2

SFP/PCIe


FilteringFile store

IB/PCIe

DABC node3

SFP/PCIe


FilteringFile store

IB/PCIe

DABC node4

Eth


FilteringFile store

IB/PCIe

Infin

iBan

d

swit

ch


DABCBeamtime in June 2010 – realistic plan

FEB4nx SiCBM01B2

ROC

EthA

BSYNC-S

AUX

FEB4nx SiCBM01B2

ROC

EthA

BSYNC-S

AUX

FEB4nx SiCBM01B2

ROC

SFPA

BSyncS

AUX

FEB1nxGen

BEAMAUX...

ROC

SFPA

BSyncM

AUX

ROC

SFPA

BSYNC-S

AUX

ROC

SFPA

BSyncS

AUX

ROC

SFPA

BSYNC-S

AUX

ROC

SFPA

BSyncS

AUX

DABC node1

SFP/PCIe


FilteringFile store

IB/PCIe

DABC node2

SFP/PCIe


FilteringFile store

IB/PCIe

DABC node3

SFP/PCIe


FilteringFile store

IB/PCIe

Infin

iBan

d

swit

ch

DABC node4

SFP/PCIe


FilteringFile store

IB/PCIe

Any other frontend


DABCBeamtime in June 2010

• Necessary components:– Up to 8 ROCs, 4 ABBs– SyncM/SyncS cabling– 4x Opteron-IB DAQ cluster with 4 TB storage– DABC with ROC-ABB and ROC-BNet plugin– Several PCs for monitoring

• Still to do:– reliable work of ABB with both SFPs– modifications in ROC-ABB plugin– Intensive ROC-BNet tests – can be done in next two months

• Fallback solution (if optic fails)– Same electronics, but readout via Ethernet– Reduced datarates– With or without BNet


DABCCBM DAQ perspectives - dataflow

FEE

Readout

Buffering

BNet

FLES

SLES, storage

Switch ~1 TB/s of raw data over networkProvides combined time slices to the FLES

Throttle data in case of buffers/networks overflow


DABCBNet simulations with SystemC (2005)


DABCInfiniBand performance tests (2006-2007)


DABCDABC development (since 2007)

• General purpose DAQ framework

• Data-flow core– threads, device & transports– user modules, application

• Configuration, control, monitoring, GUI

• Provides components for custom event building network - BNet

• Real data sources – MBS, ROC/UDP, ROC/Optic, generic PCIe board driver

• Used as DAQ for CBM 2008/2009 beamtimes

• Mid- and short-term plans:– Improved command interface (deadlocks recognition)– Support of other kinds of control system (EPICS)– Connection to cluster/job management system


DABCBNet demonstrator

• Since 2007:– better understanding of traffic/datarates– more realistic trigger algorithms were proposed– DABC was established

• One can launch activity for new BNet demonstrator:– with more realistic traffic generators– with ~100-200 nodes InfiniBand cluster (~20% of final CBM)– use DABC-based infrastructure– emulation of FLES activity (CPU consumption)

• Aim of demonstrator:– where InfiniBand technology now, that is near perspectives?– that kind of traffic shaping we need, if any?– is time-scheduled transport gives us significant benefits?– reliability and errors recovery issues?


DABCCBM DAQ perspectives - throttling

FEE

Readout

Buffering

BNet

FLES

SLES, storage

No backpressure until BNet.

How to throttle data when data rates higher than output link performance?


DABCThrottling algorithm

• While all messages are time-stamped, one can provide very simple throttle algorithm, based on absolute epoch number. For instance:

– drop every 8-th epoch, if not enough– drop every 4-th epoch, if not enough– drop every 2-nd epoch, if not enough– drop everything, but first epoch

• Any other dropping pattern can be implemented

• Algorithm required memory for at least last 8 epoch– no problems in readout buffer or BNet

• What to do in FEE, where buffer capacity is limited?

E4E3E2E1 E8E7E6E5


DABCThrottling in FEE

• Doing same as in readout buffers and Bnet – requires large buffer space and complex arbitration logic, implemented in hardware => too complicated

• As proposed by W. Müller on last DAQ meeting, set busy bit (flip-flop) if FIFO full and reject any new data until next epoch. Efficiency of such algorithm is highly depends from relation between FIFO buffer space, epoch length (E) and output link performance (RATE):

– can reasonably work only when FIFO << E * RATE, but potentially high data lost – if FIFO ≈ E * RATE, it is about 50% probability if data of even or odd epoch will be

excluded by the FEE chip => completely unusable– if FIFO >> E * RATE, id of dropped epochs will vary from chip to chip, no any real

correlation, also unusable

• The only way to drop data in the FEE – introduce message counter per epoch. If amount of data in current epoch exceeds limit, all consequent messages in current epoch will be dropped. Guarantees, that at least in the epochs begin all data are preserved. Open question – per-channel or per-chip logic?

roc-based daq: latest developments and perspectives

Documents