9th Workshop on Electronics for LHC Experiments 1CERN/EP-CMD

FEDkit: a design reference for CMS data acquisition inputs

V. Brigljevica, G. Brunoa, E. Canoa, S. Cittolina, S. Erhanb, D. Gigia, F. Glegea, R. Gomez-Reino Garridoa, M. Gulminia, J. Gutlebera, C. Jacobsa,

M. Kozlovszkya, H. Larsena, I. Magrans De Abrila, F. Meijersa, E. Meschia, S. Murraya, A. Oha, L. Orsinia, L. Polleta, A. Racza, D. Samyna,

P. Scharff-Hansena, C. Schwicka, P. Sphicasa,c, J. Varelaa

aCERN, Geneva, SwitzerlandbUniversity of California, Los Angeles, USA

cUniversity of Athens, Athens, Greece

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 2

Topics

Introduction S-LINK64 protocol and frame format FEDkit

– Hardware architecture

– Software architecture

– Environment

– Performance Next step for final CMS DAQ system: FRL

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 3

CMS data acquisition (DAQ)

Several sub-detectors with various technologies ~640 inputs to readout systems of DAQ Throughput requirements: 200 MB/s average, bursts of

400MB/s Common interface based on S-LINK64 specification

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 4

S-LINK64 interface

Extension of already defined standard from CERN (S-LINK)

Recommends Common Mezzanine Card (CMC) format

Simple, FIFO-like interface from mother board to link daughter board

64-bit wide, up to 100MHz Flow control included

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 5

Data frames

Encapsulation of the FED-specific data in common data format

Header contains identification of fragment, some statuses Payload protected by CRC

16 15

8 7

12 1132 31 4 3 060 5963 56 55

60 5963

20 19 4 3 0

032 31

Evt_stat(8)

BOE_1K LV1_id (24) BX_id (12)

CRC (16)

D Sub-detector payload

K

D Sub-detector payload

32 3160 5963

Evt_ty

56 55

EOE_1 Evt_lgth (24)xxxx

Source_id (10+2) FOV

xxxx

$$Hx

1

BOE_2K $$

$$Tx

60 5963 032 31

EOE_2K $$

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 6

Generic III & Daughter boards

Generic III – PCI board with S-LINK64

connector embedded

– Can be used as receiver (FEDkit) or sender (FED emulator)

Sender and receiver daughter boards communicate with LVDS cable (max 17m for 480MB/s cable speed)

Sender implements the S-LINK64 protocol for FED

Receiver board plugs into GIII

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 7

Hardware architecture

Sender board– FPGA implements S-LINK64

protocol and controls LVDS chips

Receiver daughter board – LVDS chips and 32kB FIFO per

link GIII board

– FPGA used to control reception FIFO and PCI 64-bit/66 MHz

– Implements protocol with host PC

– Hardware CRC check on the fly

– Hardware link test mode

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 8

Software architecture

Buffer loaning scheme Parallel operation of DMA, block handling and

fragment arrival notification

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 9

Software architecture (cont.)

Software provided– Linux driver

– FEDkit library

– Example applications, application for XDAQ, the data acquisition framework used in CMS DAQ

OS bypass for performance FEDkit library API (in C) matches high level

functionality (simple to use) Fragment handling functions provided

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 10

FEDkit environment

FEDkit requires a PC with PCI 64bit 3.3V bus Only OS supported is GNU/Linux Can read FED data up to LVDS wire speed (480 MB/s) Tested in PentiumIII, Xeon, Athlon machines Fragment sizes up to 16MB Hardware verification of CRC in receiver Provided to FED developers Successfully used in test beams (with FED emulator as

source)

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 11

Performance

Full LVDS wire speed always achievable

On Xeon more that twice the bandwidth of requirements

Software overhead hidden by DMA as fragments get bigger

Software overhead – on PIII: 5.3µs

– On Xeon: 3.9µs Hardware overhead (never masked)

– 579 ns per fragment

– 182 ns per block

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 12

Final system design: FRL

FRL is the evolution of the FEDkit for the final system

Reuses a lot of FEDkit architecture

(Almost) same sender board, cable

Sends to intelligent NIC instead of host PC’s memory

Compact PCI form factor

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 13

FRL architecture

LVDS link part very close to FEDkit’s one

Main FPGA uses similar protocol with intelligent network interface card (NIC)

Bridge FPGA allows control from host PC and spying (sampling of fragments)

Local DAQ through cPCI without full DAQ (but ~1kHz) for detector commissioning

Main FPGA used as data source for DAQ commissioning without FED data

9th Workshop on Electronics for LHC ExperimentsCERN/EP-CMD 14

Conclusion

FEDkit implements the CMS DAQ readout functionality for lab tests– Realistic performance

– Simple, direct access to fragments with provided library

Plug and play replacement in the final DAQ system with FRL

http://cmsdoc.cern.ch/cms/TRIDAS/html/Documents.html