leb 2002 @ colmar cms daq overview data to surface (d2s) requirements topology hardware...
Post on 19-Dec-2015
217 views
TRANSCRIPT
LEB 2002 @ ColmarLEB 2002 @ Colmar
•CMS DAQ overview•Data to Surface (D2S)
•Requirements•Topology•Hardware implementation•Measurements and simulations•Readout kit for FE designers•DAQ staging policy
•Conclusion
CMS Data to SurfaceCMS Data to Surfacetransportation transportation architecturearchitecture
[email protected] behalf of the CMS DAQ group
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 2
1/8th DAQ Readout Builder:Lv-1 Max. trigger rate 12.5 kHzRU Builder (64x64) .125 Terabit/sFB fragment <size> ≈ 16 kBRU-BU systems 64Event filter power ≈ .7 TeraFlopEvent flow control ≈ 105 Mssg/sLocal mass storage 10 TByte
Data to surface:Average event size ≈ 1 MbyteFED S-link64 ports 658DAQ links (5 Gb/s) 512FED fragment <size> ≈ 2 kBFED builders (8x8) 64
TTC Timing, Trigger and Control TPD Trigger Primitive DataaTTS asynchronous Trigger Throttle SystemD2S Data to SurfaceFRL Frontend Readout LinkRU Readout UnitBU Builder UnitFU Filter UnitFFN Filter Farm NetworkEVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkDCN Detector Control networkDSN DAQ Service Network
8-fold DAQ architecture8-fold DAQ architecture
DAQ staging
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 3
D2S requirementsD2S requirements
• Reads the data from the FE data sources (~630 sources)
• Merging of 2 “small” sources (nominal size is 2 KBytes)
• Leveling effect on un-balanced data sources
• Staging capability according to the 8-fold DAQ architecture
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 4
Data to Surface (D2S)Data to Surface (D2S)
Short link(20m)
Long link(200m)
Underground area
Surface area
FED/DCC
FRL
FED Builder
RU
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 5
Data sourcesData sources
Detector Number of data sources
Number of DAQ links
Event size per link
KB
Fluctuations RMS
Nominal event
size KB (pp run)
Number of
trigger partition
s
Pixel 40 36 (merg) 1.2 – 2.0 30% 72 2
Tracker 440 272 (merg) 0.4 – 1.5 0.27 KB 300 4
Preshower
47 47 2.3 ? 110 2
ECAL 52 (DCC) 52 2 1 KB ~100 4
HCAL 24 (DCC) 24 1.7 ? 48 6
Muons CSC
9 (DCC 4-1)
9 2 Huge… 16* 2
Muons RPC
5 (DCC) 5 .3 ? 1.5 4
Muons DT 5 5 1.6 ? 8 2
Glob. trig 4 4 2 none 8 3
DT track f.
4 4 2 none 8 2
Total 630 458 671.5 31
* Most of the time, the detector is empty: one muon produces 5.5KB
These numbers are currently revised/updated and they will change !Event sizes are strongly dependent on simulated occupancies…
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 6
D2S in vivo…D2S in vivo…
From one end of the DAQ buildingto the end of the control room,thecable path is 187 m long and there are~90 m of verticality
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 7
Data sources location (1)Data sources location (1)
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 8
Data sources location (2)Data sources location (2)
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 9
S-L
lin
k64
to
FR
L M
erg
er
FE
D B
uil
de
rR
ead
ou
t U
nit
In
pu
t
D2S baseline implementationD2S baseline implementationLVDS + Custom FRL+MyrinetMyrinet
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 10
Short distance linkShort distance link
• LVDS signals, S-link64 compliant– 869 MB/sec @ 10m– 480 MB/sec @ 17m
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 11
AlteraACEX
60MHz
FED
FED
AlteraACEX
60MHz
FIFO32kB
FIFO32kB
FRL
S-Link64 -64-bit up to 100MHz -Backpressure -auto-test (Slink spec.) -12 reserved pins
LVDS transfer -64-bit @ 60MHz (480MB/s) -cable length 15m max.
ALTERAStratrix
-no Slink pinout-64-bit PCI clock-able to read one port at the time
FRL merging featureFRL merging feature
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 12
FRL baseline implementationFRL baseline implementation
PCI Myrinet board
160 mm 30 mmStandard size
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 13
DAQ Opticable: a candidateDAQ Opticable: a candidate
•200m long•72 fibers, 50/125um multimode, LC-LC connector•32 DAQ links (4 spares) per cable,•16 cables for CMS DAQ
Polyester Tape
Aramid Yarn
PVC Coated Aramid C.S.M.
Aramid Yarn
Tight Buffer Optical Fiber
PVC Sub-unit Jacket *
Sub-unit
HFFR Jacket *
Rip Cord.
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 14
0
10
20
30
40
50
60
70
80
90
100
0 0.25 0.5 0.75 1 1.25
RMS/average
Max
imu
m U
tilis
atio
n (
%)
EVB traffic
FED builder performances FED builder performances (Myrinet)(Myrinet)
• rms=0 (fixed size): puts itself in Barrel Shifter • variable size and no traffic shaping: ~55% usage
RU1
8x8 crossbar
RU2 RU3 RU4
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 15
0
10
20
30
40
50
60
70
80
90
100
0 0.25 0.5 0.75 1 1.25
RMS/average
Max
imu
m U
tilis
atio
n (
%)
balanced (2.0)
unbalance ratio=2 (1.33 2.66)
unbalance ratio=3 (1.0 3.0)
FED Builder - Unbalanced Inputs FED Builder - Unbalanced Inputs
•Maximum usage: roughly 50%•No significant loss due to un-balanced sources
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 16
• Myrinet Lanai9 (1 link @ 2.5 Gbit/sec), jumbo event (200k)
Time
Event
ID
TimeEvent
ID
no MTU with MTU
““Jumbo” fragment effect…(1)Jumbo” fragment effect…(1)
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 17
• Myrinet Lanai10 (2 links @ 2.5 Gbit/sec), jumbo event (200k)
Time
Event
ID
TimeEvent
ID
no MTU with MTU
““Jumbo” fragment effect…(2)Jumbo” fragment effect…(2)
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 18
Readout Kit for FED designersReadout Kit for FED designers
• PCI generic III (see D.Gigi’s talk), Tx-card (PMC form factor) + Rx-card, cable (15m max.)
• Software libraries and drivers under Linux
• Supported by X-DAQ (Common DAQ software framework)
• Documented !!!
• WEB download: http: //cern.ch/cano/fedkit/
• Provides an easy transition from local readout to central DAQ readout
• Price: ~ 2500 FS
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 19
DAQ staging policyDAQ staging policy
Staging
FED/DCC
FRL
FED Builder
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
FED/DCC
FRL
1 RU builder 64x64, 1MByte @ 12.5KHz
RURU
2 RU builders 64x64, 1MByte @ 25KHz3 RU builders 64x64, 1MByte @ 37.5KHz
RU8 RU builders 64x64, 1MByte @ 100KHz
RURURURURU
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 20
ConclusionConclusion
• D2S structure and implementation well defined and matches the requirements
• Minimal custom electronics developments– Can profit from the technological improvements
• Unexpected situations can be handled simply by adding devices– Event size increase– Number of sources increase
• D2S pre-building allows a progressive DAQ deployment
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 21
FED/DCC DAQ portFED/DCC DAQ port
SpecificDetector Electronic
S-LINK64 port
LinkFPGA
FED/DCC
Detector links
Storagearea
S-LINK64 port isthe border between FEand DAQ
SetupControlMessages - out-of-sync - failureMonitoringLocal readout
VME HostInterface
Fast signals: (FMM) - busy/ready - overflow warning - out-of-synch
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 22
S-LINK64 featuresS-LINK64 features
• Based on S-LINK
• Specifies 2 connectors but not the physical link in-between
• S-LINK is 32 bits @ 40 MHz max, flow-control, K/D words
• To match CMS needs:
– 64 bit data path, clock speed up to 100 MHz, but link speed is ~500 MB/s
– no other changes to keep compatibility with S-LINK products
• S-LINK64 specs. available at:
– http://cmsdoc.cern.ch/cms/TRIDAS/horizontal/
9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 23
16 15
8 7
12 1132 31 4 3 060 5963 56 55
60 5963
20 19 4 3 0
032 31
Evt_stat(8) xx$$
BOE_1K LV1_id(24) BX_id(12)
CRC (16)
D Sub-detector payload
K
D Sub-detector payload
FED common data formatFED common data format
32 3160 5963
Evt_ty
56 55
EOE Evt_lgth(24)xxxx
Source_id(10+2) FOV
xxxx
$$Hx
1
BOE_2K $$
The FED is generating the above encapsulation.