jin huang (bnl) thanks to discussions from nils · 02/04/2018 · sphenix use a next-generation...
TRANSCRIPT
![Page 1: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/1.jpg)
Jin Huang (BNL)Thanks to discussions from Nils
![Page 2: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/2.jpg)
Versatility of EIC event topology calls for trigger-less DAQ◦ 0.5 MHz interaction at top luminosity
Start using sPHENIX-EIC full detector simulation to estimate trigger-less DAQ◦ Total data rate on order of 100 Gbps
Match well with sPHENIX TPC/MVTX DAQ through-put rate◦ Similar architecture with ATLAS/LHCb/ALICE DAQ
upgrade in 2020+
Leverage streaming readout and EIC-related detector R&D at sPHENIX
Cold QCD TG meetingJin Huang <[email protected]> 2
![Page 3: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/3.jpg)
I was asked to give a presentation in EIC stream readout meeting about PHENIX/sPHENIX experience
Taking the opportunity to look into a possible stream readout for the EIC detector based on sPHENIX. ◦ What is the data volume? ◦ Reuse parts of sPHENIX DAQ?◦ Event building or stream readout?
Event building is not easy since eRHIC has 10ns crossing and BBC+calorimeter do not cover all interested cross section
Cold QCD TG meetingJin Huang <[email protected]> 3
![Page 4: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/4.jpg)
sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA ASIC : 20 Mhz 10-bit ADC , hit wavelet forming in chip 3.6 Tbps bi-direction fiber link to counting house
Cold QCD TG meetingJin Huang <[email protected]>
Buffer box
FEE data stream: 600 6.6+ Gbps bidirectional fibers linksMax continuous: 4 Gbps / fiberAverage continuous: 1.6 Gbps x 600 fibers
TPC DAM L3 Scope, 1.2.6
Output data stream to buffer box: 24 x 10 Gbps Ethernet via fiberAfter FPGA based triggering and clustering
Total continuous output to disk: 80 Gbps
DAQ L2 ScopeWBS 1.6
Clock/Trigger input: Optical linksClock = 9.4 MHzTrigger Rate = 15 kHz
FEE L3 Scope1.2.5
Structure/GEM L3 Scopes
WBS 1.2.1-4
Transfer to RCF continuously
150k chan
600 FEEs
24 sectors
Continuous
readout
4
![Page 5: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/5.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 5
ATLAS DAQ structure for mid-2020s:
• Using high performance large-FPGA optical link cardto bridge custom FEE and commodity computing
• Besides ATLAS, similar architecture also proposed for ALICE and LHCb upgrades
• 48 x 10 Gbps bi-directional optical link to FEE• Kintex7 Ultrascale FPGA• 100 Gbps PCIe Gen3 x16 link to CPU
Front-End Link eXchange (FELIX)FPGA-based PCI express card
FELIX 2.0 card in pre-production
Currently used in sPHENIX test stands
• You can also consider it as a 48-link network switchrunning specialized network protocol for 1 FEE/linkData reduction on FPGA, optimized for cost / link
![Page 6: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/6.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 6
BNL EIC taskforce studieshttps://wiki.bnl.gov/eic/index.php/Detector_Design_Requirements
Considering the case of 20+250 GeV e+p collisions at luminosity of 1034 cm-2 s-1
◦ 100x p+p luminosity
Cross section: 54 µb◦ Based on EIC task-force simulation
/gpfs/mnt/gpfs04/sphenix/user/nfeege/data/copy_eic_data_PYTHIA_ep/TREES/pythia.ep.20x250.1Mevents.RadCor=0.root
◦ 0.1% of p+p cross section
Collision rate: 500 kHz◦ 10% of p+p collision rate
Multiplicity: 1 /dη @ η=0◦ Fraction of p+p multiplicity
Total particle rate: a few M particle /s◦ sPHENIX AuAu records 30M particle /s
sPHENIX DAQ bandwidth may be able to accommodate full EIC data without triggering (full stream readout)◦ Event forming and reconstruction either in
online / offline computing farm
![Page 8: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/8.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 8
Multiplicity check for all particlesMinimal bias Pythia6 e+p 20 GeV + 250 GeV53 µb cross section
BNL EIC taskforce studieshttps://wiki.bnl.gov/eic/index.php/Detector_Design_Requirements
Based on EIC task-force pythia set /gpfs/mnt/gpfs04/sphenix/user/nfeege/data/copy_eic_data_P
YTHIA_ep/TREES/pythia.ep.20x250.1Mevents.RadCor=0.root
![Page 9: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/9.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 9
Raw data: 16 bit / MAPS hit
Raw data: 3x5 10 bit / TPC hit + headers (60 bits)
Raw data: 3x5 10 bit / GEM hit + headers (60 bits)
![Page 10: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/10.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 10
Raw data: 31x 14 bit / active tower +padding + headers ~ 512 bits / active tower
![Page 11: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/11.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 11
Raw data: 31x 14 bit / active tower +padding + headers ~ 512 bits / active tower
![Page 12: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/12.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 12
Obviously this is very preliminary, and evaluated at top lumi 1034 cm-2 s-1: Tracker+ calorimeter FEE output signal only = 32 Gbps, with header ~40 Gbps
◦ minimal account to bring out of tracker+ calo FEE via fiber ◦ No noise, and no PID detectors
FEE output with headers ~ 40Gbps * 1.5(+PID) * 2 (noise) ~ order of 100 Gbps ◦ A earlier estimate by BNL EIC taskforce BeAST detector:
Record 2.5M track / s → 160 Gbps (T. Ljubicic). In the same ball park◦ sPHENIX TPC accommodate a 4 Tbps FEE -> counting house traffic
sPHENIX TPC/MAPS DAQ is designed to record 100 Gbps @ 30M track /s The sPHENIX DAQ may serve as foundation for stream readout in EIC, that record all
detector hit at EIC without need of event building
![Page 13: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/13.jpg)
EIC detector’s data rate for all collisions (~100 Gbps) seem fit into the sPHENIX DAQ bandwidth for disk recording◦ Online/offline computing farm perform collision ID, event
association and reconstruction
4 Tbps (FEE-link) / 100 Gbps (disk) DAQ for sPHENIX TPC/MAPS can provide a solid foundation that accommodate stream readout of an EIC detector◦ On-going: advanced R&D and beam tests◦ FEE -> fiber -> PCIe stream readout for TPC&MAPS FEE,
consistent with ATLAS/ALICE/LHCb in 2020+
May become a section in the new LOI
Cold QCD TG meetingJin Huang <[email protected]> 13
![Page 14: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/14.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 14
![Page 15: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/15.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 15
sPHENIX use a next-generation gate-less TPC for outer central tracking, Require stream readout FEE by design
SAMPA ASIC : 20 MHz 10-bit ADC , hit wavelet forming in chip. 3.6 Tbps bi-direction fiber link to counting house
Triggered readout for sPHENIX (save 75% of physics data size), Plan to also support full stream readout with firmware switch on FELIX FPGA (for commissioning / testing)
![Page 16: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/16.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 16
sPHENIX calorimeter: T1044
INTT: T1439 sPHENIX Si strip test
MVTX: T1441 sPHENIX MAPS test
TPC prototype being prepared tooMAPS Online monitoring with 4 staves in beamPreliminary alignment, ~O(100um)
![Page 17: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/17.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 17
100kHz collision in continuous DAQ triggerIn TPC DAQ simulation
FEE -> DAM limit : 6 Gbps x 8b/10b per FEEReference design rate: 1.9 Gbps, far lower than limitMax rate: 200kHz + 48 rings → max 7.2 Gbps @ module 1
― All collisions― Triggered collisions
![Page 18: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/18.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 18
Collect data from 600 bi-directional 4+ Gbps fiber links to FEEs at rate of 940 Gbps
Reducing data via triggering, clustering and compression to reduce data to ~80 Gbps
Driven to use large-FPGA based data acquisition cards with multi-10Gbps input and PCIe Gen3 output, hosted on commodity servers.
◦ Similar architecture adopted for ATLAS, LHCb, ALICE upgrade for 2020+
Default implementation taking advantage of BNL-developed FELIX PCIe card
◦ 48-bidirectional 10 Gbps optical link, large FPGA: Xlinx Ultrascale (XCKU115)
◦ PCIe Gen3 x16 link to server, demonstrated to 101 Gbps (require ~10 Gbps)
ServerFELIX v1.5 card FELIX in serverMTP <-> LC BreakoutFEE Prototype
![Page 19: Jin Huang (BNL) Thanks to discussions from Nils · 02/04/2018 · sPHENIX use a next-generation gate-less TPC for outer central tracking Require stream readout FEE by design SAMPA](https://reader034.vdocuments.site/reader034/viewer/2022050217/5f630c89170be0039e29e394/html5/thumbnails/19.jpg)
Cold QCD TG meetingJin Huang <[email protected]> 19
Online display
PHENIX event builder/ Data storage
Standalone data(calibration, etc.)
FPHX Chip
Sensor
HDI
Ionizing Hit
IR
DA
Q R
oom
768 fibers
1.9 Tb/s
17k LVDS
3.2 Tb/s
48 fibers
~8kHz trigger
8 fibers
Data cable/bandwidth shown on this slide only
PHENIX Timing
Timing / Trigger
Stream
Triggered d
ata to d
isks