off-detector & software development of daq system for the eudet calorimeters
Post on 14-Jan-2016
38 Views
Preview:
DESCRIPTION
TRANSCRIPT
Off-detector & Software development of DAQ System for
the EUDET calorimeters
Tao WuOn behalf of CALICE-UK Collaboration
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 2
EUDET DAQ Targets
Maximizing to use of off-the-shelf commercial components, cheap, scalable and maintainable
Provide well-defined interfaces between DAQ components to allow for minimizing costs and development cycles;
A control system to easily integrate the rest of sub-systems of detectors
Software to build events from bunch train data and disparate sources into single event data
Manage network and data storage
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 3
DAQ Mechanical Design targets
Targeting EUDET Module and ILC/CALICE; Aim to develop a generic system e.g. ECAL/HCAL Triggerless DAQ system: use C&C instead
All data are sent off detector within a bunch train Use bunch struct. as advantages: 1ms in bunch train, rea
d out data within 200ms of inter-train gap;
DAQ system will also control power cycling of readout ASICs
A funnel-like DAQ to collect, wrap and transmit data in stages before sending to central storage
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 4
DAQ Architecture Overview
Slab hosts VFE chips (ASICs)Slab hosts VFE chips (ASICs)
DDetector Interfaceetector Interface connected to Slabs
LDA servicing DIFs
Link/Data AggregatorLink/Data Aggregator
LDAs read out by ODR via opto-links
Off-Detector ReceiverOff-Detector Receiver
PC hosts ODR PCI-Express driver software
Local Software DAQ
Full blown Software DAQ
LDALDA
PCPC
e.g. e.g. ECALECALSlabSlab
DIFDIF
ODRODR
DriverDriver
OptoOpto
OptoOpto
Ccc
- &
Dat
a-lin
k
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 5
DAQ Architecture Hierarchy
LDA
LDAHost PC
PC
Ie
ODR
Host PC
PC
Ie
ODR
DetectorUnit
DIF
C&C
DetectorUnit
DIF
DetectorUnit
DIF
DetectorUnit
DIF
Storage
1-3Gb Fibre50-150 Mbps HDMI
cabling
10-100m0.1-1m
Det
ecto
r
Co
un
tin
g R
oo
m
Covered by this talkCovered by this talk Covered by ValeriaCovered by Valeria
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 6
Current Architecture: ODR
Assemble data into a usable form for processing
Logical tasks: Receive from LDA Process e.g. event building Store
Current input is Ethernet for testing performance Internal FPGA test data generator; External data stream via network
DAQ PCDAQ PC
LDALDA
CC
C-li
nk
ODRODR
Data-link
DIFDIF
AS
ICs
AS
ICs
DIFDIF
AS
ICs
AS
ICs
DIFDIF
AS
ICs
AS
ICs …
Clock /Clock /FastFast
ControlControl
FEFE
ODRODR
StoreStore
Machine clock
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 7
Off Detector Receiver (ODR)
Hardware: Using commercial FPGA dev-board:
PLDA XPressFX100 Xilinx Virtex 4, 8xPCIe, 2x SFP (3
more with expansion board)
SFPs for optic link
Expansion (e.g. 3xSFP)
Receives modularized data from LDA Realized as PCI-Express card, hosted in DAQ PC. 1-4 Opti-links/card (Gigabit), 4 LDAs/card, 1-2 cards/PC Buffers and transfers to store as fast as possible
Sends controls and configs to LDA for distribution to DIFs Performance studies & optimisation on-going
B.G., A.M @ RHUL
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 8
ODR Data Access Rate
transfer the data from ODR memory to the user-program memory via 4 links~700 MByte/secby 25 DMA buffers
B.G., A.M @ RHUL
25 DMA buffers
0
100
200
300
400
500
600
700
800
32 125 130 547 929 1351 2040 3800
Data Size [bytes]
Tra
ns
fer
Ra
te [
MB
/s]
25 DMA buffers
All measurements: single requester thread, no disk write, data copied to the host memory.
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 9
Clock & Control (C&C) board C&C unit provides machine clock and
fast signals to 8x ODR/LDA. A low jitter and fixed latency; Logic control (FPGA, connected via USB) Link Data Aggregator provides next stage
fanout to Detector Interfaces Eg C&C unit 8 LDAs 8 DIFs = 64
DUs. Signalling over same HDMI type cables Facility to generate optical link clock
(~125-250MHz from ~50MHz machine clock)
Board is already designed, will be build soon
LDA
LDA
Host PCHost PC
PC
Ie
ODRODR
C&C
Host PCHost PC
PC
Ie
ODRODR
Machine
Run-Control
M.P., UCL
busy
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 10
EUDET DAQ software: StateState = DeadState = Dead
Transition = PowerUpTransition = PowerUp
suceed failed
State = ReadyState = Ready
State = RunningState = Running
State = ConfiguredState = Configured
State = InBunchTrainState = InBunchTrain
Transition = PowerDownTransition = PowerDown
Transition = StartRunTransition = StartRun Transition = EndRunTransition = EndRun
Trans. = StartConfigurationTrans. = StartConfiguration Trans. = EndConfigurationTrans. = EndConfiguration
Trans. = BunchTrainStartTrans. = BunchTrainStart Trans. = BunchTrainEndTrans. = BunchTrainEnd
Preparing Run number & type
Ending/Finishing
Checks status & communications
Distributes configurations
Starts the data taking
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 11
Transition: StartRun
DAQ DAQ PCPC
DAQ DAQ PCPC
DAQ DAQ PCPC
RCRC FCFC ConfConfDBDB
filefile filefile filefile
read system status
send run number & type
send configurations
filefile filefile
Files to be written for book keeping: system status by DAQ PC run info by RC PC system status by FC
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 12
Data StorageScenario I
DAQ DAQ PCPC locallocal
diskdisk
CentralCentralstorestore
Scenario II
DAQ DAQ PCPC
CentralCentralstorestore
Scenario III
DAQ PCDAQ PCfile in memoryfile in memory
CentralCentralstorestore
RAIDArray
Which scenario to choose depending on the bandwidth with which the data gets produced: (1) up to 200Mbit/sec, (2) up to ~1600Mbit/sec, (3) from there on Estimation of the data rate for the EUDET DAQ prototype has to cope with ~400Mbit/s, however it depends on the detector and the choice of VFE.
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 13
DAQ Software: DOOCS framework
Software development and code base
Computer Infrastructure
Device layer
Middle layer
Communication
Application layer
Sun/Linux Cluster
Software Libs
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 14
DAQ Software of DOOCS
Fast
Linac
Image SlowADCADCADC
ADCADCADC
BMBM
DSDSDS MLML
FCFC SCSC
DAQ server
EVBEVB Operator GUIOperator GUI
Local GUILocal GUIRemote GUIRemote GUI
dCachedCache
DCCPDCCPDCCPDCCPDISK
DISK
RCRC
DBDB
RC GUIRC GUIRC GUIRC GUI
ROOT
storagestoragestoragestorage
Data streams
FC/SC:Fast/SlowCollector
BM:Buffer Manager
EVB:Event Builder
DS:DAQ Server
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 15
Adapting DOOCS to EUDET DAQ
Modeling hardware card via device server Existing: VME, Sedac, Profi-Bus; ODR/LDA/DIF/ASICsODR/LDA/DIF/ASICs
Equipment Name ServerEquipment Name Server (ENS): Facility(F) / Device(D) / Location(L) / Property (P) e.g. CALICE.ECAL/ODR/ODR1/Status
► F: CALICE.ECAL, CALICE.AHCAL, CALICE.DHCAL► D: ODR, LDA, DIF, ASICs;► L: ODR1,ODR2,ODRX; LDA1,LDA2,LDAX; DIF1,DIF2,DIFX;► Property: X X X ?
Build interface talking to hardware (ODR) as a starting point;
Classify the properties and functionalities of each device for our EUDET DAQ system (DIF, LDA, ODR, etc.)
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 16
Future plans
Classify the necessary properties and functions of all hardware components;
Continue their building & testing, debugging & improving meanwhile;
Continue to develop interface to hardware (ODR) talking to DOOCS;
Investigate & define the DIFLDAODR links How to deal with (a)synchronization for C&C? Putting (all) components together, test…
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 17
Summary
Off-detector Receiver has been built for receiving, event building & data storage; its performance has been testing;
Clock & Control instead of triggers, the board will be built soon;
DOOCS framework is reusable & suitable for our DAQ system.
DAQ software is in designing phase…
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 18
DAQ architecture
Link Data Aggregator(LDA)
Detector Interface(DIF)
Detector Unit ASICs
Off Detector Receiver(ODR)
P. Göttlicher, DESY
DAQ software
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 19
Current ODR PrototypeHost System
EthernetEthernetInterfaceInterface
Test DataGenerator
SourceSelector
Buffer pointerBuffer pointerManagerManager
BufferBufferMemoryMemory
UsedFIFO DMA DMA
EngineEngine
PCI Express Buses
Local StorageLocal StorageRAID ArrayRAID Array
Gigabit NICTest source
External data stream
Optical fibre
data stream
data stream
data stream
data stream
Address
da
ta stream
da
ta stream
da
ta stream
Acce
ss p
ointe
rs
Even
t e
xtracts
Rea
d p
oin
ters
Te
st D
ata
Gen
era
tor
Con
tro
l
Off-Detector-Receiver
Card driver codeCard driver codeDMA engine DMA engine ODR local softwareODR local software
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 20
RequesterRequester
Current ODR Prototype“Requester”: multithreaded application,data copied to the host memory
ODR VHDL code RequesterRequester
FIFO Dup.
Data BuffData Buff
Data Buff
DMA FIFO to
User buffer
Write pointer
Read pointer
DMA data to user buffer(s)(host memory)
DMA transfer in maximum 4 k (page size) blocks.
Supervisor
Multiple instances of the “requester” thread. Each accessing different ODR card or different channel.
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 21
Channel 0
0
100
200
300
400
0 1000 2000 3000 4000
Data size [bytes]
tran
sfer
rat
e [M
B/s
]
1 Buff
5 Buff
10 Buff
15 Buff
20 Buff
25 Buff
Channels comparison
290300310320330340350360
0 1000 2000 3000 4000
Data size [bytes]
Tra
nsf
er r
ate
[MB
/s]
Ch0
Ch1
Ch2
Ch3
Establishing optimum numberof user-defined DMA buffers.No performance improvementabove 20 allocate buffers,
Comparison of all channels.Number of data buffers: 20
Performance (Int.Data Gen)
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 22
Simultanious access
0
200
400
600
800
0 1000 2000 3000 4000
Data size [bytes]
Tra
nsf
er r
ate
[MB
/s]
Ch 0
Ch 0&1
Ch 0&1&2
Ch 0&1&2&3
Improvements in performance when accessing more then one channel. The maximum performance achieved for 3-channels access. Adding additional Channel (4-channel access) does not yield improvement in transfer rate.
Performance (continuation)
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 23
Equipment Name Server (ENS)NN N+1N+1 N+2N+2
N+2,1N+1,1N,1 N+1,2
N+2,1,1N+1,1,1 N+1,2,1 N+2,1,2
…,1 …,2 …,1 …,1 …,1 …,2 …,3
K K+1 K+2 K+3 K+4
Facilities
Devices
Locations
Properties
Entries
Select between devices of the same type in different facilities in on-line addressing of the control system.
Provide the resolution of the names of the devices used in the control system with one entry per device server.
Define user authentication parameters which are used by the control software to restrict the access to the specified control devices
ILC/ECFA 2008 Warsaw EUDET/CALICE UK DAQ 24
ENS
Client programs
ENSFacility/Device/Location/Property
Query device name
Equipment device
Addresses of equipment server
actual data request to the
equipment server
Facility: CALICE.ECAL, CALICE.AHCAL, CALICE.DHCAL Device: ODR, LDA, DIF, ASIC1, ASIC2, ASIC3 Location: ODR1, ODR2, ODRX;
LDA1, LDA2, LDAX; DIF1, DIF2, DIFX Property: X X X ?
ENS can signal connectionssignal connections by additional properties, e.g. for device DIF:CALICE.ECAL/DIF/DIF1/ODR_CONCALICE.ECAL/DIF/DIF1/LDA_CON CALICE.ECAL/DIF/DIF1/DEBUG_MODE
top related