sept. 3. 2007tpc readoutupgade meeting, budapest1 daq for new tpc readout ervin dénes, zoltán...
TRANSCRIPT
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 1
DAQ for new TPC readoutDAQ for new TPC readout
Ervin Dénes, Zoltán FodorKFKI, Research Institute for Particle and Nuclear Physics
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 2
DDL architectureDDL architecture
PCI Bus
Concentrator
DAQ Read-out Receiver
Card (D-RORC)
SourceInterface
Unit
ForwardChannel
(Raw data)
BackwardChannel
(Pedestals, control)
DestinationInterface
Unit
Detector Data Link (DDL) :- Source Interface Unit- Transmission media- Destination Interface Unit
Standarddetector/DAQ interface
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 3
ALICE Detector Data LinkALICE Detector Data Link
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 4
0,00
50,00
100,00
150,00
200,00
250,00
300,00
10 100 1000 10000 100000 1000000
Block size (32-bit words)
Ba
nd
wid
th (
MB
/s)
DDLD-RORC
Read-Out Receiver Cards (D-RORC)Read-Out Receiver Cards (D-RORC)
• PCI-X adapter for 2 DDLmax 100 MHz
• PCI master:autonomous DMA
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 5
DDL FeaturesDDL FeaturesInterface:• Full duplex 32-bit data path on the destination interface (DIU card) • Half duplex 32-bit data path on the source interface (SIU card) • Full duplex flow control (XON/XOFF) • Interface clock up to 66 MHz (easy integration with PCI 66) • 264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.)
Implementation:• Duplex LC optical link up to 300 m• 2x FC or 2x GbE physical layer components• Small Form Factor Pluggable (SFP) optical transceivers • Bit error rate < 10-12 • Robust error detection: very low undetected bit error rate < 10-40 • Automatic link synchronization and management • Radiation tolerant Source Interface
Extras:• Stand-by support (low power consumption) • In-system reconfiguration / Remote system upgrade • Monitoring of the aging of laser diode of optical transceivers
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 6
Readout System PerformanceReadout System Performance• Motherboard with dual Xeon CPUs @ 2.4 GHz
• Six PCI-X slots, 4 bus segments (3+1+1+1), 2 controllers
• Linux OS
• ALICE Data-Acquisition software (DATE)
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 7
Performance: 6 D-RORCsPerformance: 6 D-RORCs• Testing the fully populated PC using data source internal
to PCI interface – Interoperability test
– Measure the maximal input bandwidth
PCI #6
PCI #5
PCI #4
PCI #3
PCI #2
PCI #1Seg
men
t #1
#2#3
#4
Con
trol
ler
#1#2
1 Ch 1 Ch
1 Ch 1 Ch 1 Ch
1 Ch 1 Ch 1 Ch 1 Ch
1 Ch 1 Ch
1 Ch 1 Ch 1 Ch
1 Ch 1 Ch 1 Ch 1 Ch 1Ch 1 Ch 1 Ch
264 464 424 528 792 1045 840
264 232 141.3 264 264 261.3 140
Aggregate Bandwidth [MB/s]Aggregate Bandwidth [MB/s]
Normalized Bandwidth Normalized Bandwidth [MB/s/Ch][MB/s/Ch]
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 8
DDL SoftwareDDL Software• All functions accessible as interactive
commands or API• Script-based interpreter for sequence of
operations:– Sending command to the FEE– Reading FEE status
• printing the status• comparing the status• polling the status
– Downloading data into the FEE from a file
– Reading data from the FEE• writing data into a file• comparing data with data in a file
• TPC configuration: < 0.3 s
FEROFERO
DDL
define pedestal_addr 0x1FFF
define enable_pedestal 0x2C
reset SIU
write_command enable_pedestal
write_block pedestal_addr pedestal.hex %x
read_and_check_block pedestal_addr pedestal.hex %x
define pedestal_addr 0x1FFF
define enable_pedestal 0x2C
reset SIU
write_command enable_pedestal
write_block pedestal_addr pedestal.hex %x
read_and_check_block pedestal_addr pedestal.hex %x
D-RORCD-RORC
LDCLDC
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 9
DDL TransactionsDDL Transactions
write_command <FEE command>
FEE command
No reply from the FEE
RORC Conc. FEE
BUS
FEE command
acq
Writing a command to the FEE (e.g. CLEAR)
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 10
DDL TransactionsDDL Transactions
read_and_print <FEE address> “<format>” [<stream>]
FEE addressRORC Conc. FEE
BUS
BUS
BUSFEE statusFEE status
read_and_check <FEE address> <expected status> <mask>
read_until <FEE address> <expected status> <mask> <s>
At least 6FEE clocks
FEE address
Reading a status from the FEE
acq
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 11
DDL TransactionsDDL Transactions
write_block <FEE address> <file> [<format>]
FEE address
FEERORC Conc. BUS
Data
FEE address
block lengthacq
Writing data to the FEE (e.g. pedestal)
end transactionacq
acq
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 12
DDL TransactionsDDL Transactions
FEE address
BUS
BUS
block length
Data
end transaction
end block
FEE address
acq
acq
Reading data from the FEE (e.g. read back pedestal)
RORC Conc. FEE
BUS
SW
read_and_check_block <FEE address> <file> [<format>]
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 13
DDL TransactionsDDL Transactions
start run
BUS
BUS
event length
Data
end transaction
end of event
Rdy to receive
acq
acq
Data collectionRORC Conc. FEE
BUS
SW
Data
event lengthend of event
end run
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 14
The Free FIFOThe Free FIFO
D-RORC PC memory bankFirmware
Event building
page address
page address
page address
Free FIFO
PC CPU
Allocation of free pages
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 15
Direct Memory AccessDirect Memory Access
D-RORCFirmware
PC memory bank
DDL
No involvement
PC CPU
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 16
The Ready FIFOThe Ready FIFO
D-RORC PC memory bank
Event building
DDL
Ready FIFOFirmware
lengthpage status
lengthpage status
lengthpage status
Delivery of filled pages
PC CPU
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 17
NA61 DAQ Configuration NA61 DAQ Configuration
VTPC1 VTPC2
MTPCR
MTPCL
Motherboards and concentrator
CAMAC TOFs
Trigger system
PC
RORC
RORC
RORC
RORC
DDL links
Gate & Clock to Motherboards
Busy
VME to PCI
?
Sept. 3. 2007 TPC readoutupgade meeting, Budapest 18
Timing DiagramTiming Diagram
Gate for event
Transfer to PC
Readout of FEE
Busy
Evt 1. Evt 2. Evt 3.
Read FEE to mem 1 Read FEE to mem 2
Transfer evt 1. to PC Transfer evt 2..
Busy generally = sampling time + FEE readout time, if the data flow fluid
Busy = 1, during the readout of the FEE or no free memory to read it