pipelined electronics
DESCRIPTION
Pipelined Electronics. Design Concept: Low Cost. Add-on modules. Sub-detector I/F User-defined module for analog part only. Minimizing the development cost. Readout CPU Commercially available PMC module. No hardware development cost. Common pipelined-readout platform (motherboard) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/1.jpg)
Apr.22,2005 Super B Factory Workshop
Detector (DAQ/Computing) Parallel
Takeo HiguchiInstitute of Particle and Nuclear Studies, KEK
on behalf of the COPPER working group
Pipelined ElectronicsPipelined Electronics
![Page 2: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/2.jpg)
• Add-on modules
• Common pipelined-readout platform (motherboard)– Generalized DAQ software.
• Common pipelined-readout platform (motherboard)– Generalized DAQ software.
Design Concept: Low Cost
Sub-detector I/F User-defined module for
analog part only. Minimizing the
development cost.
Sub-detector I/F User-defined module for
analog part only. Minimizing the
development cost.
Readout CPU Commercially available
PMC module. No hardware
development cost.
Readout CPU Commercially available
PMC module. No hardware
development cost.
![Page 3: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/3.jpg)
Common Readout Platform: COPPER
PMCProcessor
Trigger
GenericPMC slot
Sub-detector I/F
Sub-detector I/F
Sub-detector I/F
Sub-detector I/F
On-board Ether
Form factor = VME 9U
Sub-detector I/F slot 4Pipeline FIFO (1MB) 4PMC slot 332-bit local bus
32-bit 33MHz PCI bus
Local-PCI bridge
FastEthernet I/F
VME I/F
Sub-detector I/F slot 4Pipeline FIFO (1MB) 4PMC slot 332-bit local bus
32-bit 33MHz PCI bus
Local-PCI bridge
FastEthernet I/F
VME I/F
Data transfer on COPPERmeasured >65 MB/s.
>30 kHz L1 rate @ 1.6 kB/ev.Data size corresponds to typical SuberKEKB DC/COPPER.
Data transfer on COPPERmeasured >65 MB/s.
>30 kHz L1 rate @ 1.6 kB/ev.Data size corresponds to typical SuberKEKB DC/COPPER.
![Page 4: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/4.jpg)
PMC / PMC Processor
• PCI Mezzanine Card
– 100% compliant with the PCI.– Suitable for high density applications.
– Many commercial products are available: Ethernet cards, GbE cards, memory modules, CPUs, etc.
• RadiSys EPC-6315– PentiumIII 800 MHz w/ 512 MB main memory.
– 32-bit 33/66 MHz PCI bus interface.
Contains Linux kernel image
PCI NIC PMC NIC
![Page 5: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/5.jpg)
Sub-Detector I/F: FINESSE
• FINESSE: 18676 mm2 daughter card on the COPPER– Receives and digitizes sub-detector analog signals.– Holds the digitized signals for the level-1 trigger latency.– Outputs the data to the COPPER.
• Some FINESSE variations– TDC
– ADC
– TDC “tandem”
![Page 6: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/6.jpg)
SuperBelle CDC Readout: QTC+TDC
• QTC: charge to time conversion
Qevent
0 tDRIFT
L1-trigger
drift chargeto sense wire
t
T(Q)
tSTOP
Q-to-T conversionQ-to-T conversion
t
t1
t2
TDC & ADC at the same time,simplifying the readout scheme.
• t1 = drift time
• t1 – t2 = drift charge
9U-VME sized QTC modulewith LeCroy MQT300 QTC chip
9U-VME sized QTC modulewith LeCroy MQT300 QTC chip
![Page 7: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/7.jpg)
For tracking performance
Position resolution < 130 m
For dE/dx performance
Dynamic range 10 bit
Linearity < 0.5 – 1.0%
Other boundary conditions
Expected single rate per wire 100-200 kHz
Readout ch# ~ 15 k in total
For tracking performance
Position resolution < 130 m 27 m/bit (0.78 ns)
For dE/dx performance
Dynamic range 10 bit 17 bit
Linearity < 0.5 – 1.0% 0.49%
Other boundary conditions
Expected single rate per wire 100-200 kHz
Readout ch# ~ 15 k in total 24 ch/chip
SuperBelle CDC Readout: QTC+TDC
• TDC: time to digital conversion
Requirement
FINESSE with AMT-3 TDC chip (originally designed for ATLAS)FINESSE with AMT-3 TDC chip (originally designed for ATLAS)
AMT-3 TDC chip
![Page 8: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/8.jpg)
AMT-3 TDC Chip
• Up to 17 bit timing measurement: coarse + fine counters
– Pulse edge timings: coarse + fine– Trigger timings: coarse
• Trigger matching– Only the edges associated to Level-1 trigger are output.
• Level-1 pipeline provided (256 edges)
Coarse counter: 13-bit bunch crossing counter (40MHz @ LHC)Fine counter: additional 4-bit time memory cellCoarse counter: 13-bit bunch crossing counter (40MHz @ LHC)Fine counter: additional 4-bit time memory cell
SuperBelle system clock• Bunch crossing rate: LHC = 40 MHz,
KEKB = 508 MHz.• We choose SuperKEKB system clock
= 42.33 MHz = 508/12.
SuperBelle system clock• Bunch crossing rate: LHC = 40 MHz,
KEKB = 508 MHz.• We choose SuperKEKB system clock
= 42.33 MHz = 508/12.
![Page 9: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/9.jpg)
Level-1 pipeline FIFO(256 edges)
Readout FIFO(64 edges)
Channel buffer(4 edges)
Channel buffer(4 edges)
Channel buffer(4 edges)
Trigger matching
Trigger timing FIFO(8 events)
24
AMT-3
24-channel inputs
AMT-3 output
Level-1 trigger inputs
AMT-3 TDC Chip – Cont’d
collisiontime
L1 trigger
trigger latency (~5s?)
edge search
OK NG
Trigger matching
![Page 10: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/10.jpg)
of AMT-3
No hardware output indicating FIFO full– Unexpected data loss is only notified by an “error flag” in the next
event output.
Readout “readout FIFO” of AMT-3 quickly
to COPPERDREADY
DATA
AMT-3AMT-3FPGA
w/ FIFOFPGA
w/ FIFO
SCLK: 42.3 MHz
BUSY
No further trigger
![Page 11: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/11.jpg)
AMT-3 L1-Pipeline Depth
Less depth: 256 edges shared by 24-input channels– LeCroy TDC (we’re currently using): 16 edges / each channel.– Is it sufficiently large? MC simulation is needed.
• MC parameters – Assuming SuperBelle drift chamber– # of input channels: 24– Random wire hit: 100 kHz (Poisson distribution)– Level-1 trigger rate: 50 kHz (Poisson distribution)– # of gen. words per wire hit: 2 for signal edges + additional for noise– Level-1 trigger latency: 5 s
Extreme case
![Page 12: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/12.jpg)
“Toy” MC Simulation: Buffer Usage
Level-1 pipeline usagesampled at every clock [edges]
En
trie
s / 1
ed
ge
13.7 s
1.37 s
137 ms
13.7 ms
FIFO full point
Extrapolate … = ~104
Extrapolation assuming the right side slope unchanged.
FIFO/AMT-3 won’t get fullbefore O(105) sec.
Very preliminary
![Page 13: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/13.jpg)
TDC FINESSE with AMT-3
• Block diagram of TDC FINESSE (not finalized yet)A
MT
-3A
MT
-3 Dataformatter
Dataformatter
Triggerreply logic
Triggerreply logic
FIFO-fullmonitor
FIFO-fullmonitor
Busygenerator
Busygenerator
FIFO
42.3MHzclock Level-1 trigger
Trigger busy
Local-busI/F
Local-busI/F
Sub
-det
ecto
rS
ub-d
etec
tor
CO
PP
ER
eve
nt F
IFO
CO
PP
ER
eve
nt F
IFO
Local bus (A7D8)
COPPER FIFO full
Xilinx Spartan-3
![Page 14: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/14.jpg)
[Readout data] [Readout data]
COPPERCOPPER
The First AMT-3 Signal on COPPER
Pulsegenerator
Pulsegenerator
NIMECLconverterNIMECLconverter
FINESSEFINESSE
AMT-3AMT-3250 ns 4.2 s
w#1: 00111010010111011110110110000110
w#2: 00111010010110011110111011011010
@ ch 0x0b
content tag(0x3 = edge data)
ch id.(ch = 0x0b) 1: leading edge
0: trailing edge 17-bit edge timing
pulse width
Measured pulse width dist.
peak = 251 nsrms = 0.93 ns
CPUCPU
w#1: 00111010010111011110110110000110
w#2: 00111010010110011110111011011010
![Page 15: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/15.jpg)
Future Plan
• Short term (-Jul.): System test– TDC resolution study with test input pulses.– Investigation of FIFO full behavior of AMT-3.– Stability tests: stability against random trigger, long-run stability, etc.– Software development: device driver, readout software, etc.
• Middle term (Aug.-Nov.): First step integration to Belle DAQ (EFC)– EFC is the smallest sub-detector (total = 320 ch).– EFC has two alternative readout paths (CDAQ + EFC own): failsafe and easier cross
check.
• Long term (Dec.-): CDC readout with COPPER based system
![Page 16: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/16.jpg)
First Step Integration into Belle DAQ
• Replace Belle EFC DAQ with COPPER EFC =Extreme Forward Calorimeter
EFC =Extreme Forward Calorimeter
ForwardEFC
ForwardEFC
BackwardEFC
BackwardEFC
160 ch
160 ch
COPPER COPPER TDCTDCTDCTDC
TDC3
TDC3
FASTBUS
VxWorksEB
VME crate
COPPER3
COPPER3
Eth
ern
et s
wit
ch
PC
TDC “Tandem”FINESSE
TDC “Tandem”FINESSE
LocalSTORLocalSTOR
![Page 17: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/17.jpg)
Summary
• We have developed module-structured system for SuperBelle readout consisting of COPPER boards and FINESSE cards.
• The QTC and TDC will be employed to readout SuperBelle CDC data.
• We use AMT-3 TDC chip. The TDC FINESSE card with the AMT-3 has been developed.
• Level-1 pipeline depth in the AMT-3 is examined by MC simulation.
• We have observed the AMT-3 is correctly digitizing the input pulses.
• The first step integration of the COPPER system into the Belle DAQ is planned in this August using EFC.
![Page 18: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/18.jpg)
16 ch
16 ch
8 ch
8 ch
TDC “Tandem” FINESSE
• LeCroy TDC: 6 connectors• COPPER: 4 FINESSE slots
• To use present analog cables…– We’ve made a ‘tandem-FINESSE’
module equipped with 3 connectors.
• Channel density– 48 ch / tandem-FINESSE.– Same as present LeCroy TDC.
Tandem-FINESSE
AMT-3AMT-3
AMT-3AMT-3
2
![Page 19: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/19.jpg)
Readout PC and Event Builder
copper0copper0
copper1copper1
copper5copper5
…
Readout PC
Present event builder
Sw
itchi
ng h
ubS
witc
hing
hub
100 Base-T
GbE GbE GbE
• Readout PC:– CPU : Intel Dual Xeon 2.46 GHz– OS : RedHat9(2.4.18-20smp)
• Readout PC:– CPU : Intel Dual Xeon 2.46 GHz– OS : RedHat9(2.4.18-20smp)
Data transferperformance
Faster readout PC and/or software tuning are requested for SuperBelle use.
![Page 20: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/20.jpg)
Calibration Run
• Calibration run– Daily detector calibration with test pulse and/or test trigger.
• Readout PCs should take responsibility to collect data– O(100) COPPERs / sub-detector 6 readout PCs Event building mechanism is needed for calibration run.
![Page 21: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/21.jpg)
CPU Boot-up Mechanism
• Compact Flash– Boot up procedure is well established.
– Distribution of updated software to all CF cards is non- trivial.
– Write access to CF card shortens the card lifetime.
– CF card is getting out of date.
• Network boot– Distribution of updated software to all CF cards is easy.
– We have no established method yet to boot-up EPC-6315 from network.
– Boot-PC farm serving multiple boot requests must be developed. It is never trivial.
![Page 22: Pipelined Electronics](https://reader036.vdocuments.site/reader036/viewer/2022081506/56814ab7550346895db7cdf5/html5/thumbnails/22.jpg)
Run Control
copper0copper0
copper1copper1
copper5copper5
…
NSM Gateway
NSMNSM or other mechanism
To avoid increase of NSM nodes, we put NSM gateway (like SVD2).
fbdaqprivate network
Readout PC(l0efc)
NSM Master