kip ivan kisel, uni-heidelberg, rt20031 22 may 2003 a scalable 1 mhz trigger farm prototype with...

Ivan Kisel, Uni-Heidelberg, RT2003 122 May 2003

KIPKIP

A Scalable 1 MHz Trigger Farm Prototype A Scalable 1 MHz Trigger Farm Prototype with Event-Coherent DMA Inputwith Event-Coherent DMA Input

V. Lindenstruth, D. Atanasov, V. Lindenstruth, D. Atanasov, I. KiselI. Kisel, A. Walsch, A. Walsch( KIP, Uni-Heidelberg, Germany )( KIP, Uni-Heidelberg, Germany )

H. Muller, D. Altmann, A. Guirao, F. Vinci dos Santos H. Muller, D. Altmann, A. Guirao, F. Vinci dos Santos ( CERN, Geneva, Switzerland )( CERN, Geneva, Switzerland )

•LHCb Level-1 Trigger•Trigger Concept•Trigger Prototype•Trigger Simulation•Trigger Algorithm


KIPKIP

ee

hh

Level-1 Trigger for LHCbLevel-1 Trigger for LHCb

1. Find VELO 2D tracks and reconstruct 3D primary vertex2. Reconstruct high-impact parameter tracks in 3D3. Extrapolate to TT through small magnetic field PT4. Match tracks to L0 muon objects PT and PID5. Select B–events using impact parameter and PT information6. Use T1—3 data to improve further selection (5—10% of events)

VELOVELO TTTT (T1—3)(T1—3)


KIPKIP Trigger ConceptTrigger ConceptReduce rate from 1 MHz to 40 kHz

Send data to RU (8 kB/evt 8 GB/s): VELO + TT + L0DU T1—3 (~5—10% on CPU demand)

Traffic shaping --> use Scheduler !Traffic shaping --> use Scheduler ! NIC with Remote Direct Memory Access !NIC with Remote Direct Memory Access ! PrototypePrototype : 2D torus with 32 dual nodes at 1.24 MHz: 2D torus with 32 dual nodes at 1.24 MHz Trigger farm Trigger farm : 3D torus with up to 1200 CPUs: 3D torus with up to 1200 CPUs

TagNet

Data

PC farm (2D torus)

Scheduler

- RU- CN

X->Y routing

x

y


KIPKIP SchedulerScheduler - Basic Block Diagram - Basic Block Diagram

The supervisor of the system. Handle a coherent data transfer between RUs and CNs. Feed the TagNet with tags for synchronous data transfer in the RUs.

Scheduler

Core

TagOutputStage

TagNetFeed

TagNetFeedback

User Control

Event entries

FreeCN IDsentries

List of free computing

nodes

ControlRegister

StatusRegister

TagInputStage


KIPKIP

Tag In Tag Out

Data In

Sub

even

tB

uffe

r

DMA

PCI BusNIC Interface

NIC OutNIC In

C/M ?

CommandExecution

MessageExecution MUX Tag

Buffer

Readout Unit (RU)Readout Unit (RU)Command Tag Message Tag


KIPKIP Trigger Farm Prototype in HeidelbergTrigger Farm Prototype in Heidelberg

>1 MHz

64 CPUs

2D torus

6 Gbit/s NIC

1 year

480 MB/s p-p

450 MB/s x-y


KIPKIP

•Automatic setup of the compute farm

•Configure and control processes on every CN

GUI of PrototypeGUI of Prototype


KIPKIP

Scheduler

TagNet 3D Core 1D Cover

Data

•TagNet – schedule and send small data packets•Core network – distribute data to the target compute nodes•Cover network – increase number of compute nodes•X->Y->Z routing path

- RU- CN

3D Torus Topology 4x4x(1+2+1)3D Torus Topology 4x4x(1+2+1)

x

y

z


KIPKIP Ptolemy II Simulation of the Trigger Ptolemy II Simulation of the Trigger 3D Torus (6x6x8)3D Torus (6x6x8) 275 CNs275 CNs


KIPKIP Simulation of the Trigger --- ResultsSimulation of the Trigger --- Results

128 B/RU 2.1 MHz measured !

1200 CPU

+5% T1--3

CompactCompact

ScalableScalable

FastFast

Fast ResponseFast ResponseVELO

ZVELO events

VELO T1-3 New VELO

Scheduler


KIPKIPTracking Efficiency and PV ResolutionTracking Efficiency and PV Resolution

Track subsetsTrack subsets

Reference B longReference B longReference prim. longReference prim. longReference BReference primaryReference setAll setExtra setCloneGhost

97.797.799.199.196.698.797.093.681.1 4.5 6.3

95.195.197.197.193.393.992.387.570.2 4.0 9.3

2D % 3D<70> <8>

ZZcorecore 46 46 mm

X/YX/Ycorecore 17 17 mm


KIPKIP Trigger PerformanceTrigger Performance

time (ms)

E

vent

s

17

17

ms

ms

Mean: 15 15 ss

Max: ~130 s

CPUCPU4.8 ms4.8 ms

1) Tracking efficiency 97—99% 2) PV resolution 46 m 3) Timing 4.8 ms

Expect a factor 7—8 in CPU power in 2007(PASTA report)

=> we are already within 1 ms !=> we are already within 1 ms !

•Cellular Automaton algorithm•FPGA co-processor at 50 MHz•8 processing units running in parallel

=> 15 => 15 s !s !

FPGA FPGA co-processorco-processor

E

vent

s

time (s)


KIPKIP

Summary:Summary:

Demonstrated Architecture with 3D torus and TagNet Prototype of 64 CPUs has shown stable work at > 1 MHz The Simulation is based on the prototype measurements The Algorithm has high performance on tracks and vertices The Cost is 1300 kCHF (500 CPU) / 2300 kCHF (1000 CPU) The Team --- Heidelberg, CERN and Dubna

kip ivan kisel, uni-heidelberg, rt20031 22 may 2003 a scalable 1 mhz trigger farm prototype with...

Documents