A proposal for a full upgrade of the L1CTT/DFE for Run2b

Meenakshi NarainBoston University

Outline: Why do we need a full upgrade? The new proposed design..

Groups Simulation and Algorithm development:

Kansas: Graham Wilson, Carsten Hensel

Manchester: (Liang Han), Terry Wyatt

Notre Dame: Mike Hildredth

Hardware: Boston University:

Meenakshi Narain, Eric Hazen, Ulrich Heintz, Shouxiang Wu

FNAL: Marvin Johnson, Stefan Gruenendahl, Jamieson

Olsen

L1CTT Upgrade Motivation

In Run IIa and RunIIb, the L1 Track Trigger Provides: CFT tracks for L1Muon Seeds High pT isolated track trigger capability CFT tracks +CPS clusters embryonic electrons Found tracks for STT b-tagging

We believe the Run IIa L1CTT will fail at the occupancies we expect in Run IIb, resulting in Many more fake high-pT tracks for all systems The loss of rejection from combining subsystems

Physics Motivations Must trigger on high-pT leptons with high

efficiency, low fake rate: HWbbe or bb, HZbbll Cal-Track match is important/necessary here

Must provide clean track samples for b-tags with the STT HZbb, Zbb

Could (Must?) provide high-pT tracks for hadronic tau triggers H(gets you 35% equivalent lumi boost) Cal-Track match is important/necessary here

(Correlations between Systems is Necessary)

Run IIb Luminosity Projections

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10Start of Fiscal Year

~1.6e32

~2.8e32Accelerator draft plan:Peak luminosities

Pea

k Lu

min

osity

(x10

30cm

-2se

c-1)

We are here~5.6e31

Run IIa at high luminosity

Even at modest occupancies, the high-pT single track trigger would fire on ~2% of events (>100kHZ)

Isajet MC events

Nominal 5E32(1 high pT track)

(1 high+1 medium pT)

L1CTT Granularity Tracking trigger

rates sensitive to occupancy

Upgrade stategy: Narrow tracker

roads by using individual fiber hits (singlets) rather than pairing adjacent fibers (doublets)

Cal-track matching

Run IIb

Use full fiber resolution to restrict roads

Run IIa

L1CTT Algorithm Results

With baseline version of new L1CTT:Scheme/pT range

Tracking Efficiency (%)

Rate of Fake Tracks (% events)

Resources

ABCDEFGH (RunIIa)(pT>10 GeV)

96.9 1.02 0.10 11k X 8

abcdefgh(pT>10 GeV)

98.03 0.22

0.056 0.009 9.4k X 16

abcdefgh(5 GeV<pT<10 GeV)

99.20 0.14

0.89 0.11 8.9k X 12

abcdEFGH(3 GeV<pT<5 GeV)

98.40 0.20

4.5 0.2 11.3k X 12

abcdEFGH(1.5 GeV<pT<3 GeV)

95.15 0.32

25.4 0.2 15.5k X 12

L1CTT architecture A multistage system

Analog Front End (AFE): Signals from the tracker

Mixer Sort signals in trigger sector wedges

Digital Front End (DFEA): Track Trigger logic

Octant (CTOC) Combine track informationfrom several DFE boards

CTTT Construct track trigger terms

Trigger Manager Construct 32 AND/OR terms used by the L1 Trigger Framework in forming the trigger decision

L1CTT architecture The approved upgrade:

Replace Digital Front End daughter boards (DFEA):

Need to fit 3-5 times more equations

Rebuild these cards using larger FPGAs

Use XC2V6000 Chip with 6Msytem gates

Equation Resource studyAs a function of eff /number of equations:

Even the worst case equation scenario all singlets in 4 pT bins – fits in two XC2V6000

DFEA daughter board First prototype with XC2V4000 FPGAs

in hand and tested

Lessons from Run2a commissioning

Current “approved plan” upgrade of the DFEA daughter boards. Provide a 5x increase in available logic resources.

With the experiences over the last year while commissioning the present Run2a L1CTT system a few very important issues have surfaced:

Lack of extensive testability of the input and output information from the DFEA

Excessive Firmware download times currently about four hours (with 2 parallel jobs)

– may need restarts (5 automatic, 2-3 manual) Limited by the slow 1553 interface Scales up to about 12 hours/job for RunIIb equation file sizes

Clock and SCL signal distribution (dependent on upstream info)

A shortage of spare backplanes (there are none)

Desired Improvements Reduce FPGA configuration download times:

From many hours/ half a day to minutes Concerns about reliablility of the download of firmware 3-5x the

existing size. Permits and improves reliability. realistic times for debugging and updates

Improve Testability: Provide diagnostic access to the processor FPGAs on the DFEA

DB. Improved ability to inject test data and capture input data for

in-situ testing. Enhance post-manufacturing testing (boundary scan or other

type)

Provide a means to inject SCL-provided control signals (RF Clock, First Crossing, etc) to an entire crate for bench-test

and commissioning. This will permit stand-alone operation of the DFEA system for debugging and commissioning

Move input cable plant from front of the crates to the rear. Provide LED status indicators on the front panel. Improve reliability over the existing daughterboard connectors

The Proposal In order to add testability to the system, one needs:

A new motherboard (daughterboard – to a less extent) design.

Improved diagnostics: input and output buffers, and L3 capability.

To alleviate colossal download times we need: A new DFE crate controller with a

faster connection to the D0 online computers. A new DFE backplane design

Utilize this chance to put the cables to the back. No transition boards.

Detailed specifications of the boards are being developed – follow links through the Run2 trigger web page

DFE Backplane Standard VME size, 21 slot, 6U monolithic backplane with

custom connectors Slot 1 is used for the DFE crate controller, DFE modules are

located in slots 2-21. In slots 2-21, the backplane will support up to ten input cables

and four output cables passing through the backplane.

The backplane carries a simple read/write bus for communication with the DFE modules.

Also carries clock and control bits from the SCL receiver The DFE boards can also share data with their neighboring

boards

A high-voltage (48V) power distribution scheme is under consideration.

Eliminates potentially hazardous high-current low voltage supplies

allow more power to be distributed over the backplane with smaller conductors, eliminates heavy cables, remote sensing oscillations

However the concern is here noise in the calorimeter – tests will be done before deciding on this scheme.

DFE Crate Controller The controller must download firmware to the track-

finding FPGAs on the logic boards via the backplane read/write bus.

In addition, it provides external access to the bus for configuration, monitoring and debugging.

The existing DFE controller: flash memory for local FPGA configuration storage, and a 1553

control bus interface for external access.

The new controller will prove a direct interface to the read/write bus via a high speed (gigabit class) fiber optic link (also simpler).

The flash memory and microcontroller will be eliminated. The firmware for the track-finding FPGAs will be downloaded

directly over the fiber optic link at very high speed, allowing fast firmware changes for updates and debugging.

In addition, test patterns may be downloaded to the DFEA boards and captured output data read out over the link.

It is currently planned to use a commercial gigabit-ethernet fiber interface module to simplify the controller logic such that it may be implemented in a single FPGA.

A suggested solution based on CMS design:

VMEbus Controller with Gigabit Ethernet – A custom board designed and developed at OSU– Based on XILINX Virtex-II Pro– Optical transceiver (for Gbit Ethernet) – Communicates with stand-alone PC via Ethernet– Inexpensive: ~ $600 each, tests begin soon.

DFE Mother Boards Redesign for increased testability

Direct read/write access to daughter board FPGAs Status LEDs on front panel Replace or eliminate unreliable spring contact

connectors Eliminate rear transition modules

DFEA DAUGHTERBOARDA

B

B

L

SLDB

SLDB

POWER

MOTHERBOARDLOGIC

DFEA DAUGHTERBOARD

TestPoints

JTAGConn.

LEDsButtons

COAX“PIGTAIL”

Daughterboard(sector n)

BackplaneInterface

FPGA

J102.sxd

SLDB

LVDS channel link to CTOC (LVDS-1)

LVDS channel link to STOV/STSX (LVDS-2)

Coaxial Cu link to L1muon (OUT1)

Coaxial Cu link to L1caltrk (OUT2)

Daughterboard(sector n+1)

SLDB

LVDS channel link to CTOC (LVDS-3)

LVDS channel link to STOV/STSX (LVDS-4)

Coaxial Cu link to L1muon (OUT3)

Coaxial Cu link to L1caltrk (OUT4)

LVDS-1 (RED)sector n-1

LVDS-2 (ORANGE)sector n

LVDS-3 (GREEN)sector n

LVDS-4 (PURPLE)sector n

LVDS-5 (RED)sector n+1

LVDS-6 (BLUE)sector n+1

LVDS-7 (YELLOW)sector n+1

LVDS-8 (GREEN)sector n+2

LVDSCHANNEL LINK RX

PROM

FRONTPANEL

DFE Backplane R/W Bus

LVDS channellink TX

TIMING

DAT

A

IsoBits to/from sector n-1

IsoBits to/from sector n+2

IsoBitsCLKBUF DFE Backplane

Timing + CLK

TIMING

DAT

A

28/

28/

18/

28/

28/

18/

28+clk/

LVDS channellink TX

L3

L3

L3 Data to DFEC

DFEA daughter board First prototype in hand and tested Will need to make a few changes due to the

new design of the MB etc Improved system level diagnostics

By improving ability to send test patterns Capture input/output buffers and enhanced

Level3 support

Improve reliability over the existing spring-contact PCB surface daughterboard connectors

Incremental Equipment Cost

Cost k$Backplane 14Crate Controller 7.5Optical download and control link 5KTest equipment + splitters + crate 17Motherboards 61MB Prototypes + test stand 25Total 129+Labor Costs

New Schedule Aim for:

installation in Summer of 2005 A slice test during Fall 2004 shutdown.

Installation & Commissioning

The proposed scheme is driven by the desire to enhance testability and reliability of firmware downloads.

Eases Installation and commissioning

Plan to run on a partial crate of the new system on the platform in parallel with the existing DFEA using LVDS splitters

Key to successful commissioning of the new system

Advantages: Can assemble the whole crate outside of the collision hall Extensive testing of the entire chain possible before putting in

collision hall Enhanced testing capability Use Run2a Data derived test vectors to verify

Take out the old crates (2 of them) and replace them with new ones

Use the existing Run2a tools used for commissioning Low level changes will be transparent at user level

Summary Proposal to upgrade the entire L1CTT DFE

hardware Based on lessons learnt from Run2a L1CTT

commissioning Provide more in-situ testing capability

Add more I/O buffers and L3 readouts Reduce the firmware download times

significantly Use high speed fiber optic links

Installation a bit simpler extensive tests can be performed prior to installation

Commissioning times will be reduced Faster downloads Uses existing infrastructure

DFE Motherboards The Digital Front End (DFE) Motherboard is a general purpose, high

bandwidth platform for supporting reconfigurable logic such as FPGAs. It is intended for applications where a DSP or other microprocessor is too

slow.

The DFE motherboard is a 6U x 320mm Eurocard with fully custom hard metric backplane connectors.

Eight point-to-point links bring data onto the DFE motherboard at an aggregate data rate of about 12 Gbps.

Each physical link consists of five twisted pairs (Low Voltage Differential Signals)

These are terminated with hard metric female connectors on the DFE backplane and passed through to the DFE motherboard.

After entering the DFE motherboard, the ten links are sent to receivers, which convert the serial data back to a 28 bit wide bus running at 53 MHz.

These busses, in turn, are buffered and routed to the two daughtercards. The outputs from the daughterboards are fed back to the

motherboard, converted to serial data by channel link transmitters and SLDB (serial link

daughter boards) mounted on the motherboard. The serial data are passed through the backplane to the output cables.

A “local bus” bridged to the backplane bus will allow direct read/write access to the individual FPGAs on the DFEAs for debug and monitoring.

Additionally, a dedicated bus for cross-communication between the two DFEA daughterboards will be provided.

Core Trigger Menu Simulations

Total L1 bandwidth budget= ~3 kHz

Total rate: ~15kHz ~30kHz ~3.2kHz

Additional headroom available from• topological cuts available in upgraded L1cal•Higher muon pT threshold with upgraded CTT

Trigger Example Physics Channels

L1 Rate (kHz)

(no upgrade) L=1E32

L1 Rate (kHz)

(no upgrade) L=2E32

L1 Rate (kHz) (with

upgrade) EM (1 EM TT > 10 GeV)

eW , SUSY, jjeWH

0.65 1.3 0.7

Di-EM (1 EM TT > 7 GeV, 2 EM TT > 5 GeV)

Z ee, W , SUSY

ZH eejj

0.25 0.5 0.1

Muon (muon pT > 11 GeV + CFT Track)

W WH jj

3 6 0.4 Di-Muons (2 muons pT > 3 GeV + CFT Tracks)

Z J

ZH jj

0.2 0.4 < 0.1

Electron + Jets (1 EM TT > 7 GeV, 2 Had TT > 5 GeV)

WH e+jets tt e+jets

0.4 0.8 0.2

Muon + Jet (muon pT > 3 GeV, 1 Had TT > 5 GeV)

WH +jets tt +jets

<0.1 < 0.1 < 0.1

Jet+MET (2 TT > 5 GeV, Missing ET> 10 GeV)

bbZH , SUSY

1.1 2.1 0.8

Muon + EM (muons pT > 3 GeV+ CFT track + 1 EM TT > 5 GeV)

H WW, ZZ <0.1 < 0.1 < 0.1

Single Isolated Track (1 Isolated CFT track, pT > 10 GeV)

H W 8.5 17 1.0 Di-Track (1 isolated tracks pT > 10 GeV, 2 tracks pT > 5 GeV, 1 matched with EM energy)

H SUSY

0.6 0.6 <0.1

1E32 2E32 2E32