cms calorimeter trigger phase 1 upgrade

T. Gorski and P. Klabbers TWEPP 2011 – Vienna, Austria - 1

CMS Calorimeter Trigger CMS Calorimeter Trigger Phase 1 UpgradePhase 1 Upgrade

CMS Calorimeter Trigger CMS Calorimeter Trigger Phase 1 UpgradePhase 1 Upgrade

P. Klabbers1, T.Gorski1, W. H. Smith1,

S. Dasu1, K. Compton1, M. Schulte2,

M. Bachtis1, I. Ross1, A. Farmahini-Farahani1, R. Fobes1,

D. Seemuth1, M. Grothe1, A. Gregerson1

1University of Wisconsin, Madison, WI, USA2AMD Research

TWEPP 2011

September 28, 2011

The pdf file of this talk is available at:https://indico.cern.ch/contributionDisplay.py?

contribId=102&sessionId=43&confId=120853


CMS Calorimeter GeometryCMS Calorimeter GeometryCMS Calorimeter GeometryCMS Calorimeter Geometry

EB, EE, HB, HE map to 18 RCT crates

Provide e/ and jet, ET triggers


Current Calorimeter Current Calorimeter Trigger AlgorithmsTrigger Algorithms

Current Calorimeter Current Calorimeter Trigger AlgorithmsTrigger Algorithms

e/ Rank = Hit+Max Adjacent Tower• Hit: H/E < Small Fraction• Hit: 2 of 5-crystal strips >90% ET in 5x5 Tower (Fine Grain)

Isolated e/ (3x3 Tower)• Quiet neighbors: all 8 towerspass Fine Grain & H/E

• One of 4 corners 5 EM ET < Thr.

Jet or ET

• 12x12 trig. tower ET sliding in 4x4 steps w/central 4x4 ET > others

: isolated narrow energy deposits• Energy spread outside veto pattern sets veto

• Jetif all 9 4x4 region vetoes off


Current CMS RegionalCurrent CMS RegionalCalorimeter Trigger (RCT*)Calorimeter Trigger (RCT*)

Current CMS RegionalCurrent CMS RegionalCalorimeter Trigger (RCT*)Calorimeter Trigger (RCT*)

• A very reliable low-latency system based on GaAs ASIC Technology

• Design based around 5 high-speed custom ASICs operating at 160MHz

• Low latency Cu links deliver HCAL and ECAL TPs at 4.8 Gbps w/no errors

• 1026 x 4 links aligned

• > 8000 calorimeter towers

• Very flexible LUTs for decompression, addition of towers, bits for e/ and algos, and masking of bad channels

• Crates operate synchronously• Dedicated low skew clock distribution

• Input aligned

• Sharing aligned

• Continuous monitoring• Link for errors and misalignment

• Real time emulation for data checks

• Offline for detailed analysis

*For more info on the RCT see TWEPP 2009, 2008, and 2007 proceedings

Front

Rear


Current CMS RCT*Current CMS RCT*Current CMS RCT*Current CMS RCT*• Physically large

• 18 9U and a 6U crate in 9 LHC racks• > 300 9U and 6U boards in operation

• 1026 4-pair Cu cables for links, 108 SCSI type for data sharing

• ECL PLCCs and ASICs (ECL) use more space than modern FPGAs

• Almost 5 kW power consumption per rack, 2 380 VAC (3) to 48V DC PSs in 9U chassis, DC-DC converters on board

• But…

• Algorithms in ASICs - inflexible

• By 2016 parts the system will have operated at CMS for >9 years

• Aging of boards and parts, obsolescence will make repair increasingly difficult

• No readout of trigger data • Must rely on input & output systems’ data

*For more info on the RCT see TWEPP 2009, 2008, and 2007 proceedings

RCT in CMS Service Cavern

RCT Receiver Card


CMS Upgrade Trigger StrategyCMS Upgrade Trigger StrategyCMS Upgrade Trigger StrategyCMS Upgrade Trigger StrategyConstraints

• Total output rate of L1 Trigger is 100 kHz

• Input rate increases 2-10 times over LHC design (1034 cm-1s-2)

• Number of interactions in a crossing (pileup) increases 4-10 times

• Thresholds will need to remain about the same to fulfill physics needs

Strategy for Phase 1 Calorimeter Trigger (operating 2016+):• Present L1 algorithms inadequate above 2×1034 or 1034 w/ 50 ns

bunch spacing

• Pileup degrades object isolation (electrons and taus)

• More sophisticated clustering & isolation needed for busier events

• Process with full granularity of calorimeter trigger information

• Current FPGAs allow more complexity and flexibility in algos and tuning of isolation and energy cuts

• Initial L1 Trigger simulations show a significant rate reduction with upgraded calorimeter trigger


CMS Upgrade Calorimeter Trigger CMS Upgrade Calorimeter Trigger Algorithm DevelopmentAlgorithm Development

CMS Upgrade Calorimeter Trigger CMS Upgrade Calorimeter Trigger Algorithm DevelopmentAlgorithm Development

• Particle Cluster Finder• Applies tower thresholds to Calorimeter• Creates overlapped 2x2 clusters

• Cluster Overlap Filter• Removes overlap between clusters• Identifies local maxima• Prunes low energy clusters

• Cluster Isolation and Particle ID• Applied to local maxima• Calculates isolation deposits around 2x2 and 2x3

clusters• Identifies particles

• Jet reconstruction• Applied on filtered clusters• Groups clusters to jets

• Particle Sorter• Sorts particles & outputs the most energetic ones

• MET,HT,MHT Calculation• Calculates ET Sums, Missing ET from clusters

EC

AL

HCAL Δη x Δφ=0.087x0.087

e/γ

EC

AL

HCAL

τ

EC

AL

HCAL

jet

η

φ

η

φ

η

φ


Isolated Electrons

Isolated Electrons

Upgrade

Existing

Isolated Electrons

Taus

Upgrade

Existing

Taus

Cal Trig. Efficiencies & RatesCal Trig. Efficiencies & Rates(CMS Upgrade)(CMS Upgrade)

Cal Trig. Efficiencies & RatesCal Trig. Efficiencies & Rates(CMS Upgrade)(CMS Upgrade)

4x Reduction in rate at 25 pileup events per crossing & improved efficiency

Upgrade

Existing

Upgrade

Existing


Upgrade Compact Calorimeter Upgrade Compact Calorimeter Trigger Architecture*Trigger Architecture*

Upgrade Compact Calorimeter Upgrade Compact Calorimeter Trigger Architecture*Trigger Architecture*

Regional results:

top e/’s and ’s,

4x4 tower sums

w/ ½ tower res, ECAL ET

21 Input Processors

x8η, x24φ Regions

21 Region Processors

x10η, x26φ Sort e/ &

Build & Sort Jets

To Global Trigger

X8 e/ and ’sx8 Jets

ECAL TowerEnergy (8bit)

Info (1bit)

HCAL TowerEnergy (8bit)

Info (1bit)

TowerEnergy (9bit)

Veto (1bit)

Process Process

Process

2 - 5 Summary Cards

Possibility to run different summary cards in parallel to optimise e/, , jet or energy sum path.

*Alternative to this pipelined architecture: See talk by G. Iles


Technology Upgrades for Technology Upgrades for the Compact Calorimeter Triggerthe Compact Calorimeter Trigger

Technology Upgrades for Technology Upgrades for the Compact Calorimeter Triggerthe Compact Calorimeter Trigger

A trigger upgrade means new hardware TCA (AMC standard)

• Compact, hot swappable boards

• System shrinks, now need only six 7U crates in one rack• Operate new systems in parallel

• Optical links instead of copper

• Compact, optical ribbon cables for data transmission

• Need to align links (data sharing, etc.)

• Advanced Monitoring and Configuration TCA Controller Hub (MCH) uses TCP/IP protocols

• 100Base-X Ethernet over backplane

• IPMI for initialization and monitoring

• Initialization and Configuration over LAN• FW upgrade and maintenance

• Algorithm Flexibility with newer FPGAs

• Currently designing for XILINX Virtex-6 (considering the Virtex-7)

• Integrated GTX links for data transmission (up to 6.5 Gbps)


1000Base-X Ethernet Demonstrator

1000Base-X Ethernet Demonstrator

• Running lightweight IP (lwIP) TCP/IP stack under Xilkernel on ML506

• Connected to departmental network

• Test #1: iPerf Xmt/Rcv between ML506 and PC:

• Rcv: 14 Mbps

• Xmt: 12-19 Mbps

• Test #2: Echo server between two ML506 bds

• Both boards running server and client app

• Bandwith sufficient for flashing, etc.

Xilinx ML-506 Virtex-5 Evaluation Board

GbE Running on SATA Cable

Test Board to Connect to µTCA Fabric A


MMC ProjectMMC Project• MMC: Module Management Controller

• IPMI endpoint for managing cards in µTCA Crates• UW Project: A “ground-up” implementation of an MMC based on an

Atmel AVR 32-bit Microcontroller• Supports the standard IPMI commands dictated by the

specifications, plus additional commands for operations outside the scope of the MMC specification• A full list of commands in backup slides

• Communicate with module prior & after FPGA initialization via LAN connection to MCH card in µTCA Crate

• Used for:• Power control & monitoring (incl. over-voltage/temp. protection)• FPGA Boot Image Selection & Load Control• Post-boot FPGA Configuration (e.g., geographical card IP address)

• Used on multiple CMS electronics upgrade designs


UW MMC Reference Design Hardware Block Diagram

UW MMC Reference Design Hardware Block Diagram

AtmelUC3A1512

Microcontroller(512KB Flash,64KB SRAM)

PrimaryFPGA

(AMC Board)

FPGAConfigFlash

(Non-volatileconfigsettings)

8KB SPIEEPROM

IPMB (I2C)

Augmented SPI

(Post bootconfig path)

ModuleBackend

Power

Temp.Sensor(TMP36)

FPGA CPU Reset

Temp.Sensor(TMP36)

ADC

Inputs

GA0-GA2FPGA Flash Load

Bac

ken

d P

wr

En

able

Boot Image

Select

ConfigLoadPath

ConsoleSIO

InterfaceBlue

LED1

LED2 IPMI LEDs

HandleµSwitch


MMC Project: Hardware & Remote Access

MMC Project: Hardware & Remote Access

Remote Linux Shell Access via ipmitool

Sensor display via NATView (Java LAN application for MCHs mfg’d by NAT GmbH)

Prototype Development Platform


Flash-over-LAN (FoL)Flash-over-LAN (FoL)

• Objective: Support remote update of FPGA Flash over the MicroTCA GbE connection

• FPGA-based server (Microblaze processor)• Uses TCP/IP stack (lwIP) running under Xilkernel• Common driver API for supporting different Flash

implementations (e.g., BPI, SPI) with device-specific drivers

• PC-based client• Connects to server on AMC cards to deliver new image

(supports MCS and binary file formats for Flash image)

• Development Platforms: Xilinx ML605 eval board for BPI flash, BU AMC13* for SPI flash• Programs ML605 parallel flash about 3x faster than

Xilinx iMPACT program (cable)

*See talk in XTCA working group by E. Hazen


Flash-over-LAN (FoL) Block Diagram

Flash-over-LAN (FoL) Block Diagram

FoL Client

(PC)

MCS

File

TCP/IP Connection (GbE) FoL

CommonServer

Device-SpecificDriver

FPGAFlash

(can be serialor parallelFlash memory)

Xilinx FPGA (AMC Card)

Fla

sh

I/O

Co

re

Advantages of the FoL Architecture:

• Primarily C/C++ implementation, as opposed to HDL (more productive development environment)

• Client model can support additional file types as necessary (e.g., SREC or binary)

• Common Server can support different device types as necessary via Device-Specific Driver interface

• Can manage multiple boot images and FPGAs on a single AMC card


Unified System Alignment StudyUnified System Alignment Study

Absolutely necessary for tower-level data sharing across calo regional boundaries•Problem: Several sources of misalignment

• Length of connection, phase and latency variation in SerDes links, and phase variation of clocks between clocks and cards

•4 Test Cards in a custom 2x2 test fabric

• Virtex-5 Rocket I/O GTP links

•LHC style clock-based timing• Local Trigger Timing and Control (TTC)

system for clock generation

•Link synchronization test bed

•Simulates 2 separate crates of 2 cards each

•Goal: demonstrate alignment of 56 separate channels all operating on the same time base


Unified System Alignment Study:

2x2 Firmware Test Bed Fabric

Unified System Alignment Study:

2x2 Firmware Test Bed Fabric

2(slave)

0(master)

1(slave)

3(slave)

TTCvi

4X (Passive)

4X

40 MHz

4X

4X

4X40 MHz

“Crate 2”

“Crate 1”

Ch14: Align CmdBrdcast

Ch15: Config Channel (Ring)

40 MHz Clk

40 MHz Clk12 Inter-Card + 2 Intra-Card Loopback X/R channels per board

56 total active channels


Unified System AlignmentUnified System Alignment

• Identify a target latency for each SerDes pair

• Set scheduled launch and arrival times for data at all SerDes endpoints per a common global reference, such as the Bunch Crossing 0 signal• Launch/arrival times derived from design

• Measure actual latencies by launching special test characters (8b/10b K char) from Tx links at scheduled times

• Measure actual arrival times of K chars at Rx links in comparison to expected arrival times from design

• Automatically compensate by adding delay at Rx end• Can add delay in fractions of LHC clock, depending on link rate

4 test cards 56 links at 1.6 Gbps synchronized:

proper alignment was verified using test pipelines to compare expected data (from links) with actual data (generated in the local pipeline)


Cal Trigger Processor Prototype Card Block Diagram

Cal Trigger Processor Prototype Card Block Diagram

MMC

Front EndFPGA

XC6VHX250T(GTX links)

Link ClockConditioning

CircuitrySDRAM

ECAL12-Channel

Optical Receiver

SecondaryPower

Supplies

Eta Sharing Links

12-ChannelOptical

Transmitter

12x8 RegionProcessing

FPGA XC6VHX250T

(GTX links)

Front Panel SideBackplane Side

Fabric A GbE (MCH1)

IPMI

TTC/DAQ to AMC13

12-Channel Optical

Receiver

12-Channel Optical

Receiver

12-Channel Optical

Receiver

Regional Outputs

Phi/Corner Sharing

FPGA Image Flash

(Parallel)

ECAL

HCAL

HCAL

Will utilize IPMI, MMC, FoL, Data alignment


Sh

arin

g I/O

Ca

rd

Sh

arin

g I/O

Ca

rd

SLHC Upgrade RCT CrateSLHC Upgrade RCT Crate

BU

AM

C1

3*

HCAL/ECAL TPGs from LIP oSLB & Minn. µHTR Cards

Outputs to GCT

Crate Output to DAQ

Clock/Control from TTC

(uTCA form factor, Vadatech VT892

style layout)

MC

H

PM

2P

M1

Sp

are

Slo

t

Ethernet Uplink(s)

Ca

l Trig

Pro

ce

ss

or

Ca

l Trig

Pro

ce

ss

or

Sh

arin

g I/O

Ca

rd

Sp

are

Slo

t

Ca

l Trig

Pro

ce

ss

or

Ca

l Trig

Pro

ce

ss

or

Ca

l Trig

Pro

ce

ss

or

Ca

l Trig

Pro

ce

ss

or

Left Sharing Data to neighbor crate(X, R MTP Ribbon)

Right Sharing Data to neighbor crate(X, R MTP Ribbon)

Sh

arin

g I/O

Ca

rd

*See talk in xTCA working group by E. Hazen

6 Crates

Full range,

12 towers


SLHC Regional Calorimeter Trigger Backplane Signal Allocation

SLHC Regional Calorimeter Trigger Backplane Signal Allocation

Fat

Pip

e (v

ia M

CH

1 o

r M

CH

2)

Cu

sto

m

Pas

sive

F

abri

c (C

PF

)

Φ-Sharing Ring

Processor Card Slots 4-9Sharing I/OSlots 2-3 & 10-11

Custom backplane needed for sharing of tower information for isolation algorithms and better position resolution

All will be done on backplane, will be done over fibers via I/O board


Calorimeter Trigger EvolutionCalorimeter Trigger EvolutionCalorimeter Trigger EvolutionCalorimeter Trigger Evolution

RegionalCalorimeter

Trigger To DAQ

Via GCT

HCAL HTR Cards

To DAQ

ViaHCALDCC2

Existing Copper Cables

ECAL TCCsTo DAQ

Via ECALDCC

Existing Copper Cables

HCAL uHTR Cards

ECAL TCCsSLHC

Cal TriggerProcessor

CardsOptic

al Ribbons

New Optical Transmitter

New OpticalReceiver

TriggerPrimitiveOptical

Patch Panel

ECAL Opti. RibbonsECAL

Indiv

. Fib

ers

(LC)

Optical Ribbons

To DAQ

Via BU“AMC13”

To DAQ

Via BU“AMC13”

To DAQ

Via ECALDCC

HCAL Opti. Ribbons

Present

Interim

Final


Conclusions and Future PlansConclusions and Future PlansConclusions and Future PlansConclusions and Future Plans

A new CMS Compact Calorimeter Trigger will meet and exceed the needs of the experiment as the luminosity and pileup increase• Flexible, low latency design using modern FPGAs

• Ease of maintenance and operation with TCA standard

• Small size allows installation in parallel to validate system with current version

• The new tools and techniques to operate the system are falling into place

• Flash over LAN, MMC with IPMI, Data Synchronization Technique

• Goal is to have demonstrator at the end of 2011

Expect to remove some of existing calorimeter cables and replace with optics in 2013• Build new system in parallel to validate

Full system installed by 2016


Backup SlidesBackup SlidesBackup SlidesBackup Slides


UW MMC IPMI Command ListUW MMC IPMI Command ListGet Device ID

Cold Reset

Broadcast “Get Device ID”

Set Event Receiver

Get Event Receiver

Platform Event (a.k.a. “Event Message”)

Get Device SDR Info

Get Device SDR

Reserve Device SDR Repository

Get Sensor Hysteresis

Set Sensor Threshold

Get Sensor Threshold

Set Sensor Event Enable

Get Sensor Event Enable

Get Sensor Reading

Get FRU Inventory Area Info

Read FRU Data

Write FRU Data

Get PICMG Properties

FRU Control

Get FRU LED Properties

Get LED Color Capabilities

Set FRU LED State

Get FRU LED State

Get Device Locator Record ID

FRU Control Capabilities

Set Backend Power

Get Backend Power

Set Payload Manager Settings

Get Payload Manager Settings

Get Fault Status

Set Boot Mode

Get Boot Mode

Set Sensor Alarm Mask

Get Sensor Alarm Mask

Set Handle Override

Set Current Requirement

Set Analog Scale Factor

Get Analog Scale Factor

Get Time Statistics

Set System Time

Get System Time

Get Nonvolatile Area Info

Raw Nonvolatile Write

Raw Nonvolatile Read

Check EEPROM Busy

EEPROM Erase

Application (06h/07h)

Sensor/Event (04h/05h)

Storage (0Ah/0Bh)

PICMG (2Ch/2Dh)

Custom (32h/33h)

NetFn Class Color Code

Poll FPGA Config Port

FPGA Config Read Status Register

FPGA Config Write Control Register

FPGA Config Write Data

FPGA Config Read Data

FPGA Config Nonvolatile Header Write

FPGA Config Nonvolatile Header Read


Tx Side Alignment Block DiagramTx Side Alignment Block Diagram

DataProcessing

Logic inFPGA Fabric

SERDESTx

Port(Dedicated

Logic)

Control

LaunchDelay

Counter

Tx ParallelData

(Locally-GeneratedControl Characters

MUX

Tx Data

Comma

Align Char

TCEna

Scheduled Delay

Link Ena

Global Align Cmd


Rx Side Alignment Block DiagramRx Side Alignment Block Diagram

DataProcessing

Logic inFPGA Fabric

SERDESTx

Port(Dedicated

Logic)Control

ACCArrival

Counter

Rx ParallelData

(Dly SrcSel Mux)

Delay

Select

MUX

Trigger Rx

Data

TCEna

Scheduled Arrival

=Align Char?

Delay Regs

Yes/No

Fixed RxDelayReg

Fixed RxDelay

Global Align Cmd

cms calorimeter trigger phase 1 upgrade

Documents