1

Electronics, trigger and physics Electronics, trigger and physics for LHC experimentsfor LHC experiments

2

The Large hadron ColliderThe Large hadron Collider

27 km length, 100 m underground, four interaction points (experiments)

proton-proton collisions, 7 TeV + 7 TeV (14 TeV in CM)

2808 bunches per beam with 11245 rounds per second = 32 Millions collisions per second

Nominal luminosity 1034 cm-2 s-1

3

The LHCb detectorThe LHCb detectorForward spectrometer to study heavy mesons (b,c) physics: rare decays and CP violation

Most of this heavy mesons are produced close to the beam axis (~ 40% in acceptance)

Low pT and high rapidity kinematic region

4

The vertex locator of LHCbThe vertex locator of LHCbElectronics and trigger for LHC

Silicon detector to track the charged particles close to the interaction region. In particular it is crucial to reconstruct the secondary vertecies

172K channels

Strips in R and φ projection (~10 μm vertex resolution)

Located 1cm from beam

Analog readout (via twisted pair cables over 60m)

from Si sensors

Analog signal to DAQ

Blue/yellow layers correspond to R and φ sensors

beam

~ 1 m

5

Digital optical linkDigital optical linkElectronics and trigger for LHC

High speed: 1Ghz - 10GHz – 40GHz

Extensively used in telecommunications (expensive) and in computing (“cheap”)

Encoding

Reliability and error rates strongly depending on received optical power and timing jitter

Multiple (16) serializers and deserializers directly available in modern chips (FPGA’s).

Transmission goodness by BER factor (bit error rate)

6

DAQ interfaces / readout boardsDAQ interfaces / readout boardsElectronics and trigger for LHC

Large Front-end data receptionReceive optical links from multiple front-ends: 24 – 96Located outside radiation

Event checkingVerify that data received is correctVerify correct synchronization of front-ends

Extended digital signal processing to extract information of interest and minimize data volume

Event merging/buildingBuild consistent data structures from the individual data sources so it can be efficiently sent to DAQ CPU farm and processed efficiently without wasting time reformatting data on CPU.Requires significant data buffering

High level of programmability needed

Send data to CPU farm at a rate that can be correctly handled by farm

1 Gbits/s Ethernet (next is 10Gbits/s)In house link with PCI interface: S-link

Requires a lot of fast digital processing and data buffering: FPGA’s, DSP’s, embedded CPUUse of ASIC’s not justified Complicated modules that are only half made when the hardware is there: FPGA firmware (from HDL), DSP code, on-board CPU software, etc.

7

New problemsNew problemsElectronics and trigger for LHC

Going from single sensors to building detector read-out of the circuits we have seen, brings up a host of new problems:

Power, Cooling

Crosstalk

Radiation (LHC)

Some can be tackled by (yet) more sophisticated technologies

8

Radiation effectsRadiation effectsElectronics and trigger for LHC

In modern experiments large amounts of electronics are located inside the detector where there may be a high level of radiation. This is the case for 3 of the 4 LHC experiments (10 years running)

Pixel detectors: 10 -100 MradTrackers: ~10MradCalorimeters: 0.1 – 1MradMuon detectors: ~10kradCavern: 1 – 10krad

Normal commercial electronics will not survive within this environment. One of the reasons why all the on-detector electronics in the LHC experiment are custom made

Special technologies and dedicated design approaches are needed to make electronics last in this unfriendly environment

Radiation effects on electronics can be divided into three major effectsTotal doseDisplacement damageSingle event upsets

1 Rad = 10 mGy1 Gy = 100 Rad

9

Total doseTotal doseElectronics and trigger for LHC

Generated charges from traversing particles gets trapped within the insulators of the active devices and changes their behavior

For CMOS devices this happens in the thin gate oxide layer which have a major impact on the function of the MOS transistor

Threshold shiftsLeakage current

In deep submicron technologies ( <0.25um) the trapped charges are removed by tunneling currents through the very thin gate oxide

Only limited threshold shifts

The leakage currents caused by end effects of the linear transistor (NMOS) can be cured by using enclosed transistors

For CMOS technologies below the 130nm generation the use of enclosed NMOS devices does not seem necessary. But other effects may show up

No major effect on high speed bipolar technologies

10

Displacement damageDisplacement damageElectronics and trigger for LHC

Traversing hadrons provokes displacements of atoms in the silicon lattice.

Bipolar devices relies extensively on effects in the silicon lattice.

Traps (band gap energy levels)Increased carrier recombination in base

Results in decreased gain of bipolar devices with a dependency on the dose rate.

No significant effect on MOS devices

Also seriously affects Lasers and PIN diodes used for optical links.

11

Single event upsets (SEU)Single event upsets (SEU)Electronics and trigger for LHC

Deposition of sufficient charge can make a memory cell or a flip-flop change value

As for SEL* (single event latchup), sufficient charge can only be deposited via a nuclear interaction for traversing hadrons

The sensitivity to this is expressed as an efficient cross section for this to occur

This problem can be solved at the circuit level or at the logic level

Make memory element so large and slow that deposited charge not enough to flip bit

Triple redundant (for registers)

Hamming coding (for memories)

* SEL: An abnormal high-current state in a device caused by the passage of a single energetic particle through sensitive regions of the device structure and resulting in the loss of device functionality

In telecommunication, Hamming codes are a family of linear error-correcting codes

12

PoweringPoweringElectronics and trigger for LHC

Delivering power to the front-end electronics highly embedded in the detectors has been seen to be a major challenge (underestimated).

The related cooling and power cabling infrastructure is a serious problem of the inner trackers as any additional material seriously degrades the physics performance of the whole experiment.

A large majority of the material in these detectors in LHC relates to the electronics, cooling and power and not to the silicon detector them selves (which was the initial belief)

How to improveLower power consumptionImprove power distributionSimulation of material budget

13

VME board plugged into backplane

Electronic crates in DAQElectronic crates in DAQElectronics and trigger for LHC

Going from single sensors to thousand channels readout forces to use a dedicated electronic design

Put many of these multi-port modules together in a common chassis or crate

The modules needMechanical supportPowerA standardized way to access their data (our measurement values)

All this is provided by standards for (readout) electronics such as VME (IEEE 1014)

14

Communication in crate: busesCommunication in crate: busesElectronics and trigger for LHC

A bus connects two or more devices and allows the to communicate

The bus is shared between all devices on the bus → arbitration is required

Devices can be masters or slaves (some can be both)

Devices can be uniquely identified ("addressed") on the bus

Famous examples: PCI, USB, VME, SCSIolder standards: CAMAC, ISAupcoming: ATCAmany more: FireWire, I2C, Profibus, etc…

Buses can belocal: PCIexternal peripherals: USBin crates: VME, compactPCI, ATCAlong distance: CAN, Profibus

Theoretically ~ 16 MB/s can be achieved

Better performance by using block-transfers

Easy to add new device, boards with standard interface

15

Network and farmNetwork and farmDAQ and trigger at LHC

For such huge amount of data to digest buses are not enoughsubdetector often very far from each other.Number of devices and physical bus-length is limited (scalability!). Useful for systems < 1 GB/s

Network technology solves the scalability issues of busesIn a network devices are equal ("peers")In a network devices communicate directly with each other (no arbitration necessary and bandwidth guaranteed)

data and control use the same path → much fewer lines (e.g. in traditional Ethernet only two)At the signaling level buses tend to use parallel copper lines. Network technologies can be also optical, wire-less and are typically (differential) serial

Examples:The telephone networkEthernet (IEEE 802.3)ATM (the backbone for GSM cell-phones)Infiniband

1

2

3

4

5

While 2 cansend data to 1and 4, 3 cansend at fullspeed to 5

2 can distributethe share thebandwidthbetween 1 and4 as needed

Network technologies are sometimes functionally grouped

Cluster interconnect (Myrinet, Infiniband) 15 mLocal area network (Ethernet), 100 m to 10 kmWide area network (ATM, SONET) > 50 km

16

A large experiment example: CMSA large experiment example: CMSDAQ and trigger at LHC

A selection mechanism (“trigger”)

Electronic readout of the sensors of the detectors (“front-end electronics”)

A system to keep all those things in sync (“clock”)

A system to collect the selected data (“DAQ”)

A Control System to configure, control and monitor the entire DAQ

Time, money, students

15 million detector channels @ 40 MHz ~15 * 1,000,000 * 40 * 1,000,000 bytes ~ 600 TB/sec (impossible to record)

HEP experiments usually consist of many different sub-detectors: tracking, calorimetry, particle-ID, muon-detectors

We need:

17

Trigger at LHCTrigger at LHCA typical collision is “boring” Although we need also some of these “boring” data as cross-check, calibration tool and also some important “low-energy” physics

“Interesting” physics is about 6–8 orders of magnitude rarer (EWK & Top)

“Exciting” physics involving new particles/discoveries is 9 orders of magnitude below tot

100 GeV Higgs 0.1 Hz600 GeV Higgs 0.01 Hz

We just need to efficiently identify these rare processes from the overwhelming background before reading out & storing the whole event

18

Technical requirementsTechnical requirementsNo (affordable) DAQ system could read out O(107) channels at 40 MHz → 400 TBytes/s to read out – even assuming binary channels!

What’s worse: most of these millions of events per second are totally uninteresting: one Higgs event every 0.02 seconds

A first level trigger (Level-1,L1) must somehow select the more interesting events and tell us which ones to deal with any further

Millions of channels → try to work as much as possible with “local” information

Keeps number of interconnections low

Must be fast: look for “simple” signaturesKeep the good ones, kill the bad onesRobust, can be implemented in hardware (fast)

Design principle:fast: to keep buffer sizes under controlevery 25 nanoseconds (ns) a new event: have to

Decide within a few microseconds (μs): trigger latency

Trigger at LHC

19

Physical requirementsPhysical requirementsRequirements driven by the physics objectives of the experiments

ATLAS and CMS (general-purpose, proton-proton, discovery physics)LHCb (B physics, proton-proton)ALICE (specialized for heavy-ion collisions)

Trigger at LHC

20

ATLAS and CMS: requirementsATLAS and CMS: requirementsTrigger for LHC

Triggers in the general-purpose proton–proton experiments,

Retain as many as possible of the events of interest for the diverse physics programs of these experiments

Higgs searches (Standard Model and beyond): e.g. H → ZZ → leptons, H → gg; also H → tt, H → bb

SUSY searches, with and without R-parity conservation

Searches for other new physicsUsing inclusive triggers that one hopes will be sensitive to any unpredicted new physics

Precision physics studies: e.g. measurement of W mass

B-physics studies (especially in the early phases of these experiments)

N.b. selections often need to be made at analysis level to suppress backgrounds, so focus especially on events that will be retained

21

ATLAS and CMS: constraintsATLAS and CMS: constraintsTrigger for LHC

L = 1034 cm-2s-1, σ = 100 mb (inelastic) → 109 interaction rate W or Z decays is O(100 Hz)

Total data flow = event rate × events size = 109 Hz × 1 MByte = 1000 TByte/s, absolutely impossible to record and also useless. Most of events are not interesting from the Physics point of view

Mandatory to insert filters (intemediate processing units) in order to reduce (order of magnitude) the events to record

Hardware filters with dedicated electronics

Software filters with online analysis and discrimination on commercial CPU farm

22

LHCbLHCbtrigger for LHC

The LHCb experiment, which is dedicated to studying B-physics, faces similar challenges to ATLAS and CMS

It operates at a comparatively low luminosity (~2×1032 cm-2s-1), giving an overall proton–proton interaction rate of ~20 MHz Chosen to maximise the rate of single-interaction bunch-crossings

The event size is comparatively small (~100 kByte)Fewer detector channelsLess occupancy due to lower luminosity

However, there is a very high rate of beauty productionGiven σ ~ 500 μb, bb production rate ~100 kHz

The trigger must therefore search for specific B decay modes that are of interest for the physics analysis

Event rate of only ~3 kHz

23

AliceAlicetrigger for LHC

The heavy-ion experiment ALICE is also very demanding, particularly from the DAQ point of view

The total interaction rate will be much smaller than in the pp experiments

L ~ 1027 cm-2s-1 R ~ 8 kHz for Pb–Pb collisions⇒

The trigger will select “minimum-bias” and “central” events (rates scaled down to total ~40 Hz), and events with dileptons (~1 kHz with only part of the detector read out)

However, the event size will be huge due to the high particle multiplicity in Pb–Pb collisions at LHC energy

Up to O(10,000) charged particles in the central regionEvent size up to ~ 40 MByte when the full detector is read out

Even more than in the other experiments, the volume of data to be stored and subsequently processed offline will be massive

Data rate to storage ~1 GByte/s (limited by what is possible/affordable)

24

High level trigger ratesHigh level trigger ratesTrigger for LHC

High level trigger rate vs event size for several experiments

It is clear the progress with time

The four LHC experiments differ mong them: from the highest L1 rate of LHCb to the huge event size of the ALICE

Rate*Size = bandwidth ~ constant for ATLAS, CMS and LHCb

25

How to defeat minimum bias: How to defeat minimum bias: transverse momentum ptransverse momentum p

TT

trigger for LHC

p-p (inelastic) collisions produce mainly hadrons with transverse momentum p

T ~ 1 GeV/c

Interesting physics (old and new) has particles (leptons and hadrons) with large p

T

W → eν, M(W) = 80 GeV/c2 and pT(e) ~ 40 GeV/c

H(120 GeV/c2) → γγ, pT(γ) ~ 50 GeV/c

B → μμ, pT(μ) ~ 3 GeV/c

Impose high threshold on pT of the particles

Implies distinguishing between difrent type of particles. This is possible for electrons, muons, jets

pp

T

26

How to defeat minimum bias: How to defeat minimum bias: transverse momentum ptransverse momentum p

TT

trigger for LHC

27

Particle identificationParticle identificationtrigger for LHC

4T μ+n

π+

e-

γ

Silicon MicrostripsPixels

TRACKER

ECALScintillating PbWO4 crystals

HCALPlastic scintillator/brasssandwich

⊗2T

SUPERCONDUCTINGCOIL

IRON YOKE MUON CHAMBERSDrift Tube ChambersResistive Plate Chambers Cathode Strip Chambers

CMS experiment

28

LHCb triggerLHCb triggertrigger for LHC

3 kHz

First Level (L0): 40 MHz → 1 MHz High-pT µ, e, γ, hadron candidates

(ECAL, HCAL, Muon).

Software level (High Level Trigger) Access all detector data.

Farm with ∼15000 CPU cores on multi-processor commodity boxes.

HLT1: Confirm L0 candidate with more complete info, add impact parameter and lifetime cuts: 1 MHz → ∼30 kHz.

HLT2: global event reconstruction + selections: 30 kHz → ∼3 kHz, where 1 kHz being dedicated to charm.

29

Event buildingEvent buildingDAQ for LHC

1) Event fragments are received from detector front-ends

2) Event fragments are read out over a network by an event builder system

3) Event builder assembles fragments into complete event

4) Complete events are sent to the high level trigger algorithm

Push based: event fragments are sent without feedback with the event builder system

Pull based: event builder system tells readout supervisor when and where (which event builder is ready) send the data

Readout supervisor

30

LHCb DAQLHCb DAQDAQ for LHC

31

LHC trigger/DAQ parametersLHC trigger/DAQ parametersTrigger levels

Level 1,2Rate (Hz)

Event size(Bytes)

Readout BW (GB/s)

HLT out (MB/s)(Events/s)

4500 (Pb-Pb)

103 (p-p)

5×107

2×106 251250 (100)200 (100)

3105 (LV1)

3×103 (LV2)1.5×106 150 300 (200)

2 105 106 100 1000 (100)

2 106 3.5×104 35 70 (3000)