17 March 2006

Christos Zamantzas1

LHC Beam Loss Monitoring System Beam Loss Monitor electronics and

Beam Dump Thresholds

LHC Machine Protection Working Group

17 March 2006

Christos Zamantzas2

LHC Beam Loss Monitoring System

BLM Overview Surface Card (BLMTC) Processes running in the Surface FPGA

Real-Time Analysis of DataThreshold Comparator & Masking

Scaling Down the Threshold Values (Proposal)

Configuration of the DAB64xConfiguration Schemes (x3)

Outlook

17 March 2006

Christos Zamantzas3

BLM Overview

Design, implementation and testing of an acquisition system that measures the particle loss rate.

Main components of the system: 3700 Detectors Current to Frequency Converters (CFCs) Analogue to Digital Converters (ADCs) Tunnel FPGAs: Actel’s 54SX/A radiation tolerant. Communication links: Redundant Gigabit Optical Links. Surface FPGAs: Altera’s Stratix EP1S40 with 780 pin.

28

Tunnel Card BLMCFC

1 23 . . .

28

Ionisation Chamber

Patch Box

Surface Card BLMTC (DAB64x)

GOH

GOH

Surface FPGA

MEMORY 16 bits

16 bits

16 bits

Anti-Fuse FPGA

Control

Control

16 bits

Control

Mezzanine

SRAM (1)Acquisition Data

SRAM (2)Acquisition Data

SRAM (3)Post-Mortem

data

data

data

Power PC

Control

Data

VMEInterface

Control

Data

Control

Control

Control

Control

1 23 . . . 8

2|||||||||||||

|||||||||||||||

|||||||||||||||

|||||||||||||||

CFC Analogue

. . .

Control

TLKOptical receiver

16 bits

Control

TLKOptical receiver

16 bits

Control

TLKOptical receiver

16 bits

Control

TLKOptical receiver

16 bits

Control

GOL

GOL

Tunnel Card BLMCFC

GOH

GOH

16 bits

16 bits

Anti-Fuse FPGA

Control

Control

1 23 . . . 8

2|||||||||||||

|||||||||||||||

|||||||||||||||

|||||||||||||||

CFC Analogue

. . .

Control

GOL

GOLIonisation Chamber

Combiner CardBLMCOM

DUMP (x3)

BEAM ENERGY

BEAM PERMIT

Beam Interlock Controller

BIC DUMP (x2)

BEAM PERMIT

Beam Energy Tracker

BETBEAM ENERGY

optical fibres

VM

E6 4

x B

us

Bac

kp

lan

e

DUMP (x3)

BEAM ENERGY

BEAM PERMIT

Control

Data

Tunnel max 2km Surface

max 300m

Ionisation Chamber

... ...

Ionisation Chamber

Post-Mortem

DataLogging

Data

17 March 2006

Christos Zamantzas4

Surface Card (BLMTC)

Stratix Device Features

Features of EP1S40 Device Available Used

Logic Elements 41,250 35,475 (86%)

M512 RAM blocks (32 ×18 bits) 384 299 (78%)

M4K RAM blocks (128 ×36 bits) 183 174 (95%)

M-RAM blocks (4K ×144 bits) 4 0 (0%)

DAB64x specifications

Stratix FPGA Vertical Migration to EP1S40

SRAM memories (x3) 512K x 32bit

Connectors for Mezzanine 2 x 64pin PMC connectors Provide 3.3V, 5V, GND, and Connection to 114 FPGA I/O pins.

Card bus VME64x

Surface Card BLMTC (DAB64x)

FPGAALTERASTRATIX

EP1S40

MEMORY 16 bits16 bits

Control

Mezzanine

SRAM (1)Post-Mortem A

SRAM (2)Post-Mortem B

SRAM (3)Unused

data

data

data

Control

Data

Config Device &

VME Interface

Control

Data

Control

Control

Control

Control

DUMP

BEAM ENERGY

BEAM PERMIT

TLKOptical receiver

16 bits

Control

TLKOptical receiver

16 bits

Control

TLKOptical receiver

16 bits

Control

TLKOptical receiver

16 bits

Control

Config Memory8 bits

Da

ta

Control

Co

ntr

ol

17 March 2006

Christos Zamantzas5

Processes running in the Surface FPGA Receive, Check & Compare (RCC)

Receives, De-serialises and decodes the transmitted packets.

Checks for errors. Compares the packets.

Data-CombineMerges the ADC and the CFC data into

one value. Successive Running Sums (SRS)

Produces and maintains various histories of the received data in the form of Moving Sum Windows.

Threshold Comparator (TC)Compares the sums with threshold

limits. Masking

Distinguishes between “Maskable” and “Not-Maskable” channels.

Supervision and Logging Error & Status Reporting (ESR) Maximum Values of the SRSs.

Processing Block LE M512 M4K

RCC 4 % 8 % -

DataCombine 4 % - -

SRS 30 % 70 % 57 %

TC & Masking 13 % - 35 %

ESR 4 % - 1 %

MAX Values 31 % - 2 %

Total Resources Used 86 % 78 % 95 %

Tables

TC &

Masking

Surface FPGA

Pri

mar

y S

ign

al A

Bea

m P

erm

it

(No

t M

aska

ble

)B

eam

Per

mit

(M

aska

ble

)

Error & StatusReporting

Pri

mar

y S

ign

al B

Red

un

dan

tS

ign

al B

RCC...

Data Combine

Data Combine

Successive Running Sums ...

Data Combine

.

.

.

Successive Running Sums

Successive Running Sums

Red

un

dan

tS

ign

al A

......

Mux

LoggingBeam Permit (Not Maskable)

MAXValues

Lo

gg

ing

VME NVRAMBeam Energy

VM

E

17 March 2006

Christos Zamantzas6

Real-Time Analysis of Data

Running Sums Multi-point Shift Registers holds data Successive calculation of Running Sums Range 40μs - 84s (12 Running Sums) Maximum values of the last second are continuously calculated and kept.

Running Sum

Running Sum

R.Sum 01

R.Sum 02

R.Sum 03

R.Sum 04

Running Sum R.Sum 05

R.Sum 06

R.Sum 10

R.Sum 11

NextBlock

.

.

.

Data

Successive Running Sums

New Value

Accumulate

Shift Register

V(n-64)

V(n-128)

A A-B B

A A-B B

signed

Vn

Vn

Acc

Acc

Subtract

signed

Running Sum Configuration

R.Sum 02

R.Sum 01

Running Sums RefreshingShift

Register Name

Signal Name

bits used40 μs

stepsms

40 μs steps

ms

1 0.04 1 0.04   RS 00 20

2 0.08 1 0.04   RS 01 22

8 0.32 1 0.04SR1

RS 02 22

16 0.64 1 0.04 RS 03 22

64 2.56 2 0.08SR2

RS 04 26

256 10.24 2 0.08 RS 05 26

2048 81.92 64 2.56SR3

RS 06 32

8192 327.68 64 2.56 RS 07 32

32768 1310.72 2048 81.92SR4

RS 08 36

131072 5242.88 2048 81.92 RS 09 36

524288 20971.5 32768 1310.72SR5

RS 10 40

2097152 83886.1 32768 1310.72 RS 11 40

Table 1: Successive Running Sums configuration

17 March 2006

Christos Zamantzas7

Threshold Comparator & MaskingThreshold Table

Threshold depends on energy and a acquisition time.

Unique thresholds for each detector. Read at system power-on from an external

Non Volatile RAM or it can be included in the Configuration file.

Space required: 32 KB

Masking Table Masking of channels is allowed only when safe. Unique masking for each detector. Read at system power-on from an external Non

Volatile RAM or it can be included in the Configuration file.

Width (Bits) Detectors Energy Levels Running Sums Values Total (Bits)

32 16 32 8 4,096 131,072

64 16 32 4 2,048 131,072

Total 12 6,144 262,144

Detector Connected Maskable

1 No Yes

2 No Yes

3 Yes No

… … …

15 Yes No

16 Yes Yes

Table 1: Memory Utilisation by the Threshold Table

Table 2: Information included in the Masking Table Threshold Comparator

Th20

text

text

text

Thresholds

Warnings

...

Level 1

Level 2

Level 3

Level 32

Threshold Table (NVRAM and FPGA)

.

.

.

Bea

m P

erm

it

(No

t-M

aska

ble

)B

eam

Per

mit

(M

aska

ble

)

text

Masking Table (NVRAM and FPGA)

Channel 1Channel 2Channel 3

.

.

.Channel 8

Th

. . .

Running Sum 01

Running Sum 02

Running Sum 11

.

.

.

Beam Energy

40

5

Masking

22

Dump Request

17 March 2006

Christos Zamantzas8

Scaling Down the Threshold Values (Proposal)

Threshold values are: Unsigned binary numbers 32 or 64 bit long Division by 2 can be achieved by Shifting Right by 1 position.

A simple solution could be to provide a register that holds information about the number of bits wanted to be shifted right (i.e. divide by 2, 4, 8, 16, …).

Can be unique for each channel. Its simplicity requires minimum resources. Secure enough (not possible to increase Thresholds). Needs to be accessed by the CPU and loaded at every boot. Adds dependency to the CPU that is against specifications.

17 March 2006

Christos Zamantzas9

Configuration of the DAB64x Configuration Data : FPGA Firmware Card ID

Threshold Table Masking Table

17 March 2006

Christos Zamantzas10

Configuration Scheme (I)

All included in the FPGA Firmware After the system design finishes the extra information will be embedded in the firmware using a final iteration in the Quartus software.

Single file to be deployed on each card. System “instantly” ready after power-on. Feature already available/operational.

Legend

Deactivated

Booting

Initialisation

17 March 2006

Christos Zamantzas11

Configuration Scheme (II)

Split of Configuration Memory Global FPGA Firmware Unique information will be stored in the unused space of the configuration memory. After the FPGA boot it will have to fetch the data from the

Multiple files to be deployed for each card. More complicated booting procedure. Future independence from Quartus Software (?).

Legend

Deactivated

Booting

Initialisation

17 March 2006

Christos Zamantzas12

Configuration Scheme (III)

Use of both Flash Memories Global FPGA Firmware resides on-board. Unique information will be stored in the mezzanine card’s NVRAM.

Multiple files to be deployed for each card. Multiple interfaces to be created.

Legend

Deactivated

Booting

Initialisation

17 March 2006

Christos Zamantzas13

Outlook Provide at least one more configuration scheme.

Preferably the partition of the configuration memory. Implementation of the Post-Mortem data recording and

readout. Implement differently the Data-Combine process to match

better with the CFC card. Not consistent in the region below one count/sec.

Finalise specifications for the Supervision and Testing procedures. To be proposed at every start-up the Combiner card to increase in steps

the Beam energy input. The Logging procedure will provide to the CPU the Thresholds which will

be used for that given Beam energy. The CPU will be able to compare them with those stored in a database.

Test of a complete crate filled with pre-series boards. Setup to be used also by the software people implementing the fixed

displays, the Logging and later the Post Mortem.

Finish the production of the BLM Mezzanine provide a testing procedure of the cards to be installed.

17 March 2006

Christos Zamantzas14

Reserved Slides

17 March 2006

Christos Zamantzas15

Packet from Transmission

Updated formatting of the packet for transmission (now 320 bits) Extended the status information Included the DAC Values of each channel.

Transmission of packet every 40μs. The data rate must be high enough to minimise the total latency of the system .

Redundant optical link In order to increase the reliability and the availability of the system.

CFC(1)8bit

ADC(1)12bit

CFC(2)8bit

ADC(2)12bit

…CFC(8)

8bitADC(8)

12bit

Frame ID

16bits

Status A

16bits

Status B

16bits

Counter + ADC Data

64bits + 96bits

DAC Values

64bits

Card ID

16bits

CRC-32

32bit

Transmitted

Packet

Data

17 March 2006

Christos Zamantzas16

Logging Data Arrangement

These data have to be read with a rate of a second in order to be stored in a database as well as to give a

graphical representation for the control room.

Error & Status Reports (ESR)(1,024 bits)

Maximum Values of the BLM Data

(8,192 bits)

Threshold Values used(8,192 bits)

LWords

1

2

3…

3233

…

288

289

…

544

1 2 3 … 16 … 32

Bits

Read every second:

Error & Status Reports (ESR)(128 Bytes) The Maximum Values of the BLM Data (1 KB) The used Threshold values. (1 KB)

Read every Start-up

Each card’s complete Threshold table. (32 KB) (4,096 x 32bit + 2,048 x 64bit = 6,144 values)

17 March 2006

Christos Zamantzas17

Error & Status Reports (ESR)

Information available: ERRA: number of CRC errors at optical link A

ERRB: number of CRC errors at optical link B

ERRC: different CRCs between A and B

ERRD: wrong Card ID

ERRF: wrong Frame ID

Dump: number of beam dump triggers

LostFramesA & LostFramesB

Card ID Each Detector’s DAC value Voltage and temperature Status

17 March 2006

Christos Zamantzas18

Post Mortem Data Recording (I)

Two circular buffersA. 2000 turns of both signals received

B. Integrals of 10 ms (data needed for further analysis)

Double the above buffer and toggle between them using the stop PM recording trigger

Never stop recording (i.e. avoid start input) Test of PM will be possible anytime Accidental/error-triggering proof

Circular Buffer A

Decoder

Data for

PM

MUX Data output

Buffer selectPM TriggerToggle

Circular Buffer B

WriteEnable

WriteEnable

17 March 2006

Christos Zamantzas19

Post Mortem Data Recording (II)

PM freeze from TTCrx through the backplane. Time-Stamp appended later by crate CPU.

At PM freeze the CPU records the time and later when it reads the PM Data appends it to them.

Calculations: Minimum required: 1000 turns

=> 2000 acquisitions * 320 bits packet * 2 packets

=> ~ 160 KB/card * 16 cards/crate = 2.5 MB / crate

Available:

3xSRAM chips of 512K x 32bit = 6 MB / card