DO

E S

BS

Review

/ SB

S D

AQ

Outline

• Experiments Setup and DAQ configuration

• Fastbus for SBS: performance and status

• GEn/GMn - trigger and DAQ

• GEp - trigger and DAQ

• GEM readout

SBS DAQ

E. Cisbani / INFN Sanità

SBS DOE Review

4-5/Nov/2014 - JLab

4/No

v/2014

1

Main guidelines

- Reuse available equipment (Fastbus) to reduce cost

- Exploit JLab CODA3 VME hardware (FADC ...)

- GEM readout based on APV25 + MPD

Contributions from:• Sergey Abrahamyan• Alexandre Camsonne• Mark Jones• Bob Michaels• Paolo Musico• Igor Rachek• …

Experimental Setup4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

2

Neutron form factor (GEn/GMn)Reaction : Quasifree electron

scattering on 3He or 2HTrigger: Single arm electronElectron singles rate: <5 kHzElectron arm:

• BigBite Magnet• 4 GEM chambers (FT)• Gas Cerenkov (GRINCH)• 1 Large GEM chamber (BT)• Scintillator paddle array• Preshower/Shower Calorimeter

Hadron Arm:• Super BigBite Magnet• Coordinate Detector• Hadron Calorimeter

Proton form factor (GEp)Reaction : Elastic electron-protonTrigger: Elastic ep coincidenceElectron singles rate: 200 kHzHadron singles rates: 2 MhzCoincidence trigger rate: 5 kHzElectron arm:

• Coordinate Detector• Electron Calorimeter

Hadron arm:• Super Bigbite Magnet• Front GEM tracker (FT)• Analyzer• 5 Rear GEM tracker (BT)• Analyzer• 5 Rear GEM tracker (BT)• Hadron Calorimeter

DAQ configuration for SBS experiments4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

3

• Reuse the NIM and Fastbus equipment already available at Jlab

• Exploit the new JLab CODA VME hardware for the «rest»

FASTBUS for SBS experiments4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

4

Struck Fastbus Interface (SFI) is the Fastbus Master (18 available at JLab)• Allows control the Fastbus modules through any VME CPU.• Had slot for standard JLab Trigger Interface Module

64 channel Lecroy ADC 1881M (113 available at JLab)• 9ms encoding time in 12 bit resolution and 12ms in 13 bit resolution.• GEp experiment will use the fast clear feature in which the module is ready to accept another

event after 1ms.

96 channel Lecroy TDC 1877s (236 available at JLab)• Built-in Data Zero Suppression and Data Compaction (sparsification)• Capable of multihit with an event buffer of 8 events. • Encoding time 1.7ms plus 50 ns per hit per channel giving a maximum encoding time of 78ms.• Fast clear settling time < 250ns

Fastbus Crates holds up to 25 modules (30 available at JLab)

Plenty of FASTBUS modules but:Fastbus standard transfer rate: 40 MB/s (sustainable 15 MB/s)

25% dead time at 5 kHz

Need to reduce Fastbus dead time!

DO

E S

BS

Review

/ SB

S D

AQ

Comm

on Stop

background

signal

window 0-32 µs

sparsifyGate

Triggers

ReadoutOverhead

Triggers

ReadoutOverhead

II. Event Blocking (8 events in TDC and ADC)Blocklevel=4 should work with pipelining VME

Buffers the deadtimeand reduces overhead

I. Sparsification (built-in feature in TDC and ADC)

Throws out background hits

III. Event Switching 3 parallel crates, triplicate equipment, but reduces rate by 3

Status: tried, works

Status: tried, works

Status: test about to start(expected to be straightforward)

Sergey Abrahamyan

Igor Rachek

Making Fastbus Faster4/N

ov/2014

5

4x(20+50) us

20+4x50 us

DO

E S

BS

Review

/ SB

S D

AQ

Fastbus expected performance and status

6

We can merge Fastbus with the rest of

the DAQ if:

• All components use blocklevel = 4

• All crates conform to the CODA

standard.

Needs to be tested

~10% deadtime at 20kHz

For a simple level-1 trigger

4/No

v/2014

Two large Fastbus systems are being assembled for test in the test lab!

TDC

ADC

Crate

SFI

Have 236 113 30 18

Need 124 94 21 21

DO

E S

BS

Review

/ SB

S D

AQ

Neutron Form Factors (GEn / GMn)4/N

ov/2014

7

Neutron form factor (GEn/GMn)Reaction : Quasifree electron

scattering on 3He or 2HTrigger: Single arm electronElectron singles rate: <5 kHzElectron arm:

• BigBite Magnet• 4 GEM chambers (FT)• Gas Cerenkov (GRINCH)• 1 Large GEM chamber (BT)• Scintillator paddle array• Preshower/Shower Calorimeter

Hadron Arm:• Super BigBite Magnet• Coordinate Detector• Hadron Calorimeter

BigBite Electron Single arm trigger (GMn,GMn)4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

8

Use existing NIM logic for preshower/Shower coincidence

BigBite Shower Trigger4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

9

S7 2 x S7 + S4

Shower7 x27

Preshower2 x27

S4

To discriminator

Bigbite trigger is ORof discriminated superblocks of

Shower+

Preshower

27 rows

27 rows

GEn and GMn: Hadron Arm DAQ4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

10

• 2 VME switched Serial (VXS) Crates

• JLAB FADC250, 16-channel 12-bit

FADC sampling at 250 MHz

• Capable of 300 ps time

resolution in Hall D tests

• TS ( Trigger Supervisor)

• Accepts electron arm trigger

• Check status of all ROCs

• Outputs L1 accept as

• stop to TDCs

• gate for ADCs

• Readout signal for GEM MPDs

• Readout signal to HCAL TI

GEM/MPD’s

SSP

Electron ArmTrigger

Optical Link

L1A

L1A

See more, next in the GEp DAQ

DO

E S

BS

Review

/ SB

S D

AQ

Proton Form Factor (GEp)4/N

ov/2014

11

Proton form factor (GEp)Reaction : Elastic electron-protonTrigger: Elastic ep coincidenceElectron singles rate: 200 kHzHadron singles rates: 2 MhzCoincidence trigger rate: 5 kHzElectron arm:

• Coordinate Detector• Electron Calorimeter

Hadron arm:• Super Bigbite Magnet• Front GEM tracker (FT)• Analyzer• 5 Rear GEM tracker (BT)• Analyzer• 5 Rear GEM tracker (BT)• Hadron Calorimeter

GEp: Electron Trigger4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

12

• Sum analog signals to form “superblock” of 4x8 blocks

• Total of 204 “superblocks” go to discriminators with threshold of 80-90% of elastic maximum energy

• L1 trigger is OR of the 204 superblocks logic signals

• 204 superblock signals sent to L2 trigger processor→ next slide

Elastic electrons at Q2 = 12 at the calorimeter

Ethr/Emax

(%)L1 Rate(kHz)

Data Rate(Mb/s)

50 3500 1400

75 320 128

85 120 48

50 ECAL blocks

GEp: Hadron Arm / HCAL DAQ and trigger4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

13

HCal Signals to FADC inputs

Same acquisition scheme of GEn, GMn

Integrated signal and timing from FADC channels collected and sent to a General Trigger Processor (GTP) every 32 ns (over optical link)

GTP

compute all 4x4 sums of adjacent channels (HCAL clusters)

get electron Arm cluster information (204 superblocks signals)

check angular e-p correlation

If correlation send level 2 trigger to Trigger Supervisor → next slide

But now HCAL is in the trigger logic:

Gep: DAQ Configuration / both arms4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

14

Trigger Supervisor• Globally controls readout of all crates• Receives T1 from ECal sends L1 to FASTBUS crates. The V1495

selects which group of FB crates to read out.• If T2 from GTP arrives then readout of VME crates and FASTBUS

( event buffer size of 4)• If no T2 then FASTBUS crates get Fast Clear

e-arm T1 triggers processing

for generation of T2

T2 triggers acquisition

GTP

(GEM)

DO

E S

BS

Review

/ SB

S D

AQ

15

GEM – Readout Electronics4/N

ov/2014

• Up to 16 APV25 cards (2048 chs) on a single MPD (parallel readout)

• Altera Arriga GX FPGA / RAM: DDR2 (128 MB)• Optical Link interface (either ETH 1Gb/s or Aurora 3.125

Gb/s protocol)• 110 MHz system clock and Front panel coax clock• Used HDMI-A connectors only for analog and digital signals• SD-card / All spare signals go to PMC compliant connectors• VME/32, VME64, VME64-VXS compliant• 4 high speed line on the VXS available for data transfer

Channels

APV25

MPDs

Front Tracker

41472 324 24

Rear Tracker

61440 480 34

• 128 analog ch / APV25 ASIC

• 3.4 ms trigger latency (analog pipeline)

• Capable of sampling signal at 40 MHz

• Multiplexed analog output (100 kHz readout

rate)

MPD

DO

E S

BS

Review

/ SB

S D

AQ

MPD v4.0 VME interface performance• All VME cycles tested including 2eSST (with STRUCK SIS-3104)

– 2eSST supported by new firmware release

– Readout speed measured by software: 100 transfer 4MB each. Data integrity checked for each block.

– Bus speed is measured directly on VME bus

CYCLEDATA period

[ck] / [ns]

Bus / Simulated / Measured Speed[MB/s]

BLT (32 bit) 16 / 150 26.6 / - / 24.3

D64-MBLT 17 / 159 50.3 / - / 47.8

2eVME 20 / 186 86.1 / - / 73.6

2eSST160 6 / 54 142 / 148 / 117

2eSST267 4 / 36 213 / 222 / 124

2eSST320 3 / 27 284 / 296 / 124

Speed limited by SIS3104 2Gb/s

fiber connection

• Optical link with ETH 1Gb tested

• Optical link with Aurora protocol connected to SSP to be tested (no surprise expected): speed up to 390 MB/s (sustainable 200 MB/s)

sustainable 200 MB/s

4/No

v/2014

16

DO

E S

BS

Review

/ SB

S D

AQ

Strip Occupancy: 60%Single MPD Transfer Rate: 45.2 MB/s (after sparsification)

200 kHz Rear Tracker – same occupancy and transfer rate

CDR rate estimation

• Expected Hits Rate (Front Tracker): 500 kHz• Samples/Events: 3• GEM signal width: 250 ns• Cluster width: 2.5 strips• Trigger rate: 5 kHz

conservative

APV25 signal output

Real time data reduction needed !

Cluster Width

4/No

v/2014

17

GEp: GEM readout performance

Pure VME64 (original design)

MPD + Optical Link + SSP + VME64

MPD/VME64 = 4need 15 VME64 crates

MPD/SSP=4, SSP/VME64=1need 15 SSP, 15 VME64 crates

DO

E S

BS

Review

/ SB

S D

AQ

GEM Making Data smaller

• Deconvolution logic in the MPD (approx. 40% FPGA resource available)– Deconvolution algorithm to time-correlation

at the level of 1/3 * 250 ns 80 ns(can even use larger number of samples) reduce data by a factor of 3

• SSP additional processing of the data– Geometrical correlation using BigBite & ep scattering & HCAL

information reduce the «signal area» to 40x40 cm2 (even smaller) keep only information of 8+8 cards/chamber; reduce data by a factor of 3 or more

– Further processing likely possible in SSP (e.g. clustering with x/y charge correlation ...)

4/No

v/2014

18

Single MPD transfer rate = 45.3 / 3 = 15 MB/sSingle SSP transfer rate with 32 MPD = 15 * 32 / 3 = 161 MB/s

Two SSPs on two separate VME64 crates

DO

E S

BS

Review

/ SB

S D

AQ

DAQ: Man power4/N

ov/2014

19

Coordination

VME-DAQ

Fastbus

HCAL-Trigger

GEM-MPD

A. Camsonne X X X X

M. Jones X

D. Adikaram X

R. Michaels X

B. Moffit X

B. Raydo X

V. Bellini X

E. Cisbani X

P. Musico X

J. Campbell(SMU

student)

X

+ Support from JLab CODA and Electronics group

DO

E S

BS

Review

/ SB

S D

AQ

DAQ: Short Term plan

• Fastbus

– Crates switching test in progress: 3 months

• MPD

– Integration in CODA (partially done) need additional 6 weeks

– Deconvolution algorithm: 4 months

– Optical link protocol with SSP (proto-firmware of Aurora available but not implemented): 4 months

• SSP

– SSP available, processing firmware development in 2015

• HCAL

– trigger development and testing: 6 months

• Small scale full setup Fastbus + HCAL trigger in 2015

4/No

v/2014

20

DO

E S

BS

Review

/ SB

S D

AQ

21

Summary4/N

ov/2014

Neutron Experiments Proton Experiment

• Single arm BigBite electron trigger at

low rate: reuse existing NIM / Fastbus

setup

• High rate T1 triggers formation of T2 on

hadron arm

• Low rate L2 generated from: ECAL &

HCAL & ep angular correlation →

trigger readout

• Fast Clear on Fastbus (if no L2)

Fastbus:• Plenty of ADC, TDC and Crates

• Use sparsification, event buffering and crate switching to reduce dead time

VME: • FADC amplitude and time information for HCAL

• MPD + Optical Link + SSP for GEM (smart processing in MPD and SSP to reduce

data by 1/10)

JLab CODA3 hardware/software framework

4/No

v/2014D

OE

SB

S R

eview / S

BS

DA

Q

22

Backup .....

GEp Electron Trigger (part 2)4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

23

Groups of 32 or 4x8 Combine four of the “groups of 8” analog sums in

linear FI/FO to get summed signal for “group of 32” or 4x8 blocks

Each “group of 32” overlaps by a “group of 8” in the horizontal and vertical direction.

Total of 204 “groups of 32” summing analog signals which go to discriminators. Send to 16 channels discriminator in groups with match momentum

Set discriminator to 80-90% of elastic maximum energy

Level One trigger (T1) is OR of the 204 “group of 32” logic signals and his gate sent to TI and then to FASTBUS ADCs and TDCs

204 “group of 32” logic signals sent to FPGA to determine angular correlation with the HCAL and form the Level 2 trigger.

If no coincidence trigger then Fast Clear sent to FASTBUS crates.

Hadron Arm - HCAL DAQ: proton trigger4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

24

• 2 VME switched Serial (VXS) Crates• JLAB FADC250, a 16-channel

12-bit FADC sampling at 250 MHz•16 FADC in VXS Crate 1

• 2 FADC in VXS Crate 2• If signal pass threshold

• Integrates signal and subtracts pedestal• Sends time frame info

• CTP (Crate Trigger Processor)• Located in VXS crate 1• Collects integrated signal and timing from FADC

channels• Sends data to SSP in crate 2 for processing over optical

link

HCal Signals to FADC inputs

HCAL DAQ4/N

ov/2014

DO

E S

BS

Review

/ SB

S D

AQ

25

• CTP (Crate Trigger Processor)• Located in VXS crate 1• Collects integrated signal and timing from FADC

channels• Sends to SSP in crate 2 for processing over optical

link• SSP (Sub System Processor)

• Collects crate 1 CTP data • Other 2 SSP for processing GEM data

• VETROC• Input register for ECAL logic signals

• GTP (Global Trigger Processor)• Located in VXS crate 2• Reads HCal info from SSP and 2 FADCs• Reads Ecal info from VETROC• Forms HCAL clusters and checks angular correlation

with ECAL• In good HCAL-ECAL coincidence then sends trigger

signal T2 to Trigger Supervisor• Trigger Supervisor

• Globally controls readout of all crates• Receives T1 from ECal sends L1 to FASTBUS crates.

The V1495 selects which group of FB crates to read out.

• If T2 from GTP arrives then readout of VME crates and FASTBUS ( event buffer size of 4)

• If no T2 then FASTBUS crates get Fast Clear

GT

P

204 Groups of 32 from ECAL

Extensive use of JLab CODA hardware

DO

E S

BS

Review

/ SB

S D

AQ

Fastbus: GEp is the most demanding

26

For Fastbus, the plan is to use second-level (L2) triggers to issue a fast clear.

Singles L1 trigger rate 100 – 300 kHz depending on thresholds.L2 rejection 80% or greater.

trigL

rejectLe

1

2

L1

4/No

v/2014

MPD (ADC + Controller) Block Diagram

EP1AGX60DF780C6N

4/No

v/2014D

OE

SB

S R

eview / S

BS

DA

Q

27

DO

E S

BS

Review

/ SB

S D

AQ

MPD Event Builder

• Implemented multi event block structure as suggested by DAQ people.

• Native data width: 24 bit, packed to 32 bit on 64 bit boundary for efficiency.

• Implemented 128MB FIFO data buffer using DDR2 SDRAM.

• Quite complicate machinery used to arbiter read and write to/from DDR2.

• Output can be read in 32/64 bit format in any of the supported VME cycles, including 2eSST.

• Performed functional simulation (FPGA + DDR2 + ADC) of quite simple events to follow all the signals.

• Some effort has to be put in DAQ driver to recover packed data.

4/No

v/2014

28