Jan.20, 2004

Hawaii, USA

Super B Factory Workshop

Takeo HiguchiInstitute of Particle and Nuclear Studies, KEK;

On behalf of COPPER working group

Readout Electronicsfor the SuperKEKB

Contents

• Introduction• Conceptual design• The COPPER-II• Readout Scheme• COPPER-II Performance• Schedule• Summary

Requirements for Readout Electronics

TriggerSystemTriggerSystem

ReadoutElec.

ReadoutElec.

EventBuilderEvent

Builder

StorageStorage

Detector

DAQ

Trigger rate•> 10 kHz

Data size•~ 300 kB/ev

Readout•~ 50 k channels

Trigger rate•> 10 kHz

Data size•~ 300 kB/ev

Readout•~ 50 k channels

SuperKEKB

Also good for J-PARC, K2K …Also good for J-PARC, K2K …

Conceptual Design

What are keys in designing the readout system?

System Design

• Event buffers for pipelined DAQ

• Data size reduction by processor on data path

ProcessorProcessor

Readout FIFOReadout FIFO

DigitizationDigitization

Data path

Consequent designof readout electronics

from detector

to EB

Design from Cost View

• Flexible system– Common platform + user-defined ADC/TDC.– Implementation of ADC/TDC and CPU as modules.

• Compact design– Less #-of-crates to house platform boards.– Smaller room of electronics hut.

• Commodity usage– Good products at low price in computer market.– Easy to follow/import newest technology.

Design Overview

PMC CPUPMC CPU

local bus PCI bus

Det

ecto

r si

gnal

s mezzanine(add-on) module

Network IFNetwork IF

Trigger Module

Trigger Module

ADC/TDC Module

ADC/TDC Module

ADC/TDC Module

ADC/TDC Module

BridgeBridge

PMCFIFO

Conceptual Design (Front-end I/F)

• User-defined ADC/TDC module

sampling clock

Detectorsignal

0xaa550x02460xf3b7 …

ReadoutFIFOADC/TDCADC/TDC

L1 triggerpipeline

L1 triggerpipeline

trigger signal

Add-on type Module

flexibilityflexibility

compactnesscompactness

Conceptual Design (L1 Trigger Pipeline)

Digitized data in

Data out(to Readout FIFO)

Alw

ays

WR

ITE

busy

Dat

a R

EA

D

Tri

gger

HO

LD

switch

Dat

a R

EA

D

Alw

ays

WR

ITE

Dat

a R

EA

D

switch

EM

PT

Y

L1 trigger Ready

pipelinepipeline

Conceptual Design (FIFO)

Detector

Detector

Detector

Detector

Local bus

BridgeBridgePCI bus

ADC/TDC Module

ADC/TDC Module

ADC/TDC Module

ADC/TDC Module

trigger signal

DMA

32

pipelinepipeline

Conceptual Design (Back-end)

• Data path = PCI bus• PCI mezzanine modules (PMC)

BridgeBridge

CPUCPU

TriggerTrigger

NetworkNetwork

EB

PCI bus

Data suppression

DMA

PCI = Up to 133 MB/sPCI = Up to 133 MB/s

trigger signal

Local bus

online data reductiononline data reduction

flexibilityflexibility

commodity usecommodity use

compactnesscompactness

Conceptual Design (Processor)

• Modular CPU– Modularity = easy to upgrade.– Intel x86 series: most familiar architecture.

• Operating System = Linux– Easy to develop device driver / software.– Low price.

flexibilityflexibility

commodity usecommodity use

PCI Ethernet card PMC Ethernet card

What’s ‘PMC’? - PCI Mezzanine Card Standard

– PMC is 100% compliant with the PCI.– PMC is suitable for high density applications.– Many commercial products are available: Ethernet cards, G

bE cards, memory modules, CPUs, etc.

74×149 mm2

PMC Size CPU

• RadiSys EPC-6315– Equipped with Intel PentiumIII 800 MHz.– RedHat Linux 7.3 runs.– 32-bit 33/66 MHz PCI bus interface.

You can login this CPU like your PC

Even physics-analysis codes run on it!Contains Linux image

Whole Design of the DAQ Platform

PMC CPUPMC CPU

local bus PCI bus

Det

ecto

r si

gnal

s mezzanine(add-on) module

Network IFNetwork IF

Trigger Module

Trigger Module

VME 9U sized board

ADC/TDC Module

ADC/TDC Module

ADC/TDC Module

ADC/TDC Module

BridgeBridge

PMCFIFO

The COPPER-II

This is our readout platform for the Super B factory.

The COPPER-II

PMCProcessor

Trigger

GenericPMC slot

ADC/TDC

ADC/TDC

ADC/TDC

ADC/TDC

VME-9U sized boardVME-9U sized board

•4 ADC/TDC slots•1 MB × 4 FIFO•32-bit local bus•3 PMC slots•32-bit 33MHz PCI bus•Local-PCI bridge•VME I/F

•4 ADC/TDC slots•1 MB × 4 FIFO•32-bit local bus•3 PMC slots•32-bit 33MHz PCI bus•Local-PCI bridge•VME I/F

COmmon Pipelined Platform for Electronics Readout

On-board Ether

ADC/TDC Module

Local bus I/F(for control)

Digitizedsignal output

Trigger busyClock, L1 trigger

Clock = 43MHz,Reduced clock = depends on sub-system

32

Photo of prototype card

76×180 mm2

Readout Scheme

How it works?

Readout Scheme (1)

• L1 Trigger comes…– Trigger module distributes trigger signal to the ADC/TDC modules.– Digitized data are held on the ADC/TDC module.– Data transfer to the COPPER-II starts.

• COPPER-II FIFOs are filled…– The COPPER-II’s FPGA formats data (header and footer).– The COPPER-II’s FPGA counts each event size in FIFOs.– A PCI interrupt is generated when a total data size in the FIFOs excee

d threshold to initiate the DMA.

Readout Scheme (2)

• PCI interrupt is generated…– The CPU checks the size of stored data counted by the COPPER-II.– The CPU starts data transfer from the FIFOs to CPU’s main memory u

sing DMA.

• A user process on the CPU is woke up after DMA…– The user process reads data and performs the data size reduction.

• The data are transferred to the event builder…

COPPER-II Performance

Up to what input rate does COPPER-II work?

Study Setup

ADC proto.ADC proto.

ADC proto.ADC proto.

ADC proto.ADC proto.

ADC proto.ADC proto.Vacant

TriggerTrigger

CPUCPU

Ether

RX

The COPPER-II

Generates virtual ‘ADC’ data.Data size / event is variable.

Self trigger.Trigger rate is variable.

Maximum Trigger Rate

Typical trigger rate

Required trigger rate

@ 416 bytes/ev/ADC-module

The COPPER-II works> 30 kHz input rate

The COPPER-II works> 30 kHz input rate

Belle’s TDC data16 bytes / ch. (drift chamber)

400 bytes/TDC-module corresponds to 100 ch. in one COPPER-II module.

Belle’s TDC data16 bytes / ch. (drift chamber)

400 bytes/TDC-module corresponds to 100 ch. in one COPPER-II module.

Profile of CPU Usage

• User time: ~2%• System time: ~20%• Idle time: ~78% = CPU power that is equivalent to

P3 ~600MHz is still available

• Large idle time fraction indicates the PCI bus works at the full performance.

– 416 bytes / ADC-module / ev × 40 kHz × 4 ADC modules= 67 MB/s.

@ 416 bytes/ev/ADC-module

Performance Degradation by Network Use@ 416 bytes/ev/ADC-module

CPUCPU

CPUCPU

FIFOReadFIFORead

FIFOReadFIFORead

NetworkTransferNetworkTransfer

proc.A

proc.Aproc.B

RX

Ethernet 11MB/s

CPU user time: ~2%CPU system time: ~20%CPU idle time: ~78%

Maximum acceptedtrigger rate: 40 kHz

CPU user time: ~2%CPU system time: ~20%CPU idle time: ~78%

Maximum acceptedtrigger rate: 40 kHz

CPU user time: ~5%CPU system time: ~31%CPU idle time: ~64%

Maximum acceptedtrigger rate: 32 kHz

CPU user time: ~5%CPU system time: ~31%CPU idle time: ~64%

Maximum acceptedtrigger rate: 32 kHz

Still works well.

Schedule

What are we planning?

Schedule in Near Future

• The COPPER-II prototype is being debugged.• We start production of the COPPER-II from the begging of

Feb.2004.

• We are planning to replace some Belle-DAQ parts with COPPER in 2005 summer.

Long Term PlanCOPPER Proto.

Upgrade

Integrated Test

Goal

Per

form

ance

Stu

dy

Per

form

ance

Stu

dy

AS

IC D

evel

opm

ent

AS

IC D

evel

opm

ent

AD

C M

odu

le D

esig

nA

DC

Mod

ule

Des

ign

TD

C M

odu

le D

esig

nT

DC

Mod

ule

Des

ign

Mas

s C

PU

Con

trol

Mas

s C

PU

Con

trol

Use

r I/

F F

ram

ewor

kU

ser

I/F

Fra

mew

ork

Eve

nt

Bu

ildin

g F

arm

Eve

nt

Bu

ildin

g F

arm

L2/

L3

Tri

gger

sL

2/L

3 T

rigg

ers

Pla

tfor

m

Har

dwar

e A

ppli

cati

ons

Sof

twar

e A

ppli

cati

ons

ADC/TDC Proto.We are here.

Summary

• We have developed new readout electronics, COPPER-II, for the Super B factory.

• Design keywords are:Pipeline, Online data reduction,Flexibility, Compactness, and Commodity usage.

• At the typical data size, the COPPER-II works up to 40 kHz trigger rate, which satisfies the requirements.

Backup Slides

Please help me…

Introduction

What is a readout electronics?What is Super B factory requiring for it?

Introduction to Readout Electronics

TriggerSystemTriggerSystem

ReadoutElec.

ReadoutElec.

EventBuilderEvent

Builder

StorageStorage

Detector

Role• Signal digitization• Event buffering• Data suppression• Data transfer to EB

Role• Signal digitization• Event buffering• Data suppression• Data transfer to EB

DAQ

Keywords in the System Design

• Five keywords, again…

pipelinepipeline

online data reductiononline data reduction

flexibilityflexibility

compactnesscompactness

commodity usecommodity use

Add-on-structure module

FIFO chip

Bridge chip

PMC size CPU

PMC size network interface

All Components on a 9U-Size Board

ADC/TDC Module

BridgeBridge

CPUCPU

NetworkNetwork

compactnesscompactness

Level 1 Trigger

Trigger Module

Trigger Module

ADC/TDC Module

trigger

busy

FINESSE TTRX

TRIGGER from the trigger module.

BUSY response by ADC/TDC module.

Data hold.

Data-transfer to FIFO.

ADC/TDC Module Readout FIFO

• Data transfer to readout FIFO

– 32-bit FIFO per ADC/TDC module.– Data size of each event is counted by FPGA

and is stored to FIFO-word-counter FIFO.

ADC/TDC Module

Readout FIFO(1 MB)

32-bit data

FIFO WENA

FIFO WCLK

spied

FIFO-word-counter

FIFO-word-counter

FIFO-word-counter FIFO

To CPU

FPGA

Readout FIFO CPU: DMA

Local-PCIBridge

Local-PCIBridge

CPUCPU

PCI bus

RadiSys EPC-6315PLX-9054

Local bus

Readout FIFO

FIFO-word-counter FIFO

Readout FIFO filled

PCI interrupt

Check event size

Initiate DMA

DMA data transferDMA over

PCI interrupt

COPPER Device Driver

interrupt_handler()interrupt_handler()

do_DMA()do_DMA()

while( data_in_FIFO ){ event_size = read_event_length_FIFO(); DMA_size += event_size;};

start_DMA( DMA_src_addr, DMA_dst_addr, DMA_size );

while( data_in_FIFO ){ event_size = read_event_length_FIFO(); DMA_size += event_size;};

start_DMA( DMA_src_addr, DMA_dst_addr, DMA_size );

interrupt_handler()interrupt_handler()

PCI interruptReadout FIFO ready (every event)

PCI interruptDMA over

DMAbufferDMAbuffer

user_read()user_read()do_DMA()do_DMA() Read new data (if exist)after last do_DMA().

Return to user code.

Return to user code.

Read all available data in FIFO (mostly one event)

CPU Event Builder

EBEthernet

CPUCPU

NetworkNetwork

PCI bus

serial linefor debug

– Are event-header/event-footer correct?

– Are data contents consistent with pre-defined ones?

– Generator module’s event counters = event tag from trigger module?

– Are all event counters from 4 generator modules consistent?

– Do event counters increase by 1 correctly?

– Are event-header/event-footer correct?

– Are data contents consistent with pre-defined ones?

– Generator module’s event counters = event tag from trigger module?

– Are all event counters from 4 generator modules consistent?

– Do event counters increase by 1 correctly?

Maximum Trigger Rate vs. Data Size

Typical data size

Data Transfer Speed on PCI vs. Data Size

Typical data size