Data Acquisition Systems

Ryan RiveraFermilab Detector R&D Retreat

May 5, 2011

Wikipedia

• Data acquisition– is the process of sampling signals that measure

real world physical conditions and converting the resulting samples into digital numeric values that can be manipulated by a computer.

Data Acquisition Systems - Ryan Rivera 2Detector R&D Retreat

This Talk

1. Current DAQ program2. Future projections– Direction of Current Program– Direction of New R&D

Data Acquisition Systems - Ryan Rivera 3Detector R&D Retreat

Current Program

• Optical Links • CAPTAN• xTCA• Mu2e

Data Acquisition Systems - Ryan Rivera 4Detector R&D Retreat

Current Program

• Optical Links • CAPTAN• xTCA• Mu2e

Data Acquisition Systems - Ryan Rivera 5Detector R&D Retreat

Optical Links

• Versatile Link Common Project• Free Space Optical Transmission

Data Acquisition Systems - Ryan Rivera 6Detector R&D Retreat

Optical Links

• Versatile Link Common Project• Free Space Optical Transmission

Data Acquisition Systems - Ryan Rivera 7Detector R&D Retreat

Versatile Link Common Project

Data Acquisition Systems - Ryan Rivera 8

• A CERN-organized common project for ATLAS and CMS.• Charge: Develop general purpose optical link @ 5 Gbps data rates.• Fermilab has back-end component responsibilities.• Leading parallel optics investigations.

Detector R&D Retreat

Optical Transceiver Test Measurements

Data Acquisition Systems - Ryan Rivera 9

Eye Diagram Measurements:• Optical Modulation Amplitude• Extinction Ratio• Rise/Fall Times

Jitter Analysis:• Deterministic Jitter Decomposition• Random Jitter (Gaussian, Unbounded)• Eye Opening @ 10-12 BER

Industry Standard Measurements and Apparatus

Detector R&D Retreat

CMS Pixel Optohybrid (POH)

• Status: Current laser no longer available– Upgrades will require new devices

• Requirements:– Rad hard– 640 Mbps

• Next: FNAL responsible for testing and characterizing devices

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 10

Parallel Optics – Technology Evolution

Data Acquisition Systems - Ryan Rivera 11Detector R&D Retreat

Parallel Optics – Device Evaluation

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 12

Parallel Optical Engine Transceiver (4 channels, 6.25 Gbps/channel)

SNAP12 Transmitter (12 channels,

2.7 Gbps/channel)

SFP+ Single Channel Transceiver (10 Gbps)

Optical Links

• Versatile Link Common Project

• Free Space Optical Transmission

Data Acquisition Systems - Ryan Rivera 13Detector R&D Retreat

10 Gbps Optical Transmitters at

different wavelengths

Silicon Detectors

10Gb/s Optical Receivers

~10- 50cm

~50-100 cm

~100-150 cm

Beam Line Center

Free Space Optical Transmission• Motivation

– Reduce material budget– Work within rigid space constraints

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 14

1440 1460 1480 1500 1520 1540 1560 1580 16000.000

0.100

0.200

0.300

0.400

0.500

0.600

Measured Transmission Spectrum of Silicon(IC grade doped)

Wavelength (nm)

Tran

smis

sion

Lens Lens

CWDM Mux

Tx LVDS Electrical Bit Streams

~2 mm Si ~2 mm Si

FPGA-Based BERT

8cm21 cm

l1

l2

l3

l4

CWDM DeMux

l4

l3

Tx

Tx

Tx

Tx

Rx

Rx

Rx

Rx

l2

l1

Free SpaceProof of Concept l1 = 1470 nm

l2 = 1490 nml3 = 1510 nml4 = 1530 nm

15

Rx LVDS Electrical Bit Streams

CWDM: Coarse Wavelength Division Multiplexing

Next Step: Evolve the Optical Design for Detector ApplicationsDetector R&D Retreat Data Acquisition Systems - Ryan Rivera

16

Operated error free for over 48 hours at 1 Gbps on all 4 four channels

16Detector R&D Retreat Data Acquisition Systems - Ryan Rivera

Free Space Proof of ConceptLab Test

Optical Links Future

• Parallel optics– Initiated US R&D request to recent DOE detector

solicitation. Need KA15 money until DOE approval.– Working with vendors on emerging devices, and

SMU/OSU on custom devices.• Versatile Link and CMS Phase 1 Optohybrid– Activities not funded by KA15, funded through CMS.

• Free space optics– Exploring low mass/power options (modulators).

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 17

Current Program

• Optical Links

• CAPTAN• xTCA• Mu2e

Data Acquisition Systems - Ryan Rivera 18Detector R&D Retreat

CAPTAN

Data Acquisition Systems - Ryan Rivera 19

What is the CAPTAN system?

Detector R&D Retreat

Compact And Programmable daTa Acquisition Node

Data Acquisition Systems - Ryan Rivera 20Detector R&D Retreat

Attributes

Data Acquisition Systems - Ryan Rivera 21

• It is simple.– It is 6” x 6” and for many systems, the only

external connections are a 3.3V power supply and a standard Ethernet cable.

Detector R&D Retreat

Attributes

Data Acquisition Systems - Ryan Rivera 22

• It is flexible.– The user can stack the foundation boards in

different combinations to give unique functionality.

Detector R&D Retreat

Attributes

Data Acquisition Systems - Ryan Rivera 23

• It is scalable.– In addition to the vertical stacking, the stacks can

be repeated arbitrarily and connected with one or many PCs in an Ethernet network.

Detector R&D Retreat

OPTICAL BUS

COOLING CHANNELVERTICAL BUS

LATERAL BUS MOUNTING HOLE

ELECTRONICS

User Template

Data Acquisition Systems - Ryan Rivera 24

• The CAPTAN architecture consists of a few core boards but is intended to be augmented by custom boards designed and built by users.

Detector R&D Retreat

Core Boards

Data Acquisition Systems - Ryan Rivera 25

“Green Board”NPCB – Node Processing and Control Board

“Blue Board”DCB – Data Conversion Board

“Red Board”PDB – Power Distribution Board.

Detector R&D Retreat

CAPTAN User Community• Fermilab• Brown• Purdue• Colorado• Milano• Lecce

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 26

Applications

• The CAPTAN system was designed to handle common data acquisition, control, and processing challenges within high energy physics.

• Examples of such applications are tracker readout systems, R&D test stands, and parallel data processing.

• As the CAPTAN system is a modular system it can be used for a wide range of applications, from very small to very large.

Data Acquisition Systems - Ryan Rivera 27Detector R&D Retreat

FTBF Pixel Telescope

Data Acquisition Systems - Ryan Rivera 28Detector R&D Retreat

PPD VIP Test Stand

Data Acquisition Systems - Ryan Rivera 29Detector R&D Retreat

QIE Irradiation BER Testing

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 30

IHEP Telescope - Beijing

Data Acquisition Systems - Ryan Rivera 31Detector R&D Retreat

T980 Crystal Collimation Telescope

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 32

CAPTAN Future

• Port CAPTAN Software to Linux

• Explore parallel processing power

• Integrate with xTCA

Data Acquisition Systems - Ryan Rivera 33Detector R&D Retreat

Current Program

• Optical Links • CAPTAN

• xTCA• Mu2e

Data Acquisition Systems - Ryan Rivera 34Detector R&D Retreat

xTCA

• ATCA (Advanced Telecommunications Computing Architecture)– Spec put forth by PICMG (PCI Industrial Computer

Manufacturers Group: a consortium of over 250 companies)

• AMC plugs into ATCA card– Or MicroTCA crate

• ATCA + MicroTCA = xTCA• Successor to VME?

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 35

12U 14-slot ATCA shelf

Advanced Mezzanine Card (AMC)

Data Acquisition Systems - Ryan Rivera 36Detector R&D Retreat

xTCA and CAPTANIntegration

Data Acquisition Systems - Ryan Rivera 37Detector R&D Retreat

VME vs. xTCA

+ Number of new developments decreasing, sales still constant

+ Many I/O modules available− Developed in 1981− Bus technology has speed limitations− Wide busses create a lot of noise in analog channels− No standard management at crate level− No standard management at module level− One damaged bus line stops entire crate

Data Acquisition Systems - Ryan Rivera 38Detector R&D Retreat

VME vs. xTCA

+ Scalable modern architecture From 1 slot MicroTCA to full mesh ATCA

+ 1 Gbps serial communication links+ Standard PCIe and Ethernet communication+

Well defined management+ Hot-swappable+ Safe against hardware/software failures+ 99.999% availability is possible

Data Acquisition Systems - Ryan Rivera 39Detector R&D Retreat

VME vs. xTCA

− Many crates and modules from different manufacturers Is there a 'standard' ?

− Learning curve to software/hardware specifications

− Need functional MMC/IPMI code for power delivery

− Imposed system management

Data Acquisition Systems - Ryan Rivera 40Detector R&D Retreat

xTCA Future

• Large experiments are considering xTCA over VME.– CMS– ATLAS– LHCb– Experiments at DESY

• Dream is to develop a readout system for strips/pixels– Evaluate for future FNAL Experiments (CMS upgrade, Mu2e)

• SBIR submitted with industry partners.• Vince Pavlicek is on committee to develop the

PhysMTCA or MTCA.4 (MicroTCA for Physics) spec.

Data Acquisition Systems - Ryan Rivera 41Detector R&D Retreat

Current Program

• Optical Links • CAPTAN• xTCA

• Mu2e

Data Acquisition Systems - Ryan Rivera 42Detector R&D Retreat

Mu2eTracker

21,600 Straws216 Readout Controllers~65 GBps*

Calorimeter2112 Crystals 44 Readout Controllers~15 GBps*

* zero-suppressed data

- Additional data (< 1 GBps) from Cosmic Ray Veto system and extinction/target monitors

Data Acquisition Systems - Ryan Rivera 43Detector R&D Retreat

Architecture Alternatives

• Streaming (this is the selected architecture)+ less front-end hardware (no concentrators or L1 Trigger)+ simpler design+ flexible trigger− higher cost

• Triggered+ less back-end hardware (networking and processing)+ lower cost+ allows higher digitization rate (e.g., waveform digitizers)− fixed L1 trigger

Data Acquisition Systems - Ryan Rivera 44Detector R&D Retreat

Implementation Alternatives

Data Acquisition Systems - Ryan Rivera 45Detector R&D Retreat

Standalone(rack-mount)

DTC

Servers

EVB Switch

EVB Switch

DTC

PCIe DTCDTC

EVB SwitchDTC NIC PCIe DTC

(event building in server)

Servers

Servers

NIC

Comparing Technology

• To meet the demands of a 100 GBps streaming DAQ, Mu2e will leverage…– 10 Gbps Networking– High Density FPGAs– High Performance Processors– Fiber-optic Communications (synergy with Optical Links)

CDF DAQ (1987) Mu2e DAQ

Data Rate 20 100,000 MBps

Buffering 400 2,000,000 MB

Processing (FP) 10 30,000,000 MFLOPS

Processing (INT) 200 60,000,000 MIPS

Historical Perspective

Data Acquisition Systems - Ryan Rivera 46Detector R&D Retreat

Design Philosophy• Commercial Slow Control hardware and software is used for

development– needed early, can’t wait for custom HW/SW– hopefully, utility extends into production and running– LabVIEW or EPICS

• Embedded Processors– use commercial modules if possible to allow early software development– embedded Ethernet and HTTP with eye towards debugging ease

Data Acquisition Systems - Ryan Rivera 47Detector R&D Retreat

Mu2e Future

• One year from starting prototype phase.• ~6 years to installation complete.• Mu2e funded by Mu2e, but components of

DAQ architecture may apply to future experiments (Project X and others).

Data Acquisition Systems - Ryan Rivera 48Detector R&D Retreat

DAQ Discussion Points

• Program Resources• National Picture Comparison• Best Future R&D Avenues

Detector R&D Retreat Data Acquisition Systems - Ryan Rivera 49

DAQ Program Resource Issues

• Limited manpower– Has hindered xTCA effort

• xTCA development accessibility– Community MMC design effort?

• FPGA power/expense/expertise

Data Acquisition Systems - Ryan Rivera 50Detector R&D Retreat

Summary of National Picture vs. Current Program

• Breaking new ground with optics, free space– Submitted SBIR and DOE R&D proposal

• Optics: NTT (Nippon Telegraph and Telephone) achieved 69.1 Tbps transmission using WDM of 432 wavelengths with a capacity of 171 Gbps over a single 240 km optical fiber on March 25, 2010. Highest optical transmission speed ever recorded at time.

• xTCA falling behind?• GPUs falling behind?• DAQ Software falling behind?

– COMEDI, EPICS, LabVIEW, MATLAB, Visual C++, ladder logic?

Data Acquisition Systems - Ryan Rivera 51Detector R&D Retreat

Best Future DAQ R&D Avenues

• Ethernet/InfiniBand: InfiniBand prevalent as the interconnect between nodes of the highest performing super computers

• Off-the-shelf components• xTCA• GPUs: Fastest computer in the world, Tianhe-1A in China uses 7,168

NVidia Fermi GPUs and 14,336 Intel Xeon CPUs. Would require 50,000 CPUs for same performance with CPUs alone.

• Optics: Commonly 100 Gbps with distances over 100 m. Becoming interconnect of choice in high speed data centers and HPC clusters. There are 40 Gbps and 100 Gbps optical Ethernet standards.

• FPGAs• EPICS: The tool is designed to help develop systems which often feature

large numbers of networked computers providing control and feedback.

Data Acquisition Systems - Ryan Rivera 52Detector R&D Retreat

Thanks!

Data Acquisition Systems - Ryan Rivera 53Detector R&D Retreat