next generation soc-based amc developments at desy

Next Generation SoC-based

AMC Developments at DESY

Michael Fenner, Simone Farina et al.

MTCA Workshop, DESY Hamburg, 02.12.2020

S. 2

Next Generation SoC-based AMC Developments at DESY

M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg Outline

• DAMC-FMC2ZUP : ZYNQ Ultrascale+ MPSoC FMC+ carrier

• DAMC-DS5G14ZUP : ZYNQ Ultrascale+ RFSoC Digitizer

• DAMC-DS812ZUP : 8 channel 800MSPS 12-bit Digitizer ZYNQ Ultrascale+ MPSoC

• DMMC-Stamp : Turnkey SoM Module Management Controller

• DAMC-FMC1Z7IO : ZYNQ-7000 GPIO extension board and FMC carrier

• DRTM-MXC : Mobile eXpress Module Carrier - Graphics carrier RTM module

S. 3


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-FMC2ZUP Overview

Hamburg

• Double-width mid-size AMC board

• 1 FMC+ slot (VITA 57.4)

• 1 FMC HPC slot (VITA 57.1)

• ZU11EG Zynq Ultrascale+ MPSoC

(653k logic cells, 2928 DSP) or

ZU19EG (1143k logic cells, 1968 DSP)

available on request

• Works as “CPU module”:

front panel USB type-C provides DisplayPort

and USB 3 & 2 host capabilities.

• Front-panel clock input

• Zone 3 connector compliant to Class D1.1

• White Rabbit endpoint capability

S. 4


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-FMC2ZUP Block Diagram

Hamburg

• Both FMC connectors are fully populated

up to the HA section

• Transceiver set is fully populated on the

FMC+ module

• 8 XCVR connected on the FMC HPC from

the PL

• 4GB DDR4 connected to PS and

1GB DDR4 16bit bus to PL

• Embedded 8GB eMMC memory and front

panel accessible SD card for Linux OS

rootfs and user data

• Voltage level translation to support legacy

RTM with Zone3 @ 2.5V

S. 5


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-FMC2ZUP Clock Tree Diagram

Hamburg

Clock Tree:

Key features:

• A cross point switch

collects/distributes clocks from

backplane, mezzanine

connectors and PLLs.

• Dedicated clock ICs for

Ethernet and PL reference

• VCXO and VCTCXO for White

Rabbit endpoint support

Clock tree chain configured by

MMC Stamp before PS boot

sequence

S. 6


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-FMC2ZUP

Hamburg

• DAMC-FMC2ZUP operating on bench with

graphical Linux OS and display output

• DAMC-FMC2ZUP collecting data from

FMC-DS500 and plotting on Display using

Python Matplotlib

• Typical use case

with fast digitizer FMCs

Courtesy of J. Marjanovic

S. 7


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-DS5G14ZUP Overview

Hamburg

• Double-width mid-size AMC board

• ZU47DR Zynq Ultrascale+ RFSoC 3rd generation

(930k logic cells, 4272 DSP),

ADC 8ch 14-bit 5GSPS,

DAC 8ch 14-bit 10GSPS

• Zone 3 connector compliant to new Class RF1.0

• Works as “CPU module”:

front panel USB type-C provides DisplayPort and USB 3

& USB 2 host capabilities.

• Configurable Front-Panel and/or RTM ADC Inputs

• DAC Outputs : 4 to Front-Panel & 4 to RTM

• RTM, Front Panel and Backplane Clock Inputs

• White Rabbit endpoint capability

S. 8


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-DS5G14ZUP Block Diagram

Hamburg

Processing System Features:

• Quad-core ARM A53 + Dual-core R5

• 4GB 64-bit wide dedicated DDR4

• Booting and storage devices:

1. Dual redundant QSPI flash memory

2. 8GB eMMC

3. Front panel uSD card

• 1000base-X ETH PHY to AMC port 0

• USB type-C with DisplayPort, USB3 & USB2

Programmable Logic connections:

• DDR4 memories :

3GB 48-bit wide & 2x 2GB 32-bit wide

• AMC ports 4-11 (redundant or single PCIe/GbE)

• Backplane MLVDS

• 100GbE zQSFP+ on front panel

• Zone3 RF1.0 digital interfaces

S. 9


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-DS5G14ZUP Clock Tree Diagram

Hamburg

Clock Tree:

Key features:

• Analog Clock section with external

dual VCO and filters. Configurable for

Deterministic Latency ADC/DAC data.

Inputs from Front Panel, RTM and

TCLKA/C. Clock Mezzanine Module

can bypass the Analog Clock Section

for fully flexible ADC/DAC clocks

• Digital Clocks can be derived from

either TCLKA/C, Analog Clock and

WR with additional option for FCLKA

• VCXO and VCTCXO for White Rabbit

endpoint support

Clock tree chain configurable by

MMC Stamp and Processing System

S. 10


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-DS5G14ZUP benefits

Hamburg

Benefits:

• Highly integrated AD and DA sections together with

FPGA logic reduce interconnection / MGT pin count

• No MGT related latency

GTYE4 MGT latency (@6.25Gbps)

83ns (28ns TX, 55ns RX) →

RFSoC ADC + DAC loopback

128ns ↓

ADC Boards Interface Resolution

[bits]

Sampling rate

[MSPS]

AD9268 SIS8300-KU LVDS 16 125

ADC12D800RF DAMC-FMC2ZUP +

DFMC-DS800 LVDS 12 800

ADS54J60 DAMC-FMC2ZUP +

ADS54J60EVM JESD204B 16 1000

XCZU28DR ZCU111 RFSoC int. 12 4096

Courtesy of J. Marjanovic, U. Mavric

S. 11


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-DS812: Low-Latency RF Digitizer

• Low-latency 8-channel 12 bit ADC board

• Xilinx Ultrascale Plus SoC: Stand-alone mode (“Scope“)

• Based on new coaxial analog Zone 3 Class RF1

• AC and DC inputs: Front and RTM 3 RF inputs

• 800 MSPS / 1600 MSPS, 4GBytes data memory

• 2.7 GHz input bandwidth (amplifiers: 4.8 GHz)

• Synchronous operation: exactly length-matched paths

• Excellent performance expected (measured on FMC)

• Dedicated per-chip clock inputs

• Low-noise PLL with on-board VCSO and TXCO

• Trigger inputs and White Rabbit support

• PCIe Gen.3 x 8 and zQSFP+ support

• Support for JESD DACs via Zone 3

(1) courtesy of J.Marjanovic

(2) courtesy of J.Zink

(3) courtesy of S.Jablonski

(3)

(2) (1)

(2)

(2)

(2)

Right-angle connector

S. 12


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-DS812: Low-Latency RF Digitizer

courtesy of R. Wedel

S. 13


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg MMC Stamp: Complete drop-in solution

• MMC is integrated: System-on-Module

• Highest integration: Power, USB, MCU, RTM, Translators

• Shipped as a pre-programmed LGA module

• Handles MCH Communication, RTM, FPGAs, JTAG, FMCs

• Simplest drop-in solution: no code, components, schematics

• High-density PCB with chip-scale components

• Code is maintained by DESY, SDK will be available

• In-system update via HPM (also for FPGA memories)

• Effort was put into code maintenance and compatibility

S. 14


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DAMC-FMC1Z7IO: Low-Cost SoC

Hamburg

Xilinx Zynq-7000

Default FPGAs:

XC7Z030-1FBG676C (4 MGTs)

XC7Z030-1FFG676C (4 MGTs)

Extended FPGAs

XC7Z030 (4 MGTs)

XC7Z035 (8 MGTs)

XC7Z045 (8 MGTs)

DESY

MMC Stamp

AMC Connector

84x LVCMOS25 / 42x LVDS

1 Gigabyte DDR3

32bit

1GBIT SPI FLASHRedundant 256 Megabit

SPI X4 FLASH

PS

DD

R

PS

MIO HR

20x LVCMOS18

BOTTOM I/O

320Pin

HPC connector

48x50pin

AMP

Connector

4x LVCMOS18

SPI CTL

JT

AG

4x G

PIO

JT

AG

4x G

PIO

16-ch PMBUS

co

ntr

ol

16x M-LVDS17-20

inp

ut

5x L

VC

MO

S3

3

ou

tpu

t

8x L

VC

MO

S3

3

34x LVDS or 34x LVCMOS (1.2-1.8)V

LA0-33

DATA

CTRL CPLD

I/O

Buffer

(6x)

+

Level Shift

(6x)

PCIe

1GbE

M-LVDS

4 links

Zone 3

1

GTXE2-X0Y0

GTXE2-X0Y1

LLL/PCIe P2P links

LVCMOS / LVDS

0

PS

MIO

RG

MII

1000Base-X

PHY

GTP1

GTP0

4

5

1GbE

0

HP

HP

MLVDS

Transceivers

InterlockOUT0-2

inp

ut

3x L

VC

MO

S3

3

SD CARD

PS MIO

GTXE2-X0Y2

2 SATA

DP0

DP1

PCIe

PCIe

DP2

DP3

SATA

GbE

PMBUS controller

LPC

PCIe

HPC

Lane x1

Lane x1

GTXE2-X0Y7

GTXE2-X0Y6

USB2.0 PHY

Micro-USB-B

5V/3.3V I/O

FMC

Micro-USB-B

Micro-HDMI HDMI TX

16 Gigabyte

eMMC

6

7

1

GTXE2-X0Y5

GTXE2-X0Y4

PCIe

PCIe

16-bit Interface

H/VSync+I2C

PMBUS

12-15

12x12

Cross-

Point

Switch

(CPS)

LLL/PCIe

GTXE2-X0Y3Dashed MGTs are only available

on XC7Z35 and XC7Z35

Interlock

• Double-width mid-size AMC board, based on Zynq-7000 SoC

• Designed as a lowest-cost solution

• FPGA options: XC7Z030-1FBG676C (low cost version)

XC7Z035-2FFG676E (standard version)

XC7Z045-2FFG676E (largest version)

• 48 I/Os, programmable direction & voltage level (5.0 V/ 3.3 V)

• FMC LPC with HPC transceiver pins connected four fast transceivers (up to 12.5 Gbps)

• Full RTM and AMC backplane connectivity (like „big“ boards)

• Front panel clock input

• Zone 3 Class D1.1 compliance

• Compatible to all our existing FMCs and RTMs - Motor drivers, piezo drivers, AD/DA cards etc.

• System-on-Chip (SoC) - Dual-Core ARM Cortex-A9 processor

- Can run without external CPU Module (int. PCIe root complex)

- Front panel micro USB (host) and micro HDMI output (monitor)

S. 15


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg DRTM-MXC: GPU Carrier in MicroTCA.4

• Idea: contribute to MicroTCA ecosystem

• Solution for GPU processing in MTCA

• Machine learning (CUDA, TensorFlow)

• Designed for concurrent AM900 CPUs

• Space available in typical installations

• Host standard MXM graphics cards

• Internal or external power input (12V-19V)

• 2x Display Port, 2x USB on rear panel

• MXM module can be installed by user

• Cooler: copper core, backplate + bushing NVIDIA Quadro RTX 3000

S. 16


M. Fenner, S. Farina et al. 02.12.2020, 9th MTCA Workshop, DESY Hamburg

Vielen Dank für Ihre Aufmerksamkeit

Thank you for your attention

next generation soc-based amc developments at desy

Documents