XFEL Meeting, DESY XFEL Meeting, DESY

5 May 20115 May 2011

Erdem Motuk, Martin Postranecky, Matt Warren, Matthew WingErdem Motuk, Martin Postranecky, Matt Warren, Matthew Wing

XFEL 2D Pixel Clock and Control SystemXFEL 2D Pixel Clock and Control System

2

OUTLINEOUTLINE

• Test card A progressTest card A progress

• RTM card progressRTM card progress

• Firmware progress Firmware progress

• Veto logicVeto logic

• Future plansFuture plans

3

Test card A progressTest card A progress

• The manufactured boards (PC3461M) are receivedThe manufactured boards (PC3461M) are received– 8 boards in total8 boards in total

– Initial test (voltages, LEDS) carried out – all working orderInitial test (voltages, LEDS) carried out – all working order

• Each board provide 1 channel TX and 1 channel RXEach board provide 1 channel TX and 1 channel RX• 1 channel -> FAST clock, FAST data, Status, Veto -> 1 RJ45 conn.1 channel -> FAST clock, FAST data, Status, Veto -> 1 RJ45 conn.

• AC coupling circuitries on the RX channel (clock,data,veto)AC coupling circuitries on the RX channel (clock,data,veto)

• Further tests are carried out Further tests are carried out – A pair of boards used for telegram RX/TX and FAST RX/TX testsA pair of boards used for telegram RX/TX and FAST RX/TX tests

– 11stst board -> Telegram TX – FAST RX board -> Telegram TX – FAST RX

– 22ndnd board -> Telegram RX – FAST TX board -> Telegram RX – FAST TX

– Tests were running successfully – without errors Tests were running successfully – without errors

4

Test card A progressTest card A progress

• Stress testing the boards to verify the AC coupling circuitryStress testing the boards to verify the AC coupling circuitry– Random data pattern (LFSR generated) sent in between long Random data pattern (LFSR generated) sent in between long

durations of zeros or onesdurations of zeros or ones

– Clock (99 MHz) is sent along with the data on a separate channelClock (99 MHz) is sent along with the data on a separate channel

– Tests were run overnight using shielded cables (~5m)Tests were run overnight using shielded cables (~5m)

– No errors observed due to the AC coupling and the transmissionNo errors observed due to the AC coupling and the transmission• Results observed by ChipScopeResults observed by ChipScope

• A few errors observed due to the test firmwareA few errors observed due to the test firmware

• The choice of the sync word is the cause – one random sequence is lostThe choice of the sync word is the cause – one random sequence is lost

• Tests carried out on all 8 boardsTests carried out on all 8 boards

• Boards available to FEE developers if neededBoards available to FEE developers if needed

5

Test card A progressTest card A progress

6

UTCA crate progressUTCA crate progress

• The purchased uTCA crate (xTCA) is kitted with the TR The purchased uTCA crate (xTCA) is kitted with the TR and the DAMC2 boardsand the DAMC2 boards

• SUSE Linux is installed on the crateSUSE Linux is installed on the crate

• The boards were powered without any problems The boards were powered without any problems

• IPMI tools detected the boards presence and the board IPMI tools detected the boards presence and the board information can be readinformation can be read

• A simple test firmware to test the PCIe functionality of the A simple test firmware to test the PCIe functionality of the DAMC2 board generatedDAMC2 board generated• DAMC2 board showed up on the PCIe tree without any problemsDAMC2 board showed up on the PCIe tree without any problems

• A simple test driver is generated for the boardA simple test driver is generated for the board

• Was able to access memory mapped registers on the board Was able to access memory mapped registers on the board

7

RTM card progressRTM card progress

• Schematic design for the RTM prototype is finishedSchematic design for the RTM prototype is finished

• The board provides 16 channels The board provides 16 channels • 1 channel -> FAST clock, FAST data, Veto, Status Feedback1 channel -> FAST clock, FAST data, Veto, Status Feedback

• 1 spare channel (RJ45 connector) could also be fit1 spare channel (RJ45 connector) could also be fit• 2 pairs input (with the AC-coupling circuitry)2 pairs input (with the AC-coupling circuitry)

• 2 pairs output (with the LVDS buffer)2 pairs output (with the LVDS buffer)

• Designed to provide maximum flexibility for testingDesigned to provide maximum flexibility for testing• On-board (~9 MHz) oscillatorOn-board (~9 MHz) oscillator

• Clock multiplexers to select between on board clock and DAMC2 Clock multiplexers to select between on board clock and DAMC2 supplied clocksupplied clock

• User programmable LEDs (on the 2x8 RJ45 connector)User programmable LEDs (on the 2x8 RJ45 connector)

• Clock feedback to the DAMC2 boardClock feedback to the DAMC2 board

• Extra prototyping areaExtra prototyping area

8

RTM card progressRTM card progress

9

RTM card progressRTM card progress

• RTM card designed according to the uTCA specsRTM card designed according to the uTCA specs• Mechanical details to fit double width, full height faceplateMechanical details to fit double width, full height faceplate

• An I2C extender (PCF8574)An I2C extender (PCF8574)

• A temperature monitor ()A temperature monitor ()

• An EEPROM ()An EEPROM ()

• Currently in the layout stageCurrently in the layout stage• Liaising with the RAL DOLiaising with the RAL DO

10

• Joined the DESY-wide effort for standardising FPGA Joined the DESY-wide effort for standardising FPGA development for DAMC2 – FPGA Experts Groupdevelopment for DAMC2 – FPGA Experts Group

• A unified firmware/software framework to use the DAMC2A unified firmware/software framework to use the DAMC2

• A central project/core repositoryA central project/core repository

• Will use cores from the repository and contribute to it if Will use cores from the repository and contribute to it if neededneeded

• First aim is to provide a startup project for the DAMC2First aim is to provide a startup project for the DAMC2– A general UCF fileA general UCF file

– PCIe block that is common to all cardsPCIe block that is common to all cards

– II Interface – A versatile bus/register interface between different II Interface – A versatile bus/register interface between different cores on the FPGAcores on the FPGA

– Cores for using SFP links, DDR2 RAM on boardCores for using SFP links, DDR2 RAM on board

– Cores for using different peripherals – e.g. Clock MUXCores for using different peripherals – e.g. Clock MUX

Firmware progressFirmware progress

11

Firmware progressFirmware progress

• What this means for the CC cardWhat this means for the CC card• The standard PCIe core for the DAMC2 is usedThe standard PCIe core for the DAMC2 is used

• Based on Xilinx’s PCIe endpoint block + programmable Based on Xilinx’s PCIe endpoint block + programmable memory access corememory access core

• Standard Vendor ID + device ID for the DAMC2 cardStandard Vendor ID + device ID for the DAMC2 card

• Basic driver already been developedBasic driver already been developed

• A GUI to access individual registers (defined according to A GUI to access individual registers (defined according to application)application)

• II Bus/Interface will be used to integrate PCIe to CC related II Bus/Interface will be used to integrate PCIe to CC related modules in the systemmodules in the system

• Individual registers + memory area defined in a VHDL Individual registers + memory area defined in a VHDL package filepackage file

• Use of internal and external registers and methods to access Use of internal and external registers and methods to access themthem

12

Firmware progressFirmware progress

• Firmware block diagram with the II InterfaceFirmware block diagram with the II Interface

• Control and clocking part involves cores for the clock MUX, Control and clocking part involves cores for the clock MUX, synthesiser etc. on DAMC2synthesiser etc. on DAMC2

• The registers block on the old firmware diagram is replaced by IIThe registers block on the old firmware diagram is replaced by II

13

Firmware progressFirmware progress

• Telegram data transferTelegram data transfer– Being redeveloped according to the latest Being redeveloped according to the latest

– Basic serial transmit and receive module based on a UART like Basic serial transmit and receive module based on a UART like handshaking and synchronising is in the repository for the TR firmwarehandshaking and synchronising is in the repository for the TR firmware

– The telegram clock changed to 130 MHzThe telegram clock changed to 130 MHz

– A protocol on top of this physical layer has to be defined for getting CC A protocol on top of this physical layer has to be defined for getting CC related information from the TR boardrelated information from the TR board

– On-going work with P. Gessler and H. KayOn-going work with P. Gessler and H. Kay

• FAST data transmitFAST data transmit‒ The existing module will be modified to integrate to the II interfaceThe existing module will be modified to integrate to the II interface

14

Veto logicVeto logic

• Use of the TPS (Toroid Protection System) for a possible Use of the TPS (Toroid Protection System) for a possible VETO sourceVETO source‒ Developed by M. Werner and D. NoelleDeveloped by M. Werner and D. Noelle

‒ A distributed system consisting of coils detecting the charge A distributed system consisting of coils detecting the charge along the beam line plus their backend electronicsalong the beam line plus their backend electronics

‒ The analogue charge data is digitised and processed to provide The analogue charge data is digitised and processed to provide information for MPS and other systemsinformation for MPS and other systems

‒ A chain-like topology using digital fibre transmission lines to see A chain-like topology using digital fibre transmission lines to see how much charge has reached the next charge monitorshow much charge has reached the next charge monitors

Undulator

Undulator

Toroids

Guns

Linac Sections

15

Veto logicVeto logic

• Each toroid backend consists of a DAMC2 board and a custom RTMEach toroid backend consists of a DAMC2 board and a custom RTM• SFPs on the DAMC2 are used for fibre communication to other toroids and SFPs on the DAMC2 are used for fibre communication to other toroids and

other systemsother systems

• Single-mode fibre – exact bit rate / protocol to be defined Single-mode fibre – exact bit rate / protocol to be defined

• One TX channel is available for the VETO systemOne TX channel is available for the VETO system

• Data from minimum two toroids envisaged for the VETO systemData from minimum two toroids envisaged for the VETO system

• At the beginning and end of the undulatorsAt the beginning and end of the undulators

• Exact data to be determined (normalised charge data + train/bunch ID?)Exact data to be determined (normalised charge data + train/bunch ID?)

VETO FANOUT

(RTM)

DAMC2

SFPs (4 in total)

VETO LOGIC on

FPGA

TPS

TPS

TX only @ 1.083 Gb/s

(2.16 Gb/s ?)

Charge data from the TPS + Bunch ID ?

To FEEs

16

Future plansFuture plans

• RTM manufactureRTM manufacture

• Reception of the RTMs and initial testingReception of the RTMs and initial testing• Card power-up importantCard power-up important

• Firmware debug/developmentFirmware debug/development• II Bus integrationII Bus integration

• DriversDrivers

• Testing the RTM + DAMC2 in the cratesTesting the RTM + DAMC2 in the crates• Testing Standalone OperationTesting Standalone Operation

• Testing with the TR (telegram/trigger/…)Testing with the TR (telegram/trigger/…)

• Testing with the Test Card A (FAST data + clock)Testing with the Test Card A (FAST data + clock)

• VETO logic VETO logic • TPS feasibility documentTPS feasibility document

• Initial work on VETO logicInitial work on VETO logic

• Work on TPS data RX/TX ?Work on TPS data RX/TX ?