Thoughts on the RCM and its interfaces…

Michael Huffer, SLACmehsys@slac.stanford.edu

BNL Workshop on Raft Electronics Architecture January 25 - 27, 2012

V1

Raft Interfaces & their global constraints

• The raft requires the following abstract external interfaces:– Power– Clock– Command (Resets, Start imaging etc…)– Configuration of its reprogrammable elements (FPGAs)– Monitoring & Control of its internal electronics– Video Data

• Operating in concert, within the camera these interfaces are subject to additional constraints due to the raft’s inaccessibility & its presence within an environment which is electronics “unfriendly”. These constraints:– Force one to:

• Minimize use of “active” parts• Minimize # of connections/raft• Minimize # of cryostat penetrations• Minimize thermal load and footprint

– Limit:• Choices for parts, connectors, cables & feed-throughs

– Must take into account:• Development and testability • Ease of assembly & repair

Global Timing & Configuration Requirements

• Configuration implies the mechanism for the reprogramming of any of a raft’s configurable logic. That mechanism should:– Depend only on the presence of power – Be independent of clock & command delivery– Provide consistent usage pattern whether in-situ

(the cryostat) or in isolation (the bench-top)– Provide highly reliable and prompt service– Be compatible with manufacturer standards

• The camera operates synchronously • Each raft has both external Clock and Command signals. All raft:

– Reference clocks are driven from a common source (the System Clock, @ notational 50 MHZ)– Commands are driven from a common source (the CCS)

• System clock and command are synchronized to the observatory– Allows for time correlations across entire observatory

• Synchronous operation implies:– All raft reference clocks and commands are phase aligned with respect to that system clock

• Delivery of both clock and command to raft must be highly reliable and glitch-free

“Let a thousand flowers bloom…”

• Disclaimer: What follows is not the currently accepted base-line...– Some, but not all has been vetted by some, but not all of the interested parties– It bends somewhat considerably the current architecture, however…

• Changes very little of which has been built (does not add or subtract the # of interfaces)• Changes little if any in the way of schedule & cost (except in a positive direction)• Does not in any way impact the schedule for the vertical slide test

• In a nutshell, its “Features” (in no particular order):– Changes the mapping of logical to physical interfaces for the RCM– Reduces the number of necessary external signals for the RCM– Eliminates the need for the construction of a TCM

• Incorporates its functionality into pre-existing components of the DAQ– Configuration mechanism is redefined:

• Industry standard JTAG– Changes the model for (raft dependent) cabling within the cryostat – Reduces considerably the number of cryostat penetrations (and attendant flanges)– Moves to industry standard feed-throughs– Changes signal cables within cryostat

• Round coax to flat, bendable, ribbon, Twin-Ax– Moves conversion of copper/light closer to cryostat– Adds one new (modest) board into cryostat

RCM Physical Interfaces (the Transition board)

• RCM has 4 physical interfaces on one vertical connector– POWER (as before)– DATA (as before)– TIMING (integrates CLOCK & COMMAND)

• 1 differential pair (CML)– CONFIGURATION (employs JTAG)

• 5 single-ended• Vertical connector incorporates all four interfaces• Two different Transition-Boards adapt RCM to usage:

– Inside cryostat• Allows right-angle turn for flat cabling• Merges TIMING & JTAG (4 CML pairs)• Uses 3M Twin-Axial cable

– Outside or bench-top• Allows direct connection to DAQ system• Converts to light TIMING & DATA interfaces• Allows direct connection to Xilinx JTAG dongle

Clock Recovery

• Encodes both command and clock within one signal. Its advantages:– Halves the number of signals to support both functions (nice)– Automatically addresses phase alignment between command & clock (very nice)

• Encoder must ensure good DC balance (lots of transitions between 0 & 1) – Manchester encoding (most older DVD players)– Bi-Phase-Mark (BPM) encoding (used by LHC in its TTC system)– 8B/10B encoding (1-10G Ethernet, PCIe, InfiniBand, etc…)

• Decoder recovers clock from signal & phase-lock-loops with local reference

• In all modern FPGAs this technique comes predefined as hardened silicon– For Xilinx its included within each of its Multi-Gigabit Transceivers (MGTs)

• 8B/10B encoder/decoder included along with clock recovery circuit – Good for a clock with jitter on the order of 50-100 PS (in my experience)

• Doesn’t help (or hinder) with relative phase alignment between rafts– Careful design practice in fan-out– Adjustable delay at end-points (also included in Xilinx silicon)

• Corner raft retains outside FPGA architecture…– Suitably repackaged mechanically…

• Inside FPGA architecture…– Retains RTI– Replicates SCI and controller for WF & two (2) Guider sensors

RCM Block Diagram

• RCM is one FPGA with two connectors:– One for raft’s BEE & one for external interface

• Outside FPGA… – Standard Xilinx flash to store configuration

• Loaded through FPGA’s JTAG interface– Two reference oscillators for MGTs…

• One for DATA & one for TIMING• Inside FPGA…

– Two MGTs & clock buffers ( Xilinx)• One for DATA & one for TIMING

– Two communication blocks (us)• SCI for Data, Control & Monitoring (unchanged)• RTI for Command, Reset & Clock (new)

– One Controller connected to SCI & RTI (us) • Interacts with SCI & RTI as well as BEE connector• Drives board ID to RTI

The RCE mezzanine board as a model for the RCM

Fan-Out Board

• Has two functions:– Fans-out TIMING to 25 rafts (both science and corner)– Multiplexes JTAG to 25 rafts (both science & corner)

• Contains a modest number of CML clock drivers & JTAG muxs (2-3 each)– Dead simple, no programmable or configurable logic

• For external connections has:– 25 pigtails (4-pair flat Twin-Ax) to rafts– 2 (4-pair flat Twin-Ax) connector to feed-through– For repair designed to be removable

• Located under annular feed-through ring, but…. – Physically attached to cryostat– Accessible when rafts are exposed for access

TIMING Distribution

• Fan-out (Copper)– 1 CML pair In (from penetration)– 25 CML pairs Out (to rafts)– Inside cryostat on fan-out board– “Dumb” (CML clock drivers)

• Shares with JTAG signals…– Cable to each RCM – Sub-D connector feed-through

• Fiber on flip side of feed-through…• Fiber connects to RCE in CDS crate

– No need to implement TCM• Control (software) interface trivial

– User simply specifies command:• 40-bit enable list of destinations • 8-bit command

• Command set is extensible:– Reset– Enable/Disable wall-clock– Start/Stop sequencer

JTAG Distribution

• JTAG signals are multiplexed– 5 JTAG signals In (from feed-through)– 5 JTAG signals Out (to selected raft)– Inside cryostat on fan-out board– Commercial JTAG mux (e., g., )– 5 bit selection from feed-through

• Shares with TIMING signals…– Cable to each RCM – Sub-D connector feed-through

• Flip side of feed-through contains…– 9-pin JTAG connector

• Connects to Xilinx JTAG dongle– Select pins which go to…

• On board DIP Switch• Connector to HPU

• HPU runs standard Xilinx toolkit• HPU connects dongle and select pins…

– USB– DIO

Feed-Throughs & cable routing

• Segregate power from signal– Route power & signal to different sides

• Collapse raft modularity at cold plate – Aggregate types of interfaces

• Three cables per raft– POWER (standard vacuum cable)– DATA (2 pairs, 3M Twin-Ax)– TIMING/JTAG (4 pairs, 3M Twin-Ax)

• Standard vacuum Feed-Throughs and flanges– One (1) 1 x 25 pin Sub-D connector

• Feed through for Fan-Out Board• TIMING/JTAG of 25 rafts

– One (1) 1 x 35 pin Circular connector• Feed through for POWER for 25 rafts

– Four (4) 2 x 25 pin Sub-D connector• Feed through for DATA for up to 12 SCIs

scheme “pins”/raft penetrations flanges

baseline 24+1 1144 11

proposed 12+1 260 5-6

Flange usage notational

• Conversion from copper to light occurs immediately, on flip-side of flange– Each DATA flange services up to 12 rafts