panstarrs gigapixel array controller system
DESCRIPTION
PanSTARRS Gigapixel Array Controller System. Overview Present Status CPU board FPGA board DAQ board Staffing Infrastructure Schedule. Outline. Illustration of Bar Mounting Scheme - 3. OTA flexcables attach to internal - PowerPoint PPT PresentationTRANSCRIPT
PanSTARRS Internal Spec Review 8/5/2003
PanSTARRS Gigapixel Array Controller System
• Overview • Present Status
– CPU board– FPGA board– DAQ board
• Staffing• Infrastructure• Schedule
Outline
PanSTARRS Internal Spec Review 8/5/2003
Illustration of Bar Mounting Scheme - 3
OTA flexcables attach to internalmotherboard with amplifiers. A single controller module is shown for reference (1 controller for 4 OTAs). Note: hermetic feedthrough deleted for clarity.
PanSTARRS Internal Spec Review 8/5/2003
Gigapixel Camera Controller Electronics
• Controllers are mounted behind cryostat.
• 16 controllers per Gigapixel camera.
PanSTARRS Internal Spec Review 8/5/2003
Baseline Controller Design
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA OTA OTA OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA OTA OTA OTA
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
IOTA
3U
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA OTA OTA OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA
OTA OTA OTA OTA
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
• IOTA3U controller operates 4 OTAs• For a 8 x 8 OTA focal plane, 2 rows of 1 x 8 OTA’s • Each row has 8 IOTA3U controllers.
1Gb ethernet
Power Supply
LAN Switch
Fiber isolated
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
Pixel Server
1Gb ethernet
2-row OTA Controller “Rack”
PanSTARRS Internal Spec Review 8/5/2003
Data Storage Advances
• Storage Area Networks and Network Attached Storage– Cost effective high performance– Connectivity to “second tier” processors if needed
• Pixel Servers also have built in RAIDs– Lowest cost but performance not benchmarked
Back-end LAN+SAN
SAN
SWITCH
Second Tier Processing
Fiber Broadcast
PanSTARRS Internal Spec Review 8/5/2003
1 x 4 IOTA3U Assembly
HDC
FPG
A
CO
TS C
PU
board
OTA
CPCI backplane
CO
TS 1G
ethernet
Flexprint PC
B
Cryostat wall
Signal chains and A
DC
s
Interface bd • Green areas are custom designed
• Blue areas are COTS• 3U hardware could include
connectors, front panels, guide rails, cooling components, chassis parts, etc.
Chassis
Front Panel
Cooling Cooling
Interface Board
Signal chains and A
DC
s
Signal chains and A
DC
s
FPG
A
CO
TS C
PU
Flexprint PC
B
OTA OTA OTA
CPCI backplane
OTA
Signal chains and A
DC
s
HDC
Module w
all
PanSTARRS Internal Spec Review 8/5/2003
Mechanical Check 1 x 4
Green crosshatch shows 1 x 4 OTA shadow4 sets of 2mm metric connectors shown
2 are CompactPCI bus standard2 for OTA signal processing
Standard 3U front panel shown in maroon.
PanSTARRS Internal Spec Review 8/5/2003
IOTA3U Boards
• COTS CPU board – Baseline LINUX based PowerPC– 1 or 2 channel 1G Ethernet fiber PMC
daughterboard– CPCI bus interface
• FPGA3U board– CPCI interface– FPGA +1Ge??– Buffer memory– Data interface to DAQ3U
• DAQ3U board– Data interface– FPGA– Buffer memory– Signal Chain and ADCs
• Backplanes– CPCI backplane– Interface backplane
Signal fanout Filtering Connectors
FPG
A3U
CO
TS C
PU
board
CPCI backplane
CO
TS 1G
ethernet
DA
Q3U
Signal chains and A
DC
s
Interface bd
DA
Q3U
Signal chains and A
DC
s
PanSTARRS Internal Spec Review 8/5/2003
COTS CPU – Status
• Colleagues at CFHT have evaluated an SBS RL4
– PowerPC 7400/750 CPU at 400 to 500MHz
– Fast 10/100Mbit Ethernet– PMC extension slot– Conduction cooled option
• LINUX support not good
• IfA has purchased and received a Menmicro F1N
– PowerPC MPC8245/300MHz– 1 Gbit Ethernet daughterboard– LINUX ElinOS support (RTAI also)
• STATUS – Under Test
PanSTARRS Internal Spec Review 8/5/2003
FPGA3U Board Design
Pow
er
sect
ion
To DAQ3U
128Mx32 SDRAM
1Gb/100BTPHY
Fiber coupled 100BT* /100BT /10 BT
32Kx32 Dual Port RAM Buffer
Compact PCI Bus
LDV
S
Loca
lX
tal
Optocoupled Trigger I/O
Trigger signals
Test
Con
n
Support CPLD
JTAG
FPG
A
PR
OM
Xilinx Virtex 2Pro
FPGA405 PPC
• Virtex2pro FPGA possible centroid/shift calculator
PanSTARRS Internal Spec Review 8/5/2003
Xilinx ML300 Development Board
Pow
er
sect
ion
To DAQ3U
128MB DDR SDRAM
1GPHY
Fiber coupled 1G ethernet
•PMC PCI Bus
LDV
S
Loca
lX
tal
System ACE
JTAG
FPG
A
PR
OM
Xilinx Virtex 2Pro
FPGA405 PPC
1394
PH
Y
PCMCIA RS
232
LCD
Tou
ch
scre
en
Dis
play
Par
alle
l po
rtK
eybo
ard
port
PanSTARRS Internal Spec Review 8/5/2003
DAQ3U Board Design
Analog Devices
5379 Analog switch
Analog switchAnalog
switchAnalog switchAnalog
switchesAnalog switchesAnalog
switchesAnalog switches
Voltage range buffers
TBD RAM
Analog Devices
98263 CCD outputs
Analog Devices
98263 CCD outputs
Analog Devices
98263 CCD outputs
LDV
S
Analog Devices
98263 CCD outputs
Analog Devices
98263 CCD outputs
Analog Devices
98263 CCD outputs
32 bit pixel read
bus
Optocoupled Trigger I/O
Trigger signals
Test
C
onn
Xilinx Virtex 2Pro
FPGA405 PPC
Pow
er
sect
ion
Support CPLD
JTAG
FPG
A
PR
OM
100BTPHY
Fiber coupled 100BT
• Virtex2pro FPGA clocking and pixel data aggregation (possible to incorporate into FPGA board Virtex2pro
PanSTARRS Internal Spec Review 8/5/2003
Mechanical Check DAQ3U
999999-14XYZ 04/22/23
ADC selection
Baseline AD9826 due to small size and speed but need to test for INL and ENOB (&/or oversample)
16BIT ADC Freq Ana Input Input R Tconvert S/N 100khz S/N 900khz THD 100Khz ref stab Integral nonlin Diff nonIin ENOB Size PowerAnalogic ADC4322 2Mhz +/-2.5/5/10 750/1.5k 300ns 86dBmin 75dBmin -86dBmax 15ppm/C +/- .003% +/- .75LSB 14.29s/n 2.4"x1.5" 2.3WDatel ADS-932MC 2Mhz +/-2.75 500 pipeline 81dBmin 80dBmin -88dBmax +/-30ppm/C +/- 1LSB +1LSB 13.45s/n 2.1"x1.11" 1.85WDatel ADS-935MC 5Mhz +/-2.75 400 3pipeline 83dBmin 75dBmin -86dBmax +/-30ppm/C +/- 1LSB +1LSB 13.78s/n 2.1"x1.11" 2.85WLinear Tech LT1604 333Khz +/-2.55 ? ? 88dBtyp -100dBmax +/-15ppm/C 36 SSOP 0.2WLinear Tech LT1620 500Khz +/-2.5 ? 380ns 90dBmin -88dBtyp +/-15ppm/C +/- 2LSB 36 SSOP 0.42WAnalog Devices 7671 1-0.8Msps +/-2.5/5/10 948/1.63K no pipe 90dB typ TBD external +/- 1LSB external 48 LQFP .15WAnalog Devices 7677 1-0.8Msps -.1 to 3VDE ? no pipe 94dB 20Khz n/a -110dB external +/- 1LSB +/- 1LSB external 48 LQFP .13WAnalog Devices 7676 500Ksps +/-2.5 ? no pipe 94dB 20Khz n/a 94dB 20Hz external +/- 1LSB external 48 LQFP .15WAnalog Devices 9260Analog Devices 9826 15MSPS 3CHNL 0 t0 4V* 3pipeline TOTAL 3LSB int? +/- 16LSB +/- .5LSB 13s/n 28 SSOP 400mWAKM semi SPT8100 (13ENOB) 5Mhz 0 to +5V 5.5K ohms 200ns X6 cyc 78dBmin 80dB typ -84max ? +/- 1.25LSB +/- .5LSB ? 44 LQFP 0.465mWMaxim MAX1200 1Mhz 0 t0 4V* 55K ohms 4us(pipeline) 83min -82max external +/- 3.5LSB +/- .6LSB external 44MQFP
PanSTARRS Internal Spec Review 8/5/2003
AD9826 ADC Preliminary Testing 1
• Fabricated small test PCB with same pinout as Analogic AD4322 (2MHz 16 bit) .
• 2 PCBs tested in earlier version array controller.• Quick tests showed ~X3 gain vs AD4322s, but 11-
12 ADUs Standard Deviation (range 9-12) in 1 channel Sample and Hold mode.
– Data sheet says “3LSB” PGA=1– PGA set =1, sampling freq ~5.6Mhz– Running Raytheon 206 multiplexer (3 ADUs STD
nominal)• Tested with all 8 channels and image with
Raytheon 206 mux to confirm signal gain and S/N.
PanSTARRS Internal Spec Review 8/5/2003
AD9826 ADC Preliminary Testing 2
• Bench tested with precision voltage source, logic analyzer• 3 channel CDS mode has 0-1 ADUs STD.
– PGA set =1, sampling freq ~1.9Mhz, AC coupled.– Device only wants Vreset more positive than Vsignal.– Test not valid if grounded input = 0 volts into ADC.– First/few? pixels bad.
• NOAO results:– CDS and S&H modes have 7-8 ADUs noise @ 1.6usec– Removed 2nd stage premap stage = removed “capacitive charge effect”.
None seen at IfA– “Note, apparently the AFE devices were designed to be used in a free
running mode so that the capacitive effects relating to charging the AC coupling cap to the bias clamp voltage are eliminated.” Will investigate, IfA setup had different offset/clamp circuit.
• Will retest next week to confirm results.• Samples of TI ADCs received.
PanSTARRS Internal Spec Review 8/5/2003
Clocking
Will build on previous (Redstar3) system VHDL design• 16,000 gate Altera FPGA EP6016• 128kx16 Sync SRAM• Clocking FPGA
– patterns– 7 x 1/48Mhz = 145.83nsec min pattern time– 1/48Mhz = 20.8333nsec pattern time extension– 10 bit pattern run length extension = 21.313usec – rows and frames
10 bit rep count = 1024 times– integration timer– 25usec time increments– 28 bit timer = 3.728 hrs max ITIME
PanSTARRS Internal Spec Review 8/5/2003
Pixel server = Thin Server 1U rackmount PCs
• Compaq Proliant DL360 G3 under test
• Supermicro dual Xeon under test
•Pixel servers in use on previous design with good results (faster than dual 32 bit 40MHz DSP system).
PanSTARRS Internal Spec Review 8/5/2003
OTA Controller Power Supplies
• HP 66000 System Power Supply, 8 modules/chassis• Each module has over voltage and over current protection
– Simultaneous shutdown• Configured for simultaneous turn on and off• Proven low noise and safety performance on Mauna Kea (IRTF, SUBARU)
– UPS run down and HELCO black/brownouts
• Mechanical Specifications 19 inch rackmount• Width: 426 mm (16.75 in)• Height: 178 mm (7 in)• Depth: 678 mm (26.7 in)
PanSTARRS Internal Spec Review 8/5/2003
Staffing Estimates
• Needed: 5 FTEs estimated, plus use of Contract Manufacturers 2 Electronic engineers (Onaka,TBD) 2 Software engineers (Lockhart, Isani *) 1 Electronic Tech (Ching)
• Presently: 3+ FTEs on staff– 1 new EEI causal hire => direct hire position hire started– + 1 SE interviewing*– 1 EEII (with data comm, ISCSI ASIC/IC experience) in discussion
• Contract Manufacturers (CM) for board assembly– Ching site visited 3 CMs.– IfA H85 AO 10 board run being sent through one CM as first run.
PanSTARRS Internal Spec Review 8/5/2003
Production Plan
• Two phase plan– Prototype phase
Use Contract Manufacturer (CM) for high density parts (i.e. BGA). But obtain in house ability to rework - requires some infrastructure investment
at IfA. Contract out parts of electronics design that is “industry standard” (i.e. FPGA
CompactPCI VHDL).
– Production Phase Use Contract Manufacturer (CM) to get quantity discounts. Contract out design of automated test electronics. JTAG for digital? Build multiple copies of low cost test station. Have a conservative (high) spares count.
PanSTARRS Internal Spec Review 8/5/2003
Lab Support/Infrastructure
• In-house capability to do BGA rework.– BGA inspection microscope received training done.– BGA Xray microscope received training 8/8/03.
• Previous Redstar2 and Redline controllers available for comparative measurements.
• Switching CAD software to leverage new hire experience.– Protel to Cadence (for SPECCTRA router)
Cadence university donation request sent (with UH Physics). Software download done, waiting for license.
• Xilinx FPGA requires additional tools– Symplicity VHDL synthesizer received.– Mentor Modelsim simulation models on order (not received).
PanSTARRS Internal Spec Review 8/5/2003
Development Task BreakdownPrototype Phase
FPGA3U
Basic Clock FPGA code Backplane design Trigger Clock Down
load
ADC read code
Proto Chassis
Buy Dev Xilinx Board Proto FPGA3U
Centroid and shift
Single OTA test
Proto Interface Bd
Analog Testing
DAQ3U
Proto DAQ3U Backplane interface
Interface to FPGA3U
Trigger design
Dual OTA test
COTS CPU + 1G ethernetDev
environmentMulti OTA
codeCentroid and
shift
Proto User Interface
1 G ethernet
testsData Export testing
Interface to DAQ3U
Pixel Server Workstation
1 G ethernet
tests
Analog testing
Centroid and shift
Data Storage testing
Component selection
PanSTARRS Internal Spec Review 8/5/2003
Schedule
Phase I End Goal: 4X1 OTA controller finished by Dec 2003Hardware
– Components selected Mid Aug.– PCB layout done = Oct.– Flex cabling finished Nov.– Controller prototype Dec-Jan.
Software– 1 G ethernet throughput tested – Aug. – CompactPCI throughput tested – Aug.– Centroid/shift/load – Aug/Sept.– Pixelserver code – Nov.– OTA control development during Integration Oct-Dec.
PanSTARRS Internal Spec Review 8/5/2003
Draft Workpackage Description
Prototype Phase 1.0:Predesign 1.0.X
• Workpkg 1.0.1– Title: Location of OTA centroid/clocking generation.– Type: Study+test+decision– Description: – Combination study and test code generation for embedded
PowerPC on FPGA vs. individual Pixelserver vs. top level Pixelserver.
– Metrics: Appropriate centroid algorithm, speed of calculation vs collection, clocking pattern FPGA download and swtch time.
– Estimate: 60hrs Software Engineer, 30hrs EEII.– Dependency: FPGA clocking VHDL code at testable level on
ML300.
PanSTARRS Internal Spec Review 8/5/2003
Draft Workpackage Description 2
Prototype Design workpackages: 1.1.X
• Workpkg 1.1.1– Title: Basic OTCCD clocking control– Type: Design+code+test– Description:
Software code of clocking, test of download. Hardware design on ML300, schematic and VHDL for CAD input for PCB.
– Metrics: OTCCD Sequenced readout functionality, speed.– Estimate: 60Hrs SE, 120hrs EEII, 80hrs SE*.– Dependency: none.
PanSTARRS Internal Spec Review 8/5/2003
Workpackages
Prototype Phase 1.0:Predesign 1.0.X
Workpkg 1.0.1Title: Location of OTA centroid/clocking generation.Type: Study+test+decisionDescription: Combination study and test code generation for embedded PowerPCon FPGA vs. individual Pixelserver vs. top level Pixelserver.Metrics: Appropriate centroid algorithm, speed of calculation vscollection, clocking pattern FPGA download and swtch time.Estimate: 60hrs Software Engineer, 30hrs EEII.Dependency: FPGA clocking VHDL code at testable level onML300.
Workpkg 1.0.2Title: Number of FPGAs per controller.Type: Study+decisionDescription: Study of #FPGA I/Os + interboard connector vs. added powercomplexity of 2nd FPGA per controller. Metrics: Power consumption, PCB area available, connectorsavailable, analog signal chain integrity.Estimate: 40hrs EEII.Dependency: none.
Workpkg 1.0.3Title: 1G ethernet from FPGA or embedded COTS CPU?Type: code+test+decisionDescription: Combination lab test of ML300 dev platform 1Ge throughputvs. PCI+COTS CPU (MENmicro). Metrics: Sustained throughput > = 64MBytes/secEstimate: 60hrs Software Engineer,40hrs EEII.Dependency: none.
PanSTARRS Internal Spec Review 8/5/2003
Workpackages 2
Workpkg 1.0.4Title: Real CCD test of AD9826 ADC on Leach systemType: Design+fabrication+testDescription: Design, fab and test AD9826 ADC on actual CCD to determinenoise and speed performance acceptability. Metrics: Read noise spec = CCD and = data sheet at 1MSPSEstimate: 20hrs EEII remaining (ET hrs used also).Dependency: none.
Workpkg 1.0.5Title: Release 1.0 of requirements for controllerType: DesignDescription: Hardware and software requirements document.Metrics: Completeness, science and control goals.Estimate: 20Hrs SE, 20hrs EEII.Dependency: LAN/WAN interface, Telescope scheduler reqs.-------------------------------------------------------------------Prototype Design workpackages: 1.1.X
Workpkg 1.1.1Title: Basic OTCCD clocking controlType: Design+code+testDescription: Software code of clocking, test of download. Hardware design on ML300, schematic and VHDL for CAD input for PCB.Metrics: OTCCD Sequenced readout functionality, speed.Estimate: 60Hrs SE, 120hrs EEII, 80hrs SE*.Dependency: none.
PanSTARRS Internal Spec Review 8/5/2003
Workpackages 3 see draft plan
Workpkg 1.1.2Title: FPGA PCB Type: Design+code+CAD+fabricate+testDescription: Production of FPGA printed circuit board.Metrics: Power dissipation, size, speed of interfaces.Estimate: 40Hrs EEI, 60hrs EEII, ET estimate TBD*.Dependency: Predesign phase.
Workpkg 1.1.3Title: DAQ3U PCB Type: Design+code+CAD+fabricate+testDescription: Production of DAQ3U printed circuit board.Metrics: noise performance at speed, size, power dissipation.Estimate: 40Hrs EEI, 60hrs EEII, ET estimate TBD*.Dependency: Workpkg 1.0.4.
Workpkg 1.1.4Title: Cryo flex/PCB Type: Design+CAD+fabricate+testDescription: Production of rigid-flex cable assembly. Cyrotest integrity.Metrics: noise performance,size, vacuum performance, durability.Estimate: 40Hrs EEI, 30hrs EEII, ET estimate TBD*.Dependency: OTCCD pinouts final.