infiniband fpga
DESCRIPTION
InfiniBand FPGA. Iowa State University Senior Design May 09-04. Matthew Wall Rachel Ayoroa Xiang Li Ryan Schwarzkopf Tim Prince Alex Burds Adviser: Dr. Morris Chang Client: Troy Benjegerdes. What is InfiniBand. Switched fabric communication link 2.5 Gbit /s signaling - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/1.jpg)
InfiniBand FPGAIowa State University Senior Design
May 09-04
Matthew WallRachel AyoroaXiang LiRyan SchwarzkopfTim PrinceAlex Burds
Adviser: Dr. Morris ChangClient: Troy Benjegerdes
![Page 2: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/2.jpg)
What is InfiniBand
Protocol Stack› Physical Layer› Link Layer› Network Layer› Transport Layer
Switched fabric communication link
2.5 Gbit/s signaling Primarily used in
supercomputers Specification maintained
by the InfiniBand Trade Association
![Page 3: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/3.jpg)
IP Cores
Reusable hardware cores accelerate development
E.g. DSP, communication controllers, cryptographers, memory, processors, Ethernet MAC
Opencores.org
![Page 4: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/4.jpg)
Problem/Need
Currently have to use proprietary hardware interface (e.g. Mellanox)
Can’t perform research on internals Need a open-source hardware
implementation
![Page 5: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/5.jpg)
Design Concept Hardware core in
VHDL Compatible with
InfiniBand specifications
Implementation on an FPGA or in simulator
Xilinx FPGA Chip
InfiniBand HDL Implementation
Physical Layer
Xilinx RocketIO
Link Layer
Fabric
![Page 6: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/6.jpg)
Operating Environment
Proprietary Environments Xilinx ISE (Development Environment) ModelSim (Simulator) Virtex-5 Development Board
![Page 7: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/7.jpg)
User Interface
No UI in conventional terms VHDL Synthesizer VHDL Simulator FPGA pinout
![Page 8: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/8.jpg)
Functional Requirements
FR01: The system shall provide InfiniBand compliant input/output signals.
FR02: The system shall be able to transmit and receive packets according to InfiniBand specifications.
FR03: The system shall be able to handle errors according to InfiniBand specifications.
![Page 9: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/9.jpg)
Nonfunctional Requirements
NFR01: The system shall be written in VHDL.
NFR02: The system shall be distributed under an open source license.
NFR03: The system should use only standard VHDL libraries.
NFR04: The system should be compatible with open source VHDL simulators.
![Page 10: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/10.jpg)
Market Research
InfiniBand Trade Association (Cisco, IBM, Intel, etc.) maintains specification
No open-source IP core competitors Specification is freely available/open
![Page 11: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/11.jpg)
Deliverables
CD containing:› Source Code (VHDL)› Compilation Instructions› User’s Manual
Submit to OpenCores.org
![Page 12: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/12.jpg)
Project Plan
![Page 13: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/13.jpg)
Work BreakdownInfiniBand Project
Planning Documentation Design Implementation Testing
Scheduling
Resources
Risks
Bound Report(s)
User’s Guide
Compilation Instructions
Switch Controller
Link Layer
Physical Layer
Device Integration
Requirements
Top-Level Design
Unit Testing
Layer Integration
Testing
Device Integration
Layer Design
Layer Component
Design
![Page 14: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/14.jpg)
Resources
Team Members (Alex Burds, Tim Prince, Matt Wall, Ryan Schwarzkopf, Xiang Li)
Ryan Schwarzkopf (Communications Coordinator)
Matt Wall (Project Leader) Jason Boyd (Lab Coordinator) Dr. Morris Chang (Project Advisor) Troy Benjegerdes (Client) Lab with Virtex-5 board and compatible
synthesizer (Coover 2041)
![Page 15: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/15.jpg)
Schedule Overview
Implementation begins – 1/12/09 Layers completed – 4/3/09 Layers tested – 4/22/09 System integration completed –
4/26/09 Integration testing completed – 4/26/09 System testing completed – 4/29/09 Documentation completed – 5/4/09 Bound Report completed – 5/4/09
![Page 16: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/16.jpg)
Identified Risks
Complexity of system may prevent complete hardware implementation
Complete testing may require more time than is available
May need to obtain additional test equipment
![Page 17: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/17.jpg)
Detailed Design
![Page 18: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/18.jpg)
System Overview Two Layers
› Link› Physical
Top-down design approach
Xilinx FPGA Chip
InfiniBand HDL Implementation
Physical Layer
Xilinx RocketIO
Link Layer
Fabric
![Page 19: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/19.jpg)
Link Layer Handles the sending and receiving
of data across the links at the packet level
Provides addressing, flow control, and error detection
Virtual Lanes provide buffering Link State Machine responds to
changes in the link status
![Page 20: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/20.jpg)
Components Link State Machine Packet Receiver
Machine Data Packet Check
Machine Link Packet Check
Machine Flow Control
Generator Virtual Lane
Arbitration CRC Generators
Link Layer
To Physical Layer
Link State Machine
Packet Reciever State Machine
Data/Link Packet Check Machine
Virtual Lane 0 Virtual Lane 15 (Subnet Management)
Flow Control Packet Generator
Packet Priority Selector
Virtual Lane Selector
![Page 21: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/21.jpg)
Virtual Lanes
Buffer packets to and from Link Layer Two VLs required, VL0 and VL15 VL15 reserved for subnet management
packets
![Page 22: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/22.jpg)
Flow Control
Prevent buffer overflow Send flow control packets periodically Updates credit information from Virtual
Lane Stop transmitting if receiver doesn’t
have enough credits
![Page 23: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/23.jpg)
Packet Check MachinesData Packet Check Machine Variable packet headers and payloads
mean more robust error checking Error checks performed by DPCM
› VCRC and ICRC check› Link version check› Destination Local Identifier check› VL and VL15 checks› Buffer availability check› Verifies the length of the packet
Reports errors in correct precedence and discards packets with errors
Error free packets passed to Virtual Lane
![Page 24: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/24.jpg)
Packet Check Machines
Link Packet Check Machine Only one type of packet (flow control
packet) Checks for errors within a link packet
› Verifies Link Packet CRC› Flow control opcode is flagged› Verifies correct virtual lane is selected and supported› Verifies the length of the flow control packet (6 Bytes)
Reports errors in correct precedence and discards packets with errors
Error free packets passed to Virtual Lane
![Page 25: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/25.jpg)
CRC Generators
Two CRC Types Variant
› Entire Packet (Including Payload)› Changes between transmissions
Invariant › Data payload› Static
![Page 26: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/26.jpg)
Physical Layer Maintains
physical link Three main
components› Link/Phy Layer› RocketIO Adapter› RocketIO Module
Physical Layer
Encoded Lanes
Link/Phy Layer
RocketIO Adapter
Physical Interconnects
Physical RocketIO Module
Link Layer Byte Streams
![Page 27: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/27.jpg)
Link/Physical Layer Manages link
training process Maintains link
state once completed
Inserts control sequences to allow clock synchronization
Link/Physical Layer
L.T.F.S.M.
TX Buffer
RX BufferIdle
SequenceSkip
SequenceTraining
SequenceError
Status
Control Sequence Detectors
To Link Layer
To RocketIO Adapter
![Page 28: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/28.jpg)
Link Training FSM Maintains link
state Handles errors
Polling
Configuration
Link Up
Recovery
Config OK
Config Failed
Received TS1
Recovery OK
Major Error or Link initiated
recovery
Recovery Failed
![Page 29: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/29.jpg)
RocketIO Adapter Manages Xilinx
RocketIO operation› Clock generation
and recovery› 8b/10b Encoding
Converts Link/Phy control signals into the equivalent RocketIO signals
![Page 30: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/30.jpg)
Testing
![Page 31: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/31.jpg)
Simulation Testing
Simulate individual components to determine if they behave as expected
Integrate components in a layer to determine if they work together
Use ModelSim VHDL simulation
![Page 32: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/32.jpg)
Integration Testing
Once individual layers are simulated, we will progressively integrate them together and physically test them
Phy/Phy testing › Connect the Physical layer programmed on a
single board through loopback and ensure that a data stream can be transmitted
![Page 33: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/33.jpg)
Integration Testing Contd.
Phy+Link/Phy+Link› Add the Link layer and determine if packets
can be transmitted.› Also check for packet integrity and flow control
behavior
![Page 34: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/34.jpg)
Test Harness
PPC440CPU Core
PLB
Test Harness Instance
Test Harness Instance
Test Harness Instance
Virtex-5 XC5VFX70T
RocketIO RocketIO RocketIO
Connector
SMA Cable (loopback)
Connection to an InfiniBand device or another test harness
Xilinx ML507 Development Board
Connector (e.g., SMA) Connector
![Page 35: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/35.jpg)
Test Harness
Physical Layer
TX
Link LayerTX
RocketIO
RX FIFO
Physical Layer
RX
Link LayerRX
TX FIFOControl Register
Control Signals
PLB Interface
Instance of the link and physical layers
PLB control interface
Programmable interconnects
Inject and extract data into and out of the data-path
![Page 36: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/36.jpg)
Development
![Page 37: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/37.jpg)
Accomplishments
Physical Layer compiles Physical Layer simulation Physical Layer physical testing Link Layer compiles Link Layer simulation Performed extended testing with a
5-10 Giga-sample oscilloscope
![Page 38: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/38.jpg)
Results
![Page 39: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/39.jpg)
Results
![Page 40: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/40.jpg)
Challenges Debugging the serial loop back Switching from Vertex-II to Vertex-5
board Only having 1 Vertex-5 board VHDL intense project Lost a team member
![Page 41: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/41.jpg)
Conclusion Have a springboard for further
development Further verification and development is
needed for a robust IP core
![Page 42: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/42.jpg)
Lessons Learned Evaluate project complexity in greater
detail Divide tasks into smaller parts Set stricter guidelines for tasks Obtain clearer end goal from client Define specific skill set required
![Page 43: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/43.jpg)
Future Work Verification of layers against
specifications in detail Potentially port to different transceivers Create an open hardware development
platform Make more robust for usability
![Page 44: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/44.jpg)
Demo
![Page 45: InfiniBand FPGA](https://reader036.vdocuments.site/reader036/viewer/2022081505/568163d8550346895dd52980/html5/thumbnails/45.jpg)
Questions
?