pci express over ip - accelerated - missing link electronics · pci local bus from software view...
TRANSCRIPT
(c) 2016 MLEcorp.com EW2016-Session18-MLE-FPGAAccellWithHLS 1
PCI Express over IP - Accelerated
We are a Silicon Valley based technology company with Offices in Germany. We are Partner of leading electronicdevice and solution providers and have been enabling key innovators in the automotive, industrial, test & measurement markets to build better Embedded Systems, faster.
Our Mission isTo develop and market technology solutions for Embedded Systems Realization via pre-validated IP and expert application support, and to combine off-the-shelf FPGA devices with Open-Source Software for dependable, configurable Embedded System platforms
Our Expertise isI/O connectivity and Acceleration of data communication protocols, additionally opening up FPGA technology for analog applications, and the integration and optimization of Open Source Linux and Android software stacks on modern extensible processing architectures.
Dr. Endric Schubert, Univ. Ulm & Missing Link ElectronicsAndreas Braun, Missing Link ElectronicsUlrich Langenbach, Fraunhofer HHI
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 2
Introduction and Motivation
PCIe to Ethernet Throughput Matching
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 3
PCI Express
Transaction
Data Link
Physical
Application
PCIe Layer
HeaderStart LCRCSeq. Payload ECRC
Transaction Layer Packet (TLP)
4 B 2 B 12/16 B 128/256 B 4 B 4 B
● PCIe replaces the PCI Local Bus (backwards compatible)● Full-duplex serial transmission ● At 8GT/s line rate (Gen3) on up to 32 lanes● Packet-based protocol with four layers
● Data Link layer, physical layer: Reliable transport on the link● Transaction layer:
● Transport of application data, device configuration, interrupts, Quality of service
● TLP categories: Memory, I/O, configuration, message
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 4
PCI Express
● PCIe is point-to-point. Hierarchical system topologies via switches
CPUPCIe
Endpoint
Endpoint
Endpoint
RootComplex(RC)
DP
PCIe
PCIe
PCIeUP
DP
Switch
● ID based routing (bus/device/function number) and address based routing● Transactions require completion (split-transaction) or posted transactions
possible
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 5
PCI Express
● PCI Local Bus from software view (addressing, driver, configuration, …)● PCIe devices implement a set of registers (configuration space)● PCIe topology needs to be explored at the beginning of system start-up● Enumeration of devices by completing Configuration-TLPs
BRBR
BR
Switch
Bus1 Bus2 Bus4
EP
EP
RC
Bus3
Range problem: Physical line length of PCIe on PCBs is limited to the centimeter range
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 6
State-of-the-art
InfiniBand:
ExpEther:
Source: Nec Corporation
Source: Mellanox
● Standard HPC interconnect
● PCIe Gen2 over 10GbE networks by NEC.
PCIe external cabling:● Standard for copper cables
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 9
Network Protocol Acceleration Platform Architecture
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 10
PCIe over IP
Transaction
Data Link
Physical
Application
PCIe Layer
HeaderStart LCRCSeq. Payload ECRC
Transaction Layer Packet (TLP)
4 B 2 B 12/16 B 128/256 B 4 B 4 B
IP
MAC
Ethernet
TCP
Network Layer
Application
TCP
IPMAC
TLP
Ethernet
Src.MACPreamble FCSSFD Dst.MAC Type/Len
8 B 1 B 6 B 6 B 2 B 46 B – 1500 B 4 B
TLP
TCP/IP Packet
TLP TLP TLP TLP TLP
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 11
Concept of PCIe-over-IP
FPGA 1
TLP2TCP
TCP2TLP
FPGA 2
TCP/IP
Ethernet
PCIe PCIe End-point
Host PC Device
TCP/IP 1 TCP/IP 2
Root Complex
UPSWport
DNSWport
Networkstack
Networkstack
TLP2TCP
TCP2TLP
TLP 1PCIe 1 PCIe 2 TLP 2
...... ...PCIe 1 PCIe 2TLP 2
...TLP 1
TCP/IP TLP 1 TLP ...
Distributed PCIe Switch based on “XPressRICH3” PCIe IP Core from PLDA
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 12
Implementation
Interfaceadapter
+TLP-aggr.
UPPCIePHY
NPAP
Sidebandcon.
TLPcon.
Server config
DNportedit
10GbEPHY
Interfaceadapter
+TLP-aggr.
DPPCIePHY
NPAP
Sidebandcon.
TLPcon.
Config+
Completer+
Device edit
10GbEPHY
LANMain-board
Device LAN
FPGA1 – Upstream Side
FPGA2 – Downstream SideClient config
PERST#
PERST#
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 13
Implementation
RC EPPCIe PCIe
10GbE
UP Port DN Port
Local Remote
ZC706ZC706
FMC-to-PCIeNVMe SSD
CPU
Setup
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 14
Implementation
TLP aggregation
TCPIP
MAC
TLP
(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 15
ImplementationLocal switch
UP Port DN Port
Distr. switch with TLP aggregation
RC EP
PCIe
UP Port
Local
ZC706
FMC-to-PCIeNVMe SSD
CPU
DN PortPCIe
RC EPPCIe
PCIe
10GbE
UP Port DN Port
Local Remote
ZC706ZC706
FMC-to-PCIeNVMe SSD
CPU
(c) 2016 MLEcorp.com EW2016-Session18-MLE-FPGAAccellWithHLS 16
Conclusion
● Reliable “tunneling” of PCI Express via TCP/IP● Fully transparent to PCIe Root Complex and Operating System ● FPGA processing enables bandwidth at 10 GigE line rates
● Based on “XPressRICH3” PCIe IP Core from PLDA
● Please visit us at the show:
HHI – Hall 4 Booth
MLE – Hall 2 Booth 2-421
Xilinx – Hall 1 Booth 1-205