udp packet offload example - intel · each of the four udp hardware channels have a payload...

UDP Packet Offload Example

Nios II and Altera Triple Speed Ethernet MAC

2

SOPC Builder System

Typical Nios II + TSE system configuration.Typical Nios II + TSE system configuration.

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem

3

SOPC Builder System

What if we don’t want the Nios II subsystem to deal with all the network traffic?What if we don’t want the Nios II subsystem to deal with all the network traffic?

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem

PacketOffload

Hardware

4

SOPC Builder System

Hardware UDP Packet Offload Example SystemHardware UDP Packet Offload Example System

CHK

CHK

CHK

CHK

GEN

GEN

GEN

GEN

EXT

EXT

EXT

EXT

INS

INS

INS

INS API

API

API

API

MUX

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem

GEN – PRBS packet generatorCHK – PRBS packet checkerINS – UDP payload inserterEXT – UDP payload extractorAPI – alignment pad inserterAPR – alignment pad removerMUX – packet muxDEM – packet demuxUCM – UDP channel mapperEPD – error packet discardOVF – overflow packet discardTSE – Triple Speed Ethernet MACSGDMA – Scatter Gather DMA

5

SOPC Builder System

9 Custom Components – the rest is off the shelf9 Custom Components – the rest is off the shelf

CHK

CHK

CHK

CHK

GEN

GEN

GEN

GEN

EXT

EXT

EXT

EXT

INS

INS

INS

INS API

API

API

API

MUX

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


6

SOPC Builder System

Avalon ST Domain – the data streamAvalon ST Domain – the data stream

CHK

CHK

CHK

CHK

GEN

GEN

GEN

GEN

EXT

EXT

EXT

EXT

INS

INS

INS

INS API

API

API

API

MUX

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


7

SOPC Builder System

Avalon MM Domain – things Nios II can accessAvalon MM Domain – things Nios II can access

CHK

CHK

CHK

CHK

GEN

GEN

GEN

GEN

EXT

EXT

EXT

EXT

INS

INS

INS

INS API

API

API

API

MUX

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem



Ingress packet flow description.

9

SOPC Builder System

Ingress Packet Flow DescriptionIngress Packet Flow Description

CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


The ingress packet flow begins at the TSE MAC. We configure the MAC to give us all packets, valid and errored, along with the FCS portion of the Ethernet frame.

10

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


All the received packets go thru our Error Packet Discard block. This block simply monitors the Avalon ST interface looking for a packet error signal. If it sees one, it discards the packet. If you wanted to do something more intelligent with the error packets, this is the point where you could capture them. The EPD peripheral stores each entire packet so that it knows whether the packet was good or bad before it forwards the good packets along.

11

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


All good packets that we receive have the alignment padding removed from the front of the packet so that our other custom components don’t have to deal with these pad bytes. The TSE MAC is programmed to apply two bytes of alignment padding to incoming packets so that if the packet lands in a Nios II receive buffer, the IP header will be word aligned in memory. Our custom hardware components have no need for this padding, so it is removed.

12

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


The first real work that we do on the receive packet is to see if it matches any destination UDP ports that we are expecting to capture in hardware. The UDP channel mappercomponent is programmed by Nios II to assign up to 4 different UDP destination ports to specific Avalon ST channels, so that they may be demultiplexed from the packet stream and offloaded into hardware for processing. Any packet that does not match a specified UDP port will be mapped onto the default channel which is passed to Nios II.

13

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


The channelized Avalon ST packet stream exiting the UCM block is demultiplexed to route the packets to their appropriate destinations, be that one of four hardware channels, or the Nios II receive channel.

14

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


Each of the four UDP hardware channels have a payload extraction component which will receive the Ethernet packet from the demultiplexor and extract the UDP payload from the frame. The output of this payload extraction block is a simple proprietary packet format described later in these slides as the PRBS test packet format.

15

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


The end of the line for our UDP packets that get offloaded to hardware is the packet checker component which accepts PRBS test packets from the payload extraction component and it verifies various characteristics of the packet. Legal payload lengths for the PRBS packets are 0 bytes to 64Kbytes. The length of the packet payload is verified to equal the length field of the packet header. The sequence value is verified, and the PRBS data pattern is verified as well. For short PRBS packets we only verify what we have.

16

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


For packets that did not get routed into hardware for processing, we route them towards the Nios II subsystem instead. The first stop is an overflow packet discarder that we use to throw away any packets that we receive too fast for the Nios II system to consume. If we did not discard packets at this point, the Nios II system could back pressure the whole ingress path back to the TSE MAC and block the reception of high speed data destined for our hardware offload data path.

17

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


Before we give the packets to the Nios II subsystem, we need to insert the alignment padding back into the packet so that the Nios II software sees the proper packet alignment in its memory.

18

SOPC Builder System


CHK

CHK

CHK

CHK

EXT

EXT

EXT

EXT

API OVFSRC

SNK

TSEMAC

UCM APR EPDDEM

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


And the final step is to deliver the packets to the SGDMA for reception into the Nios II subsystem.


Egress packet flow description.

20

SOPC Builder System

Egress Packet Flow DescriptionEgress Packet Flow Description

GEN

GEN

GEN

GEN INS

INS

INS

INS API

API

API

API

MUX

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


Packets from the Nios II subsystem are simply generated and transmitted out the SGDMA as they normally would be. Instead of connecting straight to the TSE MAC however, the SGDMA output is multiplexed with other potential packet streams that are generated by hardware.

21

SOPC Builder System


GEN

GEN

GEN

GEN INS

INS

INS

INS API

API

API

API

MUX

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


Hardware generated packets begin in the PRBS packet generator component. This packet generator is programmed by the Nios II to create PRBS test packets of a specific length from 0bytes to 64Kbytes in length. The packet generator inserts a packet sequence number and PRBS data that can be validated at the other end of the network by the packet checker component.

22

SOPC Builder System


GEN

GEN

GEN

GEN INS

INS

INS

INS API

API

API

API

MUX

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


The PRBS packets are passed on to the UDP payload insertion component which takes the payload section of the PRBS packets and inserts it into the payload of a UDP packet for transmission over Ethernet. The Nios II CPU programs the UDP payload inserter with the required information for it to properly construct a valid UDP packet with the PRBS packet payload.

23

SOPC Builder System


GEN

GEN

GEN

GEN INS

INS

INS

INS API

API

API

API

MUX

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


Before the TSE MAC can consume our hardware generated packets, we must apply the alignment padding that it expects, since the Nios II software is creating packets in this format, we need to ensure that our hardware packets conform to this format as well.

24

SOPC Builder System


GEN

GEN

GEN

GEN INS

INS

INS

INS API

API

API

API

MUX

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


And finally all the hardware packets and Nios II system packets are multiplexed together and sent to the TSE MAC for transmission over the Ethernet.

25

SOPC Builder System


GEN

GEN

GEN

GEN INS

INS

INS

INS API

API

API

API

MUX

SRC

SNK

TSEMAC

SRCSNK

SGDMASGDMA

M M

Nios II Subsystem


As we’ve already mentioned, the TSE MAC is programmed to expect header alignment padding on the packets that it receives for transmission. Because of this we need to make sure that we program the maximum length of a packet to account for the two extra padding bytes, or the TSE MAC will generate errors during transmission of a maximum length Ethernet packet.

General facts referenced in this example.

27

Typical Ethernet FrameTypical Ethernet Frame

46 - 15002 12468Octets 6

Preamble/SFD

Source MAC AddressDestination MAC Address

EtherType Payload

Frame Check SequenceInterframe Gap

Minimum 64 octets – Maximum 1518 octets

Minimum 84 octets – Maximum 1538 octets

28

Typical UDP PacketTypical UDP Packet

HeaderLengthVersion Total LengthType of Service

Flags Fragment OffsetIdentification

ProtocolTime to Live Header Checksum

Source Address

Destination Address

Destination PortSource Port

Length ChecksumUDP Header

IP Header

Bit0

Bit32

Bit64

Bit96

Bit128

Bit160

Bit192

Bits0 - 3

Bits4 - 7

Bits8 - 15

Bits16 - 18

Bits19 - 31

Payload

Bit224

29

PRBS Test Packet Format used in example.PRBS Test Packet Format used in example.

Bit0

Bit32

Bits0 - 3

Bits4 - 7

Bits8 - 15

Bits16 - 18

Bits19 - 31

Payload

Length SequenceMSB

SequenceLSB

30

GbE Speed LimitsGbE Speed Limits

GbE line rate is 125Mbaud1 baud carries an 8-bit octetMinimum sized packet bandwidth limit is:− 1488095.24Pps

Minimum UDP payload size 18 bytes:− 26785714.29Bps

Maximum sized packet bandwidth limit is:− 81274.38Pps

Maximum UDP payload size 1472 bytes:− 119635890.77Bps

UDP Offload Example

Demo Hardware Setup

32

2SGX90 Development Board Programming2SGX90 Development Board Programming

There are a number of archives available with this example, each contains a README.TXT file that explains how to use it’s contents in conjunction with the demo. Please refer to those documents for more information.The appropriate README.TXT will describe how to invoke one of the two basic options for running this example in hardware, you can download and run it with a JTAG cable, or you can program the configuration flash on the dev board such that it powers up into the demo directly.The demonstration requires two development boards which need to be programmed with the example and connected together to perform the demonstration.

33

Top of Board Configuration SwitchesTop of Board Configuration SwitchesThe user dip switch block S5 is used by the UDP offload example to select a preprogrammed MAC address and IP address for the board.DIP Switch 1 on this block selects the network addresses, this is the far right switch labeled 1 on the switch block and 0 on the board silk screen.When switch 1 is in the UP position, the following addresses are selected:− MAC = 00:07:ED:FF:8F:10− IP = 10.0.0.2

When switch 1 is in the DOWN position, the following addresses are selected:− MAC = 00:07:ED:FF:8F:10− IP = 10.0.0.1

One of your 2SGX90 boards should have switch 1 set UP and the other should have switch 1 set DOWN.

34

Bottom of Board Configuration SwitchesBottom of Board Configuration Switches

Ensure that the configuration switches in DIP switch block S6 on the bottom of the board are configured as shown here.All switches are in the ON(UP) position except for switch 6.

1 –U

P

2 –U

P

3 –U

P

4 –U

P

5 –U

P

6 –D

OW

N

7 –U

P

8 –U

P

9 –U

P

10 –U

P

35

Connecting the demo boards.Connecting the demo boards.

Once you have programmed the 2SGX90 dev boards to run the demo you have two possible ways to connect them together to facilitate the demo.You can use a cross over cable to connect the Ethernet port on one dev board directly to the Ethernet port on the other.You can use a 10/100/1000 type Ethernet switch and connect the two development boards to the switch, along with a PC to control and interact with them.

36

Demo Setup with X-over CableDemo Setup with X-over Cable

37

Demo Setup with X-over CableDemo Setup with X-over Cable

2SGX90 PCIe Dev Board with Marvell 10/100/1000 PHY

Cross-over CableUSB Blasters communicate with InterNiche console running on target via JTAG UART

38

Demo Setup with HubDemo Setup with Hub

39

Demo Setup with HubDemo Setup with Hub

2SGX90 PCIe Dev Board with Marvell 10/100/1000 PHY

10/100/1000 Switch

Network to PC to control and interact with demo.

UDP Offload Example

Running the Demo

41

Interacting with the demo, JTAG UART.Interacting with the demo, JTAG UART.

There are two ways to interact with the UDP offload demo, thru the JTAG UART or thru the GbE network link.If you have a USB Blaster or other such JTAG connection to the development board you can communicate with the demo with the nios2-terminal.

42

The stdout from the demo at boot.The stdout from the demo at boot.

As the demo boots, you can observe the boot sequence with the nios2-terminal connection.At the top of the boot sequence you can see the MAC address and IP address that were selected by the demo. These were selected based on the User DIP Switch setting.At the bottom of the boot sequence you can see some hints on how to get more help and information from the demo.

43

Get help.Get help.

If we type the recommended command “help udpoffload” we get the help dump from the demo menu commands as shown below.The following slides will illustrate how some of these commands are used.

44

Starting a data stream.Starting a data stream.You can use the start_stream command to initiate a data stream.The start_stream command communicates with the remote stream server and requests that it send a stream of PRBS packets at the specified size to our board.The demo is configured to support up to four data streams, so we can actually initiate up to four streams by issuing the start_stream command four times.You can see that if we attempt to initiate a fifth stream the demo informs us that we’ve started all the streams that we can support.In the example below all four streams are requesting the maximum UDP payload size of 1472 bytes.

45

Monitoring the streams.Monitoring the streams.The rx_pps command can be used to monitor the real time receive throughput that we are experiencing in the demo.This command measures the receive traffic over a 5 second period and then calculates the bandwidth that is received over each channel and the aggregate total bandwidth.The bandwidth is reported in packets per second and bytes per second.As you can see in the example on the right, the aggregate bandwidth of the four maximum sized UDP streams that we started earlier are delivering the maximum number of packets to this client over the GbE.

46

Stopping the streams.Stopping the streams.

The stop_stream command is used to stop a previously started stream.We can stop a specific single channel that is running, or we can stop all running streams.In the example below we’re stopping all the currently running channels.

47

Observing CPU loading.Observing CPU loading.

If we run the cpu_statscommand we can observe what the CPU loading is for all the various tasks that are running.

48

Observing TSE MAC statistics.Observing TSE MAC statistics.

The dump_mac_info command displays all the registers available in the TSE MAC.

UDP Offload Example

Controlling the demo over the GbEnetwork.

50

Configure your workstation on the GbE.Configure your workstation on the GbE.

If you have the development boards connected to a GbE switch along with a workstation, you can interact with it over the GbE network.Configure your workstation to reside at IP address 10.0.0.3. The default addresses for the dev boards are 10.0.0.1 and 10.0.0.2 based on the User DIP Switch setting on each board.

51

Connecting to the demo boards via GbEConnecting to the demo boards via GbE

The way to connect with the demo boards over the network is to run a telnet session and login to each boards’ telnet server provided by the InterNiche stack.

52

Telnet login.Telnet login.

The default telnet username is “guest” and the default password is “guest”.So we log into each demo board on the guest account.

53

Accessing the InterNiche console thru telnet.Accessing the InterNiche console thru telnet.

Once we’ve logged in, we have access to the same InterNiche console that we were accessing previously thru the JTAG UART.As you see below, we have access to the same collection of menu commands.

54

Starting streams from both servers.Starting streams from both servers.Just as before, we can use the start_stream command to initiate streams from a remote server.In the examples below we start minimum packet streams from one server and maximum packet streams from the other.

55

Observing the simultaneous bandwidth.Observing the simultaneous bandwidth.If we use the rx_pps command on both dev boards we can see that each stream server is sending the maximum line rate of data that can be achieved for the packet size that has been requested.

56

Adding even more network traffic.Adding even more network traffic.

Since the InterNiche FTP server is also running on the demo, we can login to the FTP server and push a file onto the development board while the two boards have already saturated the GbE network with their respective hardware generated data streams.From the example below you can see that we log into the InterNiche FTP server using the default anonymous account and then we put the file “test.txt” onto the server.

57

Observing the effects of the additional traffic.Observing the effects of the additional traffic.

If we look at the two dev boards we can see the effects of the FTP put.The “test.txt” file can be seen in the directory listing on the top window.And in the bottom window we can see the effects of the additional network traffic at the TSE MAC level. You can see how the MAC on this dev board has received 104 pause packets from the Ethernet switch, which applied back pressure to the 10.0.0.2 station so that the network traffic from the PC could be worked into the packet stream and delivered to the 10.0.0.1 station.

udp packet offload example - intel · each of the four udp hardware channels have a payload...

Documents