deliverable d1.3 hierarchical routing node analysis · the deliverable d1.3, hierarchical routing...

Converged Heterogeneous Advanced 5G Cloud-RAN Architecture forIntelligent and Secure Media Access

mÃ¶p

Project no. 671704Research and Innovation ActionCo-funded by the Horizon 2020 Framework Programme of theEuropean Union

Call identifier: H2020-ICT-2014-1Topic: ICT-14-2014 - Advanced 5G Network Infrastructure for theFuture InternetStart date of project: July 1st, 2015

Deliverable D1.3Hierarchical Routing Node Analysis

Last change date: April 01, 2017

Publication date: July 30, 2017

Submission date: March 31, 2017

Deliverable leader: Marian Ulbricht

Editors: Andreas Foglar, Christian Liss

Review: Mike Parker, Kai Habel

Dissemination Level

�3 PU: Public

� PP: Restricted to other programme participants (including the Commission Services)

� RE: Restricted to a group specified by the consortium (including the Commission Services)

� CO: Confidential, only for members of the consortium (including the Commission Services)

mailto:[email protected]





Contents

1. Introduction 9

2. TrustNode router design 102.1. Reconfigurable hardware architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1. Design flexibility and block design . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.2. Packet processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2. Hardware design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3. TrustNode software design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.3.1. FPGA serial interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3.2. FPGA MMI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3.3. FPGA Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3.4. Userspace software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3. Latency and security concept 173.1. Forwarding modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2. Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.2.1. 6Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.2.2. FlowEngine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2.3. Traffic Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.2.4. TrustNode Timing Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3. Security of CHARISMA nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.4. SDN enabled configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4. Examination 324.1. Latency and Jitter measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.2. GTP adaption – an example showing the device’s flexibility. . . . . . . . . . . . . . . . 33

5. Conclusion 34

A. TrustNode VHDL programming example 37

B. 6Tree Rest interface definition 38

CHARISMA – D1.3 – v1.0 of 38

List of Figures

1. TrustNode Platform Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102. TN internal architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113. TN ingress processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134. TrustNode Mainboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145. Important software blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156. Trustnode packet forwarding modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177. 6tree example network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188. Fast routing decision in the downstream direction on a 4-port router. . . . . . . . . . 199. Packets and Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2010. Rule Table entry types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2211. Flow Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2312. TrustNode Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2413. M/D/1 configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2514. Queue Length depending on load and packet loss . . . . . . . . . . . . . . . . . . . . . 2615. SyncE cascade of TrustNode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2816. Clock Synchonisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2917. Secure, hierarchical CHARISMA node network. . . . . . . . . . . . . . . . . . . . . . . 3018. TrustNode as OpenFlow Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3119. IXIA measurement setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3220. Average propagation latency measurement in 6Tree mode . . . . . . . . . . . . . . . . 3221. Marking packets according to the inner IP information . . . . . . . . . . . . . . . . . . 3322. CHARISMA node in metal case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34


List of Tables

1. Flow acceleration methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172. Concatenated fields for Hash generation . . . . . . . . . . . . . . . . . . . . . . . . . . 223. TrustNode quality of service (QoS) Classes Configuration . . . . . . . . . . . . . . . . 244. TrustNode QoS Classes Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255. Example Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27


Acronyms

API application programming interface

APLL analog PLL

ARP address resolution protocol

AXI advanced extensible interface

CAL converged aggregation level

CLI comand line interface

CPU central processing unit

C-RAN cloud RAN

DEMUX demultiplexer

DNS domain name system

DSCP diffserv code point

DUT device under test

FCS frame check sequence

FPGA field programmable gate array

FTP file transfer protocol

GbE gigabit Ethernet

GPRS general packet radio service

GTP GPRS tunneling protocol

HW hardware

ICMP internet control message protocol

IoT internet of things

IP internet protocol

IPv4 IP version 4

IPv6 IP version 6

JTAG joint test action group

LPC low pin count

MAC media access control

MACSec MAC security

MMI memory mapped interface

MMU memory management unit

MTU maximum transmission unit


MUX multiplexer

NAPI new network API

NoC Network-on-Chip

PABX private branch exchange

PCB printed circuit board

PCIe peripheral component interconnect express

PHY physical layer chip

PLL phase lock loop

PSTN public switched telephone network

PTP precision time protocol

QoS quality of service

RAN radio access network

REST representational state transfer

RGMII reduced gigabit media independent interface

RISC reduced instruction set computing

RRH remote radio head

RTCP realtime control protocol

SDN software-defined network

SIP session initiation protocol

SMBus system management bus

SNMP simple network management protocol

STP spanning tree protocol

SW software

SyncE synchronous ethernet

TCP transmission control protocol

TCXO temperature-controlled crystal oscillator

TDMA time division multiple access

ToD time of day

ToS type of service

TSN time-sensitive networking

UDP user datagram protocol

VHDL very high speed integrated circuit hardware description language


VLAN virtual local area network

VoD Video-on-Demand

VoIP voice over IP


Executive Summary

The deliverable D1.3, Hierarchical Routing Node Analysis, is part of the specification work in WP1of the CHARISMA 5G-PPP project. The first deliverable, D1.1, "CHARISMA intelligent, distributedlow-latency security C-RAN/RRH architecture" [6] identified the need of a research platform for ex-perimenting low latency and low jitter packet processing for a reliable backhaul. The TrustNodehardware (HW) platform was selected as possible device to fulfill CHARISMA requirements. In deliv-erable D1.2, "Refined architecture definitions and specifications" [7], the current development statusof the TrustNode was reported. These documents introduce the hierarchical converged aggregationlevel (CAL) concept from device/gateway (CAL0) up to the core (CAL3) and specify the their re-quirements. This deliverable D1.3 describes the specified improvements of the TrustNode, partiallyimplemented within WP2 and first evaluation results up to M21. In section 2 the basic offloadingconcept of the TrustNode is described, and the respective HW/SW split in field programmable gatearray (FPGA), ATOM processor and memory mapped interface (MMI). Concept and implementationof reconfigurable HW, driver and software (SW) are detailed. These features allow the use of TrustNodeat different CAL levels. Clear user interfaces are defined at board, FPGA and SW/driver level to allowoperation and adaptation of the TrustNode by project partners. In section 3 the CHARISMA en-hancements to the TrustNode are described. Latency and jitter reduction starts at concept level withthree flow acceleration methods: hierarchical routing, Ethernet switching and a dynamic flow cache.A practical QoS solution with explicit bandwidth allocation has been elaborated for low latency andnetwork slicing support. Network timing support is provided for clock synchronization and time ofday (ToD) synchronization. The inherent Layer 3 Security of the hierarchical routing is complementedby link security via MACSec. The special section 3.4 is devoted to the mapping of OpenFlow functionsto TrustNode HW/SW. In section 4 the current state of implementation is examined, in particularlow latency, low jitter and flexible configurability. Appendix A gives an example of the modular HWsource code along with a simple modification example. Appendix B describes the representationalstate transfer (REST) interface of the device for integration into the CHARISMA demonstrator.


1. Introduction

The following document describes the CHARISMA hierarchical routing node, which has been developedduring the task T1.3 of the project and is based on an adapted version of the InnoRoute TrustNodeRouter. This text extends the information provided in the deliverables D1.2 and D2.1. [7, 8] TheCHARISMA network has been designed to have a hierarchical structure, so as to exhibit low latencyoperation, and as a basis for achieving virtualised security. This has resulted in the need for the specialrequirements and capabilities of the connecting node devices that are described in this deliverable. Inparticular, the routing node device developed in the CHARISMA project, called TrustNode, benefitsfrom the intrinsically hierarchical network structure of the CHARISMA architecture, and also positivelyimpacts upon low latency packet forwarding and network security. The TrustNode device’s structureis very flexible, making it appropriate for further implementation in the data plane of the CHARISMAarchitecture, as well as in the control plane. As part of TrustNode’s operation, a new IP version6 (IPv6) based routing paradigm, 6Tree, has been developed to accelerate the packet processing inhierarchical networks, such as the CHARISMA architecture. The document is structured as follows:a detailed system description of the hierarchically node is described in section 2, followed by a deeperview on the security and latency aspects in section 3, and finally a proof of concept discussion insection 4.


2. TrustNode router design

The TrustNode router consists of two main parts: a low-latency version of the FlowEngine [19] imple-mented in the FPGA, and the Control Processor Intel ATOM, as shown in Figure 1. Packets enterand leave the TrustNode via Gigabit Ethernet ports at the FPGA and are processed by the FPGAand optionally also by the ATOM central processing unit (CPU). The connection between FPGA andATOM subsystem is implemented as a high bandwidth PCIe interface. The idea here is that most ofthe packets (>90%) are forwarded directly by the FPGA, i.e. packets enter the FPGA via a GigabitEthernet port and are forwarded to another Gigabit Ethernet port directly, without passing the ControlProcessor. Hence the FlowEngine acts as an offloading engine for the CPU. The routing software runson the Control Processor, whose main task is to process the packets and execute the ensuing actions.Typically control packets such as from the address resolution protocol (ARP), internet control messageprotocol (ICMP), spanning tree protocol (STP), and simple network management protocol (SNMP)etc. are processed by the Control Processor, while e.g. voice and video streams are forwarded bythe FPGA only. A special exception are time stamp packets, various precision time protocol (PTP)requests and ICMP message-creating events, which could be answered directly by the FPGA withautomatic timestamp update or message creation, respectively.

Figure 1: Overview on the architecture of the Trustnode device, showing the two main components:CPU and FPGA, connected via PCIe. All Ethernet interfaces are connected to the FPGA.

The TrustNode as shown in Figure 1 is a platform that is scalable in number of ports, throughput andprocessing power. In the first implementation a typical FPGA for business projects has been chosen.Yet, the initially chosen ATOM processor is over dimensioned for such a device and can be replaced bymore or less powerful processors; for example, if very low processing power is requested, for example a


reduced instruction set computing (RISC)-based (e.g. ARM-based) based solution could easy be used.The Gigabit ports could also be replaced by 10-Gb/s ports.

2.1. Reconfigurable hardware architecture

The TrustNode architecture is shown in Figure 2 with the packet processor "FlowEngine" in the upperpart and the processor subsystem containing the Control Processor in the lower half. The FlowEnginefunctions as a data pipe which forwards packets from left to right and modifies them on the fly. Allpackets are stored in the Shared Buffer from where they can be sent out to each physical or logicaloutput port. The Rx and Tx labels in Figure 2 denote the receive and transmit parts of the GigabitEthernet physical layer chips (PHYs). The control processor (CPU) is attached like an additional PHY,but with high throughput PCIe interface. With 4 lanes running at 5GT/s a total throughput of up to20Gb/s can be achieved, so that all received packets could be transferred to the Control Processor andstored there in the DRAM. For the internal data path interfaces the advanced extensible interface (AXI)bus from ARM is used.

Figure 2: The internal architecture with FlowEngine and CPU subsystem. Showing ingress path andegress path with scheduler and queuing manager in the middle. The CPU is connected viaa logical Ethernet PHY to the input MUX and output DEMUX.

2.1.1. Design flexibility and block design

For HW modifications or extensions, relevant parts of the design are provided as VHDL source code(see Appendix A). Basic modules such as the Ethernet framing interfaces, serial-to-parallel conversion,packet multiplexing and queuing mechanism are fixed, so that basic operation (packet forwarding) is


always assured. Ethernet switching with MAC learning and forwarding and the 6Tree algorithm areprotected against manipulation as well. Implementation of new forwarding mechanisms, modificationsor new functions can be affected in the ingress data path (Figure 3), in the scheduler and in the egressdata path.). A simple block design enables the possibility to easily extend or modify the packets’ datapath.

2.1.2. Packet processing

The Ingress Processing path is shown in Figure 3. In a first step the packets arriving from all inputsare submitted to header parsing, with the extracted fields forwarded as sideband signals along with thedata path. The subsequent blocks investigate the extracted header fields. The 6Tree1 unit ignores theLayer 2 media access control (MAC) header and instead only inspects the Layer 3 fields. Packets of typeIPv6 and with the prefix of both source and destination address starting with ’0xAF’ (0b1010 1111)are considered as 6Tree packets. The 6Tree algorithm includes source verification and destinationfinding, such that the output is derived from the IPv6 destination address prefix according to the6Tree mechanism (see section 3.2.1), while other packets are forwarded to the MAC lookup. TheMAC lookup block checks the Layer 2 frame header and forwards Ethernet frames without virtuallocal area network (VLAN) and with destination MAC address not matching the TrustNode’s ownMAC address, according to the destination stored in the MAC table. If the MAC address and therespective output are not found in the MAC Table, the frame is forwarded with the destination addressof the Control Processor and the source MAC address is learned. The Control Processor performs theFlooding function by replicating the packet to all other outputs. The aging function is done by theFlowEngine by scanning the MAC Table entries periodically. All other frames are forwarded to theFlow Cache unit shown in Figure 11. In particular, VLAN tagged frames are processed by the FlowCache, such that the TrustNode supports VLAN tagging, VLAN removal and VLAN updating.

1see section 3.2.1


Figure 3: The processing of the 3 possible flow types in the ingress path. Incoming data is clarifiedinto 6Tree, MAC table traffic and Flow Cache traffic. Default traffic is routed to the CPUlater in the egress data path, see Figure 2.

2.2. Hardware design

Figure 4 gives an overview of the components placed on the router mainboard. The central elementis the Xilinx Artix-200T FPGA which is connected via reduced gigabit media independent interface(RGMII) interfaces to each of the 12 Ethernet PHYs. Two of the Ethernet PHYs are synchronousethernet (SyncE) enabled, so the TrustNode is able to be part of a hierarchical clock-synchronizednetwork. The Intel Atom-E3800 CPU-module is modularly mounted on the mainboard, so that itcan be easily replaced. Besides the main FPGA chip, a smaller one is placed on the mainboard toact as the board controller to set up the power and clock distribution. The main board contains anintegrated joint test action group (JTAG) interface which enables the CPU to reconfigure the mainFPGA during runtime.

2.3. TrustNode software design

The software running on the Intel ATOM CPU is optimized for routing tasks. Here, the system usesthe well-known embedded Linux distribution OpenWRT, such that running on the X86-64 architecturemeans that this solution can be highly optimized for the routing tasks while still being flexible enoughto port further software to the system. The OpenWRT build system includes packet management


Figure 4: PCB version 1.1 with all its mounted components. The CPU module can be mounted on theupper right, as indicated. MMI is a proprietary memory mapped interface to access stateand configuration tables on the FPGA.

so that all software written is provided in OpenWRT-BuildRoot-Packages and is easily portable toother architectures. [22] Figure 5 shows the main parts of the implemented software, where thecommunication between operating system and FPGA is realised using serial-, MMI-, and Ethernet-based interfaces.

2.3.1. FPGA serial interfaces

The serial interface connection drivers establish a basic interface between the FPGAs and the CPU,while the technologies used are the system management bus (SMBus), low pin count (LPC)-bus, JTAG,and fast serial communication2. The interfaces are used for mainboard signalling, and the debuggingand configuration of the main FPGA.

2Using proprietary mode of FTDI chip.


http://www.ftdichip.com/

Figure 5: Important blocks of the integrated processor sub-board from the software point of view,comprising three driver blocks used for the communication with the hardware according totransferred data types. The physical interfaces are mapped one by one to internal virtualinterfaces. The MMI and serial interfaces are used for (re)configuration during runtime.

2.3.2. FPGA MMI

The MMI interface driver utilises the memory management unit (MMU) and PCIe express subsystemto establish a memory mapped communication interface between FPGA and CPU. This enables theCPU to directly configure and modify the flow cache rules which are handled by the FPGA packet core.

2.3.3. FPGA Ethernet

To enable the CPU to transceive Ethernet packets from the physical interfaces, a custom Ethernetdriver has been written, with this software aligned to the open source version of IXGBE-drivers fromIntel. [10] Using zero-copy and new network API (NAPI) technology the driver reaches maximumperformance on forwarding packets. [12] As shown in Figure 5 the driver maps the 12 physical Ethernetinterfaces onto the same number of virtual interfaces inside the Linux network stack. This enables otheruser space software to use the TrustNode platform as a common 12-port switch.


2.3.4. Userspace software

The installation of the well-known OpenFlow implementation OpenVSwitch enables the TrustNoderouter to act as a node in a software-defined network (SDN)-enabled environment. From a networkpoint of view, the hardware acceleration of the device is not noticeable, with the OpenVSwitch actingas a transparent endpoint for the open flow protocol. The plugin OpenVSwitch-offload designed forOpenVSwitch automatically offloads rules established in the software OpenFlow table to the hardware-accelerated flow cache. This concept is described in more detail in the section 3.4. A Python-basedREST server gains access to all other TrustNode specific functionalities, which are not mapped tothe OpenFlow protocol. This represents the detailed configuration of the 6Tree network mask, whichis essential to enable the SDN orchestration to deal with this special feature. The detailed interfacedefinition can be found in the Appendix B. A set of comand line interface (CLI)-based managementtools exposes other administrative functions, like the FPGA reconfiguration to authorized users. Forsecurity reasons, these features are not directly exposed via the SDN interfaces.


3. Latency and security concept

3.1. Forwarding modes

As mentioned in the earlier section 2.1.2 the TrustNode router supports 4 different flow types (6Tree,Flowcache, MAC table and default), with 3 of them forwarded by the FlowEngine only, as shown belowin Figure 6. The default flow is processed by the Control Processor. The acceleration flow types are

Figure 6: Structure overview showing the 4 data forwarding modes default, FlowCache MAC table and6Tree.

listed below in the Table 1. Here, the Flow Cache is controlled by the Control Processor, while 6Tree isa hardwired mechanism, and MAC table forwarding is autonomously learning and aging out accordingto the Ethernet switching standard. Accelerated flows are forwarded in a much quicker fashion; thelowest latency of less than 3µs being achieved only by such accelerated. From the three accelerationmethods, only the Flow Cache can be controlled by the user.

Method Description Type6Tree Hardwired algorithm, destination extracted from packet StaticFlow Cache Configured by user via control processor DynamicMAC Table Learned MAC address according to IEEE 802.1 Autonomous

Table 1: Flow acceleration methods


In this case, there are two basic operation modes, transparent mode and automatic mode, that areavailable:

• In Direct Control Mode, the user has full control over the Flow Cache entries. User softwarerunning on the Control Processor decides if a flow is to be accelerated or not.

• In Automatic Mode, a daemon running on the Control Processor automatically decides if a flowis to be accelerated. The decisions depend on configurable parameters such as flow rate, flowpersistence, protocol type etc. Note that this mode is being developed in the scope of the 5GSELFNET project.

The Flow Cache is a basic mechanism needed to make an OpenFlow Switch with the TrustNode. Allflow acceleration types are described in more detail in section 3.2.2.

3.2. Latency

The following section describes the latency-aligned optimisations of the routing node. As alreadydescribed, the high speed data processing is done inside the FPGA, with the following technologiesused to speed up this task.

3.2.1. 6Tree

This routing concept is based on the IPv6 protocol, particularly targeting hierarchically-organisednetworks that can use 6Tree as a means to simplify the packet routing algorithm, such that thefollowing concept can particularly derive benefit from this property. [23] In this case, the basic ideais that in a hierarchically-structured network the IPv6 address prefix is evaluated gradually by therouters. Figure 7 shows an example of a routing network with three hierarchy levels, where each router(rectangular box) is assigned an address by manual configuration. The top-level router gets the emptyaddress <>, the second highest hierarchy router gets a one-digit address <1>, <2> or <3>, and thethird hierarchy level routers get 2-digit addresses assigned. Each of the lowest level routers has up tothree terminals (rounded-corner boxes) attached, which each gets assigned a 3-digit address.

Figure 7: Hierarchically network example showing the 6Tree address scheme.


Figure 7 also shows three paths through the network:

• Path 1 from node 221 to node 223 (medium dashed).

• Path 2 from node 112 to node 133 (long dashed)

• Path 3 from node 113 to node 321 (dotted)

In this case, it is clear that path 1 is undertaken just along the lowest router hierarchy; path 2 needstwo of the hierarchies; and path 3 goes up all three hierarchies to the top-level router. While inthis example each router has 3 ports, the 6Tree network specifies 10 outputs. With 10 outputs perrouting node it is possible to use telephone numbers as IPv6 address prefix, as described in a patentof Korea Telecom, e.g.. [4] In this case, each router evaluates one digit of the prefix. Each numberrepresents a unique address on a world-wide basis, and we can assume that most people across theworld have a (landline) telephone number. Hence, the routing network is comparable to the publicswitched telephone network (PSTN); such that routers in a private network are therefore comparableto private branch exchanges (PABXs). [5, 11] The reference Ulbricht and Wagner [23] suggests thata hierarchically-ordered address plan can dramatically decrease the time packets spend in networkingdevices. Figure 8 shows this simplification in the downstream data path of a 4-port router. The

Figure 8: Concept of 6Tree port determination using two bits of the packets address to choose theoutput port of the packet. [23]

address modification eliminates the output port search from the routing process by just using specifiedaddress bits as an output port value. The implemented routing only handles packets starting with theprefix ’AF’; so 6Tree packets can be clearly be distinct from normal IPv6 traffic. This concept can actas an add-on for fast routing, and the operation of conventional devices handle all the other packets atthe normal speed. Using 6Tree addresses means that some of the network flexibility normally expectedwill be sacrificed to the needs for a lower latency and the higher security requirements (see section 3.3).The lower address flexibility will not tend to affect end user devices, because current communicationstechnologies also ensure that a common internet protocol (IP) address is not fixed assigned to an enduser. Services are addressed via the domain name system (DNS), and IP addresses have to be distinctonly for the time of the connection between two partners. These concepts will work also whilst usinglatency-optimized hierarchically communications.


3.2.2. FlowEngine

The idea of the FlowEngine is to assign each packet to a flow. Further processing of the packet withinthe FlowEngine is done per-flow as shown in Figure 9. Each packet is assigned by a Classificationmechanism to a flow by attaching a Flow-ID to the packet. The Flow-ID determines the output portof a packet, QoS class, output queue and its further processing, counting, shaping, color marking etc.It is transported within the FlowEngine in a Network-on-Chip (NoC) header which is added to eachpacket. The Flow Cache of FlowEngine version 1.0 is a table of 512 entries (note, that this number

Figure 9: Classification of packets in flows and processing of the flows.

can be increased in future versions). Each entry can be one of the 4 types described below:

• Type 1 is a Layer 2 entry with all relevant Ethernet header fields, MAC, VLAN and Ethertype.It can be used, for example, to define VLAN specific forwarding.

• Type 2 is a Layer 3 entry for IP version 4 (IPv4). It can be used, for example, to detecttransmission control protocol (TCP)/IP flows. The search pattern includes the 5-tuple (IPv4addresses, Protocol and Ports) and the diffserv code point (DSCP) field.

• Type 3 is a Layer 3 entry for IPv6. It makes use of the fact that according to RFC6437 [1]the Flow Label is a hash of a packet’s 5-tuple (IPv6 address prefixes, DSCP field and Ports).According to RFC7136 [3] the Interface Identifier should be treated as opaque (by the network)and therefore only the 64-bit prefixes of the IPv6 addresses are used. This entry is used forprioritization of a flow.


• Type 4 is reserved for user specific function. For constructing the entry these fields are providedby the parser as follows:

– Input_port (4 bit)

– Eth_dst_MAC_addr (48 bit)

– Eth_src_MAC_addr (48 bit)

– Etype (16 bit)

– Vlan_1_id (12 bit)

– Vlan_1_priority ( 3 bit)

– Vlan_1_cfi (1 bit)

– Vlan_2_id (12 bit)

– Vlan_2_priority ( 3 bit)

– Vlan_2nd_cfi (1 bit)

– Ip_v4_src (32 bit)

– Ip_v4_dst (32 bit)

– ToS (8 bit)

– protocol (8 bit)

– Flow_label (20 bit)

– IP_v6_src (128 bit)

– IP_v6_dst (128 bit)

– Traffic_class (8 bit)

– Next_header (8 bit)

– src_Port (16 bit)

– dst_Port (16 bit)

Based on this data it is easy to create new entry types by modifying a few lines of very high speedintegrated circuit hardware description language (VHDL) code and (re)synthesizing the FPGA. Anyof the above fields can be combined, while the pre-defined entry types are depicted in the figure belowFigure 10. The colours of the rectangles in Figure 10 differentiate the field types, whole the horizontalscale is in numbers of bits and shows the size of the entry. For each incoming packet all 4 entrytypes are searched sequentially in the Rule Table for matching entries. The search is done via a hashfunction, such that for each packet the specified fields are combined, concatenated and submitted to4 hash generators. The 4 hash generators reduce the bit strings of each of the 4-tuple combinationsto a 14-bit value. The hash algorithms assure that the hash values are spread equally over the totalrange of 16,384 possible values. The next step is to use the 14-bit hash values to address a lookuptable (Hash Table), which contains the 16,384 entries. Each entry is 9-bits wide and contains a pointerto the Rule Table as shown in Figure 11. With the four hash values the Hash Table is sequentiallyaddressed and the entries are used to address the Rule Table. Ideally, one of the entries in the RuleTable matches the packet fields and the respective flow is found. For OpenFlow support it is possible to


Figure 10: Four rule table entry types with location of considered fields and markers.

Tuple Packet Header Fields HashTuple 1 Eth_dst_MAC_addr : Eth_src_MAC_addr: VLAN : Etype Hash 1Tuple 2 Protocol : ip_v4_src : ip_v4_dst : src_Port : dst_Port Hash 2Tuple 3 Flow_label : ip_v6_src_prefix Hash 3Tuple 4 User specific Hash 4

Table 2: Concatenated fields for Hash generation

specify "GOTO" as an action. For each Rule Table entry action and counter table entries are available.Only an exact match is supported; i.e. it is possible to match, e.g., the IPv4dest=192.168.14.99, butnot IPv4dest=192.168.*.*. While this simplification is not suitable for a routing table, it fits wellto the idea of a Flow Cache, where only selected flows between well-defined endpoints and ports areaccelerated. This is underpinned with two examples:

For a file transfer protocol (FTP) download several flows with different port numbers are involved, butthe final bulk data transfer uses one TCP port in the range 30,000...40,000. This flow is configured inthe Flow Cache – to be forwarded within the FlowEngine, down-prioritized as background traffic.

For voice over IP (VoIP), Video-on-Demand (VoD) or video-conferencing several flows are involved forsetup and control (e.g. session initiation protocol (SIP) for VoIP, start-stop commands for VoD), butonly the final media streams are accelerated by the Flow Cache.


These flows are forwarded by the FlowEngine with realtime priority. Rule Table lookup and comparisonof the stored Tuple is done for all 4 hash values. In some cases, a few of the 4 pointers will point tounused entries of the Rule Table, while pointers could also by chance point to entries of other flows;in which case the Tuple will not match. In some cases, it is possible that 2 or more of the Tuplesmatch the packet fields. In this case a configurable priority field specifies the entry to be selected.When the Control Processor configures a new flow it calculates the hash value of the desired fields and

Figure 11: The flow classification inside the FlowCache is done by two lookups in the rule table andin the collision table. Rule table uses hash based search, collision table linear search, bothprocesses are done simultaneous.

checks if the entry in the Hash Table is free. In the unlikely event that the entry is already occupiedthe Control Processor selects a free entry in the Collision Table. Its size of 8 entries is calculated asthe expected number of collisions when distributing 512 valid entries over an ensemble of 16,384 totalentries. The term ’expected’ refers to the median value. In the unlikely case that the Collision Tableoverflows before the 512 flows are configured, the new flow must be rejected. As collisions are detectedat configuration time, there is no risk of erroneous packet forwarding. In some applications, a retrycould be made, e.g. with another port number.

3.2.3. Traffic Management

The idea of the FlowEngine is that the destination of each packet is determined by the ingress path logic.Hence when a packet is stored in the shared buffer it is simultaneously scheduled in the destination


output queue. The shared buffer of the TrustNode has a size between 100 and 10,000 packets, andthere are two output queues per port with one of them having a threshold for lower priority packetsas shown in Figure 12. They are served with a strict priority scheduler. The (smaller) high priorityqueue is served as long as it contains packets. This may lead to starvation of the (longer) lower priorityqueue; hence the load of the traffic in the high priority queue must be limited to avoid this situation.This can be achieved for example with the YouQoS concept. [18]

Figure 12: Packet scheduler with two queues handling 4 classes of traffic: low latency, controlled load,best afford and background traffic.

With the example below (Table 5) the values are: S=1200Byte and L=55500Byte. Traffic managementof the TrustNode is completed by a common threshold in the shared buffer. With these technologyapproaches, it means that up to 4 QoS classes can be supported by the TrustNode. The QoS classes

QoS Class Queue Queue Threshold Buffer ThresholdLow Latency High Prio Queue n.a. NoControlled Load Low Prio Queue No NoBest effort (Default) Low Prio Queue Yes NoBackground Low Prio Queue Yes Yes

Table 3: TrustNode QoS Classes Configuration

have the behaviour described in Table 4. The QoS classes can be grouped into two groups: rate limited(streaming) flows, and start-stop traffic. A typical example for streaming flow is VoIP, while start-stoptraffic occurs with interactive data services. Low Latency and Controlled Load class relies on a limitedbandwidth per flow. If all sources respect their bandwidth limit, maximum delay variation (jitter)and packet loss rate can then be calculated. The other two classes, Best Effort and Background, have


QoS Class Latency Rate guarantee Packet loss Traffic typeLow Latency Very low Yes Very low StreamingControlled Load Low Yes Very low StreamingBest effort (Default) Low No Low Start-stopBackground Low No Medium Start-stop

Table 4: TrustNode QoS Classes Behaviour

no rate limit. Rate control is done end-to-end via TCP or realtime control protocol (RTCP). Theseprotocols use packet losses to adjust the rate. Low Latency class is intended for very time-sensitivetraffic, such as industrial control and other internet of things (IoT) applications. As these packets arealways forwarded with priority, the latency can be below 3µs if the high priority queue is empty. Inthe case where the queue is filled, a packet has to wait for the preceding packets in the queue to betransmitted. In this case, the waiting time can be calculated as number of packets times packet sizetime link bit time; the latter being 1ns/bit in the case of 1Gb/s links. A maximum waiting time can bedefined using queuing theory. In the following the M/D/1 queuing model is assumed (see Figure 13).Here, the stochastic arrival of equal-sized packets from N independent sources is assumed. The sumrate of all flows is assumed to stay below a configured Load, which must be less than 100%. In thiscase, the system is in a steady state with controlled packet loss. More precisely, packet loss can bekept below a given value by selecting the queue length Q for the target load according to the graphicin Figure 14. In the following example for Low Latency class configuration all inputs are given with

Figure 13: Theoretical queue structure, background of the calculations for Figure 14.

a bullet, while results are marked with arrows:

• Maximum load of time critical traffic L = 10%

• Target packet loss rate < 10−11

⇒ Get maximum queue length (in number of packets) from Figure 14: 6 packets.


• Maximum packet size 200Byte

⇒ Maximum queue size for high priority queue is S = 1200Byte.

• Link rate R = 1Gb/s

⇒ Maximum delay (jitter): 9.6µs. Note that this value is the maximum queuing delay; the total delayincludes in addition the hardware forwarding delay of less than 3µs.

⇒ Maximum rate of low latency traffic: 100Mb/s. For the Controlled Load QoS class the same

Figure 14: Calculation of the needed queue length depending on the traffic load and the queue specificpacket drop probabilities for the traffic classes in Table 4.

method is applied, with the difference that the queue serving rate is the link rate minus the LowLatency rate.Example for Controlled Load class configuration:

• Maximum load of time critical traffic L = 80% (of the remaining load of 90%)

• Target packet loss rate < 10−7

⇒ Get maximum queue length (in number of packets) from Figure 10: 37 packets.


• Maximum packet size 1500Byte

⇒ Maximum queue size for low priority queue is L = 55500Byte.

• Serving rate R = 900Mb/s

⇒ Maximum delay 493µs

⇒ Maximum rate of low latency traffic: 720Mb/s.

In summary delay variation (jitter) and packet loss probability for Low Latency and Controlled LoadQoS classes can be selected freely using the recipe described in the examples above.

Table 5 summarizes the values derived in the above example for the two QoS classes.

QoS Class Min. Latency Max. Latency Jitter Packet loss rate Maximum Load **Low Latency < 3µs 13µs 10µs * 10−11 100Mb/s

Controlled Load < 3µs 496µs 493µs 10−7 720Mb/s

Table 5: Example Summary* the 10µs are achieved with a maximum packet size of 200 bytes.** 18% load are available for best effort and background QoS classes.

Note that jitter and packet loss guarantee only work if the rates of the sources are limited so that thesum of rates does not exceed the calculated limit values (rightmost column of Table 5). Some sourceshave an intrinsically limited rate, such as VoIP telephones; while for example video applications canhave a very variable rate. In this case the video stream should be split into a basic stream with almostconstant rate and a variable enhancement stream. The basic stream is mapped to the ControlledLoad class, while the enhancement stream is transported with a Best Effort class. The basic streamarrives with very high probability at the destination and maintains a stable basic picture, while theenhancement stream still has a high probability of arriving. From Figure 14 it can be seen that theaverage queue occupation is very low as compared to the maximum queue sizes. Hence most of thetime, the best effort stream finds an almost empty queue. The rates of the sources in the internet arenot controlled. In the TrustNode QoS concept all flows of a QoS class share the same resource (queue,queue range). Hence a single misbehaving flow could exceed the sum rate of flows of a QoS class, andthus cause a deterioration in the well-defined delay and packet loss rates of all other flows. Hence allpossible sources must conform to the rate agreement. If this cannot be assured by the source for someflows, it must be enforced by the network using policing methods such as colour marking (RFC2697,RFC2698). [13, 14] On network-level concepts like time-sensitive networking (TSN), consisting of a setof standards an proposals, e.g., the currently discussed by IEEE as 802.1CM [16], with time divisionmultiple access (TDMA) schemes for separating priorities and measures like frame preemption to keepthe jitter for high-priority frames low, are appropriate for the CHARISMA node, too. Another wayto keep jitter low is to limit the maximum transmission unit (MTU) to low values, e.g., using IPv6Path MTU Discovery, which is recommended for MTU’s down to 1280 bytes, but is practically usedfor much lower values. The disadvantage of this MTU discovery mechanism is that for some corner


cases it might not work reliably to have a small MTU for high-priority flows (low jitter), while keepingthe MTU for low-priority flows as high as possible (for maximum throughput).

3.2.4. TrustNode Timing Concept

The TrustNode supports SyncE according to ITU-T rec. G.8261/2/4 with:

• temperature-controlled crystal oscillator (TCXO)

• Clock derivation from uplinks and seamless switch over using high-precision phase-locked loops,digital and analog PLLs. Software-controlled reference line selection with holdover. Holdoverbased on TCXO with 4.6 ppm. Clocking of downlinks with derived analog PLL (APLL) clock.

• Automatic line selection with holdover

• Clocking of downlinks with synchronous clock

• Local clock precision ±4.6 ppm

• Time of Day (ToD) synchronization according to the PTP.

In addition to the clock frequency synchronization via SyncE, clock phase synchronization via PTP issupported to synchronize the ToD. So, as shown in Figure 15, a reference timebase can be distributedhierarchically, although the interfaces used for SyncE and PTP are independent of each other. The

Figure 15: Cascade of synchronous ethernet levels for distribution of an common clock. GbE w. SyncEmeans Gigabit Ethernet PHY with synchronous ethernet slave capability, i.e. clock recovery.


ToD synchronisation according to RFC5905 [20] is an algorithm shown in Figure 16, where a clientrequests the actual time from a server by sending a request packet and gets a reply packet back. Inthe reply packet the processing time within the server is contained, so that the client can calculate thecurrent time by assuming equal run times for both packets and subtracting the processing time.

Figure 16: Time synchronisation algorithm with compensation of the packet round trip time. [25]

The TrustNode supports this algorithm by adding time stamps to the packet at the receive time, andby updating the actual transmission time contained in the reply packet. The receive time stamp istransported within the NoC header. The Control Processor performs the packet loopback and derivesthe ToD for the local clock. According to PTPv2 the process works within one step, as the FlowEnginedoes frame check sequence (FCS) recalculation after transmission time stamp insertion.

3.3. Security of CHARISMA nodes

Targeting an all-IP infrastructure forces network designers to be aware of a secure layer 3 and layer 2communication. From a security point of view the authentication and data integrity should be providedby the network infrastructure, to provide data security beginning from the lowest level possible. Acommon routing infrastructure is flexible and doesn’t have restrictions with respect to the addresslayout. On the other hand, this flexibility makes IP-routing vulnerable to attacks focusing on packetrouting, e.g. [21, 24, 26, 2] Using IPv6 offers an interesting possibility to restructure the routinginfrastructure in both a physical and logical way. [15] One of the key design features proposed byCHARISMA is that of the hierarchically network structure. A hierarchically address layout withspecialised routing hardware can be used to speed-up the traffic forwarding. [23] Here, the CHARISMAnode offers two key technologies to address these issues. The first is MAC security (MACSec) which isimplemented in hardware and provides the integrity of the date exchanged between the devices. [17]The second technology is the fast hierarchical routing concept 6Tree. The routing of 6Tree traffic isfully implemented in hardware, as there is only one configuration parameter which tells the device itslogical position in the network (see section 3.2.1). Using this concept, the path of a packet throughthe network is strictly predefined by the value of the IPv6 address bits which are inside the protectionarea of MACSec. By applying a plausibility check for the source address on incoming packets, the


network devices are not vulnerable for address spoofing attacks, caused by the abstinence of addresssearching (see section 3.2.1). Therefore receiving a 6Tree packet, the source address and the waythrough the 6Tree network is verified which authenticates the source toward the destination. Due tothe implementation in configured hardware, the core functions of the CHARISMA node device can’tbe hacked. Also if the Linux system should be contaminated, the reconfiguration process of the mainFPGA is fully controlled by the board controller. In this way, the device can apply an additional twofactor authentication for the FPGA reconfiguration process to prevent the infected operating systemfrom unauthorized FPGA reconfiguration.

Figure 17: Simplification of an CHARISMA node based secure Network using 6Tree routing andMACSec tunnels

3.4. SDN enabled configuration

The TrustNode allows a straightforward implementation of an OpenFlow switch (Figure 18). ThisOpenFlow acceleration on the TrustNode device has been mainly developed within the H2020 projectSELFNET, so this section only gives a short description about the method. The Control Processorimplements the OpenFlow Channel, while the FlowEngine implements the extension to the OpenFlowdata path. [9]


Figure 18: Overview on the SDN control capabilities of the TrustNode. Showing the OpenVSwitchrunning on the CPU communication with an SDN controller.

From an OpenFlow point of view only the endpoint of the used OpenVSwitch software is visible. Duringsystem setup, no rule is installed inside the FlowCache, so all packets which are not tagged with the6Tree prefix are forwarded to the atom CPU. The customized Ethernet driver (see section 2.3.3) hidesthe FPGA architecture and provides a single virtual network interface for each physical port. TheOpenVSwitch is configured to serve all of these Ethernet interfaces. This means that the TrustNodebehaves as a standard OpenFlow switch In the initialisation phase. According to the OpenFlowprotocol, new rules can then be subsequently installed into the flow tables of the OpenVSwitch. Asmall plugin for the OpenVSwitch data path (see section 2.3.4) sniffs every change to these flowtables, by patching a simple hook into the data path module of the OpenVSwitch. In this way, allinformation about new flows, including the pointers to the counter values are available. For a flowupdate, the software sorts all available flows according to their priority and pushes the first N flowsto the FlowTable using the MMI interface. This action accelerates the affected flows in hardware andthe packets matching this rules will not reach the CPU and the OpenVSwitch data path. For thisreason the service has to poll the relating packet counters of the accelerated flows and the PHYs linestates via MMI, and update the packet counters in the OpenVSwitch data path. This prevents theOpenVSwitch from deleting accelerated flows by reason of the idle counter.


4. Examination

4.1. Latency and Jitter measurements

Figure 19: Measurement setup for delay analyses using IXIA network analyser device.

The initial latency measurement was done using an IXIA-XMVDC3 network analyser with the deviceunder test (DUT) being connected using two short Ethernet cables between two ports of the IXIAdevice.(Figure 19) Figure 20 shows a constant value of about 2.5µs from 128 byte onwards for packetforwarding in the 6Tree mode. The measured packet jitter was about 180 ns.

Figure 20: Average propagation delay for several packet sizes in 6Tree mode.

3Thanks to HHI for sharing the device.


Apart from the pure latency, also the jitter is of highest interest, because it accumulates over severalhops in the network, so the jitter gets bigger the longer the packet travels. The initial jitter was inthe range of 1µs, but was reduced to around 500 ns and is expected to be further be reduced to 250 nsfor the standard case without output congestion by using more sophisticated arbiters. For precisionscheduling the jitter could even be reduced to 1

10 of that, allowing for very precise network-wide TDMAschemes in networks synchronized with SyncE.

4.2. GTP adaption – an example showing the device’s flexibility.

This example show how the how the flexible block structure of the data path can be used to implementadditional functions. In the SELFNET project fast SDN, and the controlled possibility of processingGPRS tunneling protocol (GTP) tunneled packets is needed. [9] In particular, the functionality tochange the flow label of the outer IP layers according to the information provided inside a GTPtunneled packet is also necessary. (see Figure 21) To add this functionality, a new block has beeninserted into the datapath which simply matches the data of the user datagram protocol (UDP), GTPand inner IP header through a pattern and a wildcard mask. Based on this information the flowid(see Figure 11) of each packet can be changed at the line speed to cause a new action and effect amodification of the Flow Label or type of service (ToS) field.

Figure 21: Protocol stack for the inner to outer IP packets marking.


5. Conclusion

The document has described the CHARISMA hierarchical routing node in its concept, in implemen-tation on InnoRoute’s TrustNode and first experimental verification. In section 2 the FPGA-basedCPU-assisted design was introduced which was developed as a base for fast, secure and flexible trafficforwarding. The four flow types defined in section 3.2.3, ensure a latency optimized and smart traffichandling. The innovative routing protocol 6Tree which was introduced in section 3.2.1, enables a se-cure and latency optimized communication. Measurements in section 4 showed a propagation latencyvalue from just 2.5µs for different packet sizes. The hardware level security is based on the strict6Tree routing and on MACSec, as described in section 3.3. The smart combination of configurablehardware based packet processing and CPU support enables the easy integration of this, quiet special,device in standard SDN environments. (see section 3.4). The user interfaces for hardware and soft-ware programming have been explained with examples so as to demonstrate that the TrustNode is aversatile platform for hierarchical routing, for experimentation of new SDN functions and for any net-working protocol which needs hardware acceleration. Ultra-low latency makes the TrustNode suitablefor the real time requirements in the CHARISMA hierarchical aggregation network, but also for future5G challenges such as for example industrial networks with closed loop control. Figure 22 shows theCHARISMA node, ready to use, packaged in its metal casing.

Figure 22: CHARISMA node box in metal case, showing the front panel for simple 6Tree prefixmodification.


References

[1] S. Amante et al. IPv6 Flow Label Specification. RFC 6437 (Proposed Standard). Internet Engi-neering Task Force, Nov. 2011. url: http://www.ietf.org/rfc/rfc6437.txt.

[2] Hitesh Ballani, Paul Francis, and Xinyang Zhang. “A study of prefix hijacking and interceptionin the Internet”. In: ACM SIGCOMM Computer Communication Review. Vol. 37. 4. ACM. 2007,pp. 265–276.

[3] B. Carpenter and S. Jiang. Significance of IPv6 Interface Identifiers. RFC 7136 (Proposed Stan-dard). Internet Engineering Task Force, Feb. 2014. url: http://www.ietf.org/rfc/rfc7136.txt.

[4] H.G. Choi et al. Internet addressing architecture and hierarchical routing method thereof. USPatent 6,868,416. 2005. url: https://docs.google.com/viewer?url=patentimages.storage.googleapis.com/pdfs/US6868416.pdf.

[5] H.C. Choo and S.J. JANG.Method of operating internet protocol address and subnet system usingthe same. US Patent 7,693,163. 2010. url: https://www.google.com/patents/US7693163.

[6] CHARISMA consortium. “D1.1 CHARISMA intelligent, distributed low-latency security C-RANRRH architecture”. In: 5g-ppp.eu (2016).

[7] CHARISMA consortium. “D1.2 Refined architecture definitions and specifications”. In: 5g-ppp.eu(2016).

[8] CHARISMA consortium. “D2.1 Initial Architecture Design and Interfaces”. In: 5g-ppp.eu (2016).

[9] SELFNET consortium. “D3.4 Report and Prototype Implementation of the NFV & SDN Sensorsand Actuators related to the Self-Optimizing Use Case”. In: 5g-ppp.eu (2017).

[10] Intel Corporation. Linux Base Driver for the Intel Ethernet 10 Gigabit PCI Express Family ofAdapters. 2013. url: https://www.kernel.org/doc/Documentation/networking/ixgbe.txt.

[11] A. Foglar and S. Sonntag. Router IP port for an IP router. US Patent 7,499,450. 2009. url:https://www.google.com/patents/US7499450.

[12] Linux Foundation. “Linux Foundation Wiki: napi”. In: linuxfoundation.org (2016). url: https://wiki.linuxfoundation.org/networking/napi#napi-driver-design.

[13] J. Heinanen and R. Guerin. A Single Rate Three Color Marker. RFC 2697 (Informational).Internet Engineering Task Force, Sept. 1999. url: http://www.ietf.org/rfc/rfc2697.txt.

[14] J. Heinanen and R. Guerin. A Two Rate Three Color Marker. RFC 2698 (Informational). InternetEngineering Task Force, Sept. 1999. url: http://www.ietf.org/rfc/rfc2698.txt.

[15] R. Hinden and S. Deering. Internet Protocol Version 6 (IPv6) Addressing Architecture. RFC 3513(Proposed Standard). Obsoleted by RFC 4291. Internet Engineering Task Force, Apr. 2003. url:http://www.ietf.org/rfc/rfc3513.txt.

[16] IEEE. “802.1CM - Time-Sensitive Networking for Fronthaul”. In: http://www.ieee802.org (2017).url: http://www.ieee802.org/1/pages/802.1cm.html.


http://www.ietf.org/rfc/rfc6437.txt



https://docs.google.com/viewer?url=patentimages.storage.googleapis.com/pdfs/US6868416.pdf

https://docs.google.com/viewer?url=patentimages.storage.googleapis.com/pdfs/US6868416.pdf

https://www.google.com/patents/US7693163

https://www.kernel.org/doc/Documentation/networking/ixgbe.txt

https://www.google.com/patents/US7499450

https://wiki.linuxfoundation.org/networking/napi#napi-driver-design

https://wiki.linuxfoundation.org/networking/napi#napi-driver-design




http://www.ieee802.org/1/pages/802.1cm.html

[17] IEEE. “IEEE Standard for Local and Metropolitan Area Networks: Media Access Control (MAC)Security”. In: IEEE Std 802.1AE-2006 (2006), pp. 1–150. doi: 10.1109/IEEESTD.2006.245590.

[18] C. Liss et al. “YouQoS - Combining Quality of Service with Network Neutrality”. In: BroadbandCoverage in Germany. 9th ITG Symposium. Proceedings. 2015, pp. 1–6.

[19] Christian Liß et al. “Architecture of a Synchronized Low-Latency Network Node Targeted toResearch and Education”. In: 2017 IEEE 18th International Conference on High PerformanceSwitching and Routing (HPSR) (IEEE HPSR’17)[in review]. Campinas, Brazil, June 2017.

[20] D. Mills et al. Network Time Protocol Version 4: Protocol and Algorithms Specification. RFC 5905(Proposed Standard). Internet Engineering Task Force, June 2010. url: http://www.ietf.org/rfc/rfc5905.txt.

[21] Shuaishuai Tan, Xiaoping Li, and Qingkuan Dong. “TrustR: An Integrated Router SecurityFramework for Protecting Computer Networks”. In: IEEE Communications Letters 20.2 (2016),pp. 376–379.

[22] OpenWrt developer team. OpenWRT wireless freedom. url: https://openwrt.org/.

[23] Marian Ulbricht and Jens Wagner. “Accelerated Processing Delay Optimization in HierarchicalNetworks Using Lowcost Hardware”. In: 2016 10th International Symposium on Communica-tion Systems, Networks and Digital Signal Processing (CSNDSP) (CSNDSP16). Prague, CzechRepublic, July 2016.

[24] Feiyi Wang, Brian Vetter, and Shyhtsun Felix Wu. Secure routing protocols: Theory and practice.Tech. rep. Technical report, North Carolina State University, 1997.

[25] Wikipedia. Network Time Protocol — Wikipedia, The Free Encyclopedia. [Online; accessed 16-March-2017]. 2017. url: https://en.wikipedia.org/w/index.php?title=Network_Time_Protocol&oldid=769287376.

[26] Jian Yin and Sanjay Kumar Madria. “A hierarchical secure routing protocol against black holeattacks in sensor networks”. In: IEEE International Conference on Sensor Networks, Ubiquitous,and Trustworthy Computing (SUTC’06). Vol. 1. IEEE. 2006, 8–pp.


http://dx.doi.org/10.1109/IEEESTD.2006.245590



https://openwrt.org/

https://en.wikipedia.org/w/index.php?title=Network_Time_Protocol&oldid=769287376

https://en.wikipedia.org/w/index.php?title=Network_Time_Protocol&oldid=769287376

A. TrustNode VHDL programming example

Modifying the source code of the TrustNode FPGA design can be done without deep knowledge ofVHDL language thanks to the clear and well commented code as shown in the Flow Cache exampleof the following listings. This example shows the encoding of Hash 1 of table 3.2.2, consisting of thefields Eth_dst_MAC_addr : Eth_src_MAC_addr: VLAN : Etype Hash 1 is generated in two steps,hash over the individual fields and XOR over these hash values:� �

1 −−reduce Source MAC Address va lue to 14 Bit by xoreth_src_mac_addr_hash_s <= eth_src_mac_addr_s (13 downto 0) xor

3 eth_src_mac_addr_s (27 downto 14) xoreth_src_mac_addr_s (41 downto 28) xor

5 ( "00000000" & eth_src_mac_addr_s (47 downto 4 2 ) ) ;−−reduce Des t inat i on MAC Address va lue to 14 Bit by xor

7 eth_dst_mac_addr_hash_s <= eth_dst_mac_addr_s (13 downto 0) xoreth_dst_mac_addr_s (27 downto 14) xor eth_dst_mac_addr_s (41 downto 28) xor

9 ( "00000000" & eth_dst_mac_addr_s (47 downto 4 2 ) ) ;−−reduce Ethertype f i e l d va lue to 14 Bit by xor

11 ethertype_hash_s <= ethertype_s (13 downto 0) xor( "000000000000" & ethertype_s (15 downto 14 ) ;

13 −−extend ID f i e l d va lue o f f i r s t VLAN to 14 Bit by adding 0 s vlan_1st_hash_s<= "00" & vlan_1st_id_s ;

15 −− f i n a l xor the 4 component : Source MAC, Des t inat i on MAC, Ethertype , VLAN IDhash_r1_s <= eth_src_mac_addr_hash_s xor

17 eth_dst_mac_addr_hash_s xor ethertype_hash_s xor vlan_1st_hash_s ;

� ��

1 −−extend ID f i e l d va lue o f second VLAN to 14 Bit by adding 0 svlan_2nd_hash_s <= "00" & vlan_2nd_id_s ;

3 −− f i n a l xor the 4 component : Source MAC, Des t inat i on MAC, Ethertype , VLANIDhash_r1_s <= eth_src_mac_addr_hash_s xor eth_dst_mac_addr_hash_s xor

5 ethertype_hash_s xor vlan_1st_hash_s xor vlan_2nd_hash_s ;

7� �


B. 6Tree Rest interface definition

The 6Tree network-prefix can be configured using the front panel of the router. For SDN applications,an optional REST-JSON interface can be provided.

POST: URL: <ip>:6363/setprefix� �j son−input :

2 {" p r e f i x " : s t r i ng ,

4

}6 j son−output :

{8 " s t a tu s " : [ "ok" , " e r r o r " ] ,

" l og " : s t r i n g10 }� �


deliverable d1.3 hierarchical routing node analysis · the deliverable d1.3, hierarchical routing...

Documents