june 2000 msr design 1 washington washington university in st louis the washington university active...

June 2000 MSR Design 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

The Washington UniversityActive Network

Software Framework

John DeHartWashington University

[email protected]://www.arl.wustl.edu/~jdd

http://www.arl.wustl.edu/arl/projects/msr/ann_sw.pdf


Overview

• Some thoughts on active networking

• Software Architecture

• Router Plugins

• Active Router Plugins

• Multi-Port Router Architecture


Who Writes the Active Code

• End User - “classic” active networking– “Its my data, here’s how I want to manipulate it …”

• Application/Service Developer– “Its my application, I understand the format of it best,

here’s some capabilities for manipulating it …”

– e.g. Vendor X provides Active congestion control for its MPEG video conferencing system.

• Network Provider/Developer– “Its my router/network, I’ll give you these capabilities

for manipulating your data …”

– e.g. Alternate Routing and Topology based decisions.


Gre

ater

Fle

xib

ilit

yG

reater Perfo

rman

cePerformance: Executing the Active Code

• User Space Execution Environments– ANTS from MIT

– CANES from Georgia Tech

– PLAN from UPenn

– ...

• Kernel Space Execution Environments– Scout from Arizona

– ANN (Active Network Node) from WU

– ...

• Hardware (e.g. FPGAs)– P4 from UPenn

– FPX from WU


Requirements for WU AN Platform

• Code should be deployed automatically and on-demand to router– this should happen rarely!

• Simple security model we understand– secure plugins and code servers

– do not verify each packet’s secureness

• Fast enough for data-path applications – 155 Mbits/s in software

– 1.2 Gbits/s with hardware support (FPGA)


Software Architecture (NetBSD)

KernelPlugins

PluginCtl. Unit

Active Plugin Loader

PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

KeyDB

PluginDB

PolicyRules

CodeServerCode

Server

ActiveCode DB

. . .anetd

JavaVM

ANTS

NOSI

PacketClassificationand Routing

IPv4/6Header

Processing

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

TCP/UDPPE Kernel Space

RSVP/SSPRouting

PE User Space,Switch Controller,& Remote Servers

SAPF Packet

Selector/Dispatcher . . .

. . .

. . .

ActiveFunction

Dispatcher

. . .

. . .. . .

Resource Controller


Software Architecture (NetBSD)

KernelPlugins

PluginCtl. Unit

PacketClassificationand Routing

IPv4/6Header

Processing

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

TCP/UDP

SAPF Packets

Selector/Dispatcher . . .

. . .

. . .

ActiveFunction

Dispatcher

. . .

. . .. . .

Resource Controller

IP Packets

• Std. proc. for “plain” IP packets.– classification & routing, header processing, output queueing

• Active packets move through configured kernel plugins.» active function dispatcher passes packets to instances of plugin objects» instantiates objects or triggers download of plugin class, as needed

• streamlined processing of SAPF packets using pre-established state


Router Plugins (Crossbow: where we started)

• Modular kernel software architecture developed at Washington U. and ETH Zurich

• Allows for – dynamically adding code modules called Plugins at

run-time into the IP forwarding loop

– binding plugins to individual IP flows

• Provides fast packet classification


Router Plugins (con’t)

• Packet classification engine greatly simplifies implementation of plugins

• Filters to bind plugins to flows

Gate 3IP Security

DESDES

IP Security

MD5MD5

FilterFilter


Plugin Identifiers for Crossbow

Ethernet

Vers HLen TO S

ID Fragm ent O ffset

Source Address

Destination Address

Tota l length

F lags

TTL Protocol Header C hecksum

O ptions (if any)

32 bits

Protocol

Source Address

Destination Address

Source Port

...

D estination Port

IPv4

TCP

• Code identified by protocol numbers, port numbers or hardware

• Plugins applied to flows selected by filters.


Plugin Identifiers

fi2 P 1 fi3 P 2 ... ... P N

• Abstract view: •Datagram = Sequence of Code identifiers (pin) and parameters (Pn)

F

• Implement code as plugins!

• Call identifiers “Plugin Identifiers”

• Plugin Identifier (PI) determines which plugin has to be called

pi2 pi3 ...


Active Packet

Type ID=DANVersion (2) F lags

32 b its

AN EP H eader Length (2) AN EP Packet Length

0x8002001

R eserved Length (12)

W aveV ideo TAG

ANEP

DAN

Version (1)

IPv4 header (P ro toco l=A N E P )

N ext H eader=U D P

Source Port D estination Port

U D P checksumU D P lengthUDP

W aveV ideo D ata

0x8002001 (Plugin ID)

Plugin ID in packet ==> NO FILTERS!!


Plugin Invocation based on Plugin IDs

Active Function

Dispatcher(AFD)

Active Function

Dispatcher(AFD)

Plugin Control Unit

IP Security

VideoCongestion Control

IP

IP

which plugin?

return plugin to gatecall plugin

lookup plugin identifierwhich plugin?return plugin


Plugin Invocation based on Plugin IDs (con’t)

Active Function

Dispatcher(AFD)

Active Function

Dispatcher(AFD)

Plugin Control Unit

IP Security


IP

User SpaceKernel

PluginManagement

PluginManagement

IP

which plugin?which plugin?plugin not present

request plugin

check policiesrequest plugin from code server


Downloading a Plugin

plugin request

Our RouterOur Router

MPEGMPEGMPEGMPEGVideoCC

VideoCC

Code ServerCode Server

Plugin DatabasePlugin Database

InternetInternet

• check policies• locate plugin• check policies• locate plugin

send plugin

Video CC


Plugin Invocation based on Plugin IDs

Active Function

Dispatcher(AFD)

Active Function

Dispatcher(AFD)

Plugin Control Unit

IP Security


User SpaceKernel

PluginManagement

PluginManagement

IP

check digital signatures

check digital signatures

load plugin

plugin present return plugincall plugin

plugin from code server

VideoCC


ipintrq:

ipintrip_forward ip_output

ip6_forward ip6_output

atmc_input

apic_isr apic_intr

ip6_input

atmc_output

input packets interrupts output packets

TRANS

tcp/udp_input tcp/udp_output

socket layer

PORT

apic_devoutput

IP Packet Handling in ANN Kernel (APIC)

Where in this loop is ANN processing done?

Any guesses?

Where is the Packet Scheduler?

Any guesses?

LINK

DATA

NETWORK

UPPER

LOWER

apic_start

packet_scheduler

PS_Q

Where is the fast forward path?

Any guesses?

Where are the gates?

sgate

dgate



• Device Driver -- Device Dependent – apic_intr()

• read INTR_ACK: 0 => not for us, return

• apic_isr()

– apic_isr()• read NOTIFY_LIST, next rcv channel needing attention

– Process descriptors for RCV channel» swap words in each received mbuf (APIC BUG)

» link mbufs until we find end of frame

» verify CRC (just check flag set by APIC)

» atmc_input(packet)

» repeat

– repeatipintrq:


atmc_input

apic_isr apic_intr

atmc_output

apic_devoutput



• Device Driver - Dev. Independent atmc_input():– LLC/SNAP processing (if needed)

• extract type (IP/RATM/…)

– AAL5 processing• extract length

• strip trailer

– if gateway and !crossbow• ipflow_fastforward()

– aiu_getafix(packet)• aiu_getafix stores a ptr to FTE in packet

– aiu_fastforward(packet)• currently commented out. WHY?

– Enqueue in IP Queue

ipintrq:


atmc_input

apic_isr apic_intr

atmc_output

apic_devoutput



• IP Input -- ipintr():– Get next packet from IP Queue

– Do some basic checks, header, length, checksum…

– process IP options

– aiu_dgate(&m,2) Router Plugins Dynamic Gate #2

– if packet is not for us and we can forward it:• forward packet ip_forward() -- upcoming slide...

• return

– aiu_dgate(&m,4) Router Plugins Dynamic Gate #4

– protocol specific input routine e.g. tcp_input()

ipintrq:ipintr ip_forward ip_output

atmc_input

apic_isr apic_intr

atmc_output

apic_devoutput



• IP Forwarding -- ip_forward():– decide if we need to send any redirects to sender

– ip_output()

• IP Output -- ip_output()– aiu_dgate(&m,3) Router Plugins Dynamic Gate #3

– get route

– check for special processing• ANEP Options

– anep_output()» if DAN then afd_handle_dan_packet() :ACTIVE PROCESSING

– send on interface

ipintrq:


atmc_input

apic_isr apic_intr

atmc_output

apic_devoutput



• Device Driver Device Indep.-- atmc_output():– LLC/SNAP processing

– AAL5 processing (if needed)

– if packet_scheduling: enqueue for PS

– if !packet_scheduling: send to devoutput()

• Device Driver Device Dep. -- apic_devoutput()– configure an APIC descriptor for each mbuf in packet

– Resume APIC TX channel

ipintrq:


atmc_input

apic_isr apic_intr

atmc_output

apic_devoutput


Packet Classification - sys/crossbow/aiu_flow.cif (ipv6 flow_label)

index = flow_label & (FLOWTABSIZE-1)

else

index = (src_addr[3] + dst_addr[3] +src_port + dst_port + protocol) & (FLOWTABSIZE-1)

FTE = aiu_flowtable[index]

while (FTE)

{

match FTE against packet for: dst_port, src_port, src_addr, dst_addr, protocol, interface, version

if match

return FTE

else

if ((!oldflow) && (FTE is OLD)) // OLD is not touched in last 10 seconds

oldflow = fte

FTE = FTE->next

}

if (oldflow)

FTE = recycle oldflow // Garbage collection done twice per day

else

FTE = allocate a new entry and insert into table

packet->m_cbh.fix = FTE // Store FTE in mbuf structure for later re-use

return FTE


Active Packet Handling - sys/ann/afd.cfix = aui_getfix(packet)

pgicode = extract Plugin Identifier from packet

while (pgicode)

acte = get class from active class table

if (!acte)

if packet wants to be queued

enqueue packet

request plugin from APL

return NULL // signals caller to not send packet on

else

request plugin from APL

return packet // signals caller to continue sending of packet

fi

else

aite = get instance from instance table

if (!aite)

send message to plugin to create a new instance

store new instance in instance table

fi

call instance entry() function

fi

done


Flow Related Tables

Flow Table(global)

.

.

.

fix = hash(packet 5-tuple)

Plugin ClassTable

(global)

.

.

.

PluginInstance

Table(1/class)

.

.

.

.

.

.

Instance

hash(pgicode)

hash(fix)


Flow Table Entry - sys/crossbow/aiu_flow.h

/* flow table entry structure */

struct aiu_flowte {

u_int32_t interface; // interface the packet was received on

u_int32_t protocol; // protocol

u_int32_t src_port; // source port

u_int32_t dst_port; // destination port

u_int32_t flow_label; // IPv6 flow label

u_int16_t flags; // various flags

long touched; // last time touched

long filtered[AIU_MAXDAGS]; // last time filtered

struct aiu_filter *filter[AIU_MAXDAGS]; // ptrs to matching filters for each type of module supported

caddr_t data[AIU_MAXDAGS]; // opaque module specific data

u_int32_t version; // protocol version

struct aiu_flowte *next;

void (*cbpgi_flowcleanup[AIU_MAXDAGS])(short, void *); // called in case the flow gets recycled

struct ann_flow ann; // info includes ptr to plugin instance (SAPF only?)

u_int32_t hits; // hit count

};


Class/Instance Table Entries - sys/ann/afd.h

/* Class table entry structure */

struct afd_class_table_entry {

u_int32_t pgicode; /* plugin code */

struct afd_inst_table_entry **instances; /* pointer to instance table */

struct afd_class_table_entry *next; /* next class in table */

};

/* Instance table entry structure */

struct afd_inst_table_entry {

struct aiu_flowte *fix; /* flow index */

struct ann_pgiinst *inst; /* instance that handles that flow */

struct afd_inst_table_entry *next; /* next instance at this spot */

};


User Space Plugin Management


PluginDatabaseController

SecurityGatew ay

PluginRequester

ActivePlugin

Database

to CodeServer

KeyDatabase

PolicyController

PolicyRules

Active Function Dispatcher

DAN Plugin Management

User SpaceKernel



PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

KeyDB

PluginDB

PolicyRules

CodeServerCodeServer

ActiveCode DB

PE User Space,Switch Controller,& Remote Servers

Control Processor

. . .

anetd

JavaVM

ANTS

NOSI

APLAN EE’s Services

SC

. . .

Routing

RSVP

Multi-Port Router Architecture

8-P

ort A

TM

Sw

itch

(20G

b/s)

AssociationIdentification Unit

(AUI)

Packet Scheduler(PS)

Netw ork Device Driver (DD)

ActiveFunction

Dispatcher(AFD)

IPv4/v6forw arding

RSVPd/routed Proxy

Use

rK

erne

l Class X Class Y

Instances

Plugin Control Unit(PCU)

Selector Dispatcher(SD)

Plugin Managem ent Proxy


(AUI)



ActiveFunction

Dispatcher(AFD)

IPv4/v6forw arding

RSVPd/routed Proxy

Use

rK

erne

l Class X Class Y

Instances





(AUI)



ActiveFunction

Dispatcher(AFD)

IPv4/v6forw arding

RSVPd/routed Proxy

Use

rK

erne

l Class X Class Y

Instances





(AUI)



ActiveFunction

Dispatcher(AFD)

IPv4/v6forw arding

RSVPd/routed Proxy

Use

rK

erne

l Class X Class Y

Instances





ANPE Software Architecture

Network Device Driver (DD)

Class X Class Y

Instances


Plugin Management Proxy

ActiveFunction

Dispatcher(AFD)


Instances


(AIU)


IPv4/v6Forwarding

RSVPd/routed Proxy


Summary

• ANN software architecture– built on Router Plugins

– offers secure, automated kernel plugin download

– provides reasonably good performance • optimizations possible, e.g., polling vs. interrupts, SAPF

shortcut processing

• Runs on the Smart Port Card and on standard PCs

• Integral Part of MSR and NMVC


Discussion

• Crossbow vs. ANN• Performance Measurements

– What do we want to learn?

• NetBSD redesign for SPC/MSR– User space daemons vs. kernel plugins for updates

– Hardware/Software Interrupt usage

– Redesign IP path for better performance

– Redesign Interrupt structure for better performance

june 2000 msr design 1 washington washington university in st louis the washington university active...

Documents