NetServ: Reviving Active Networks

Jae Woo Lee1, Roberto Francescangeli2, Wonsang Song1, Emanuele Maccherani2, Jan Janak1,Suman Srinivasan1, Michael S. Kester1, Salman A. Baset3, Henning Schulzrinne1

1Columbia University 2Universita degli Studi di Perugia 3IBM Research

Abstract

In 1996, Tennenhouse and Wetherall proposed active net-works, where users can inject code modules into networknodes. The proposal sparked intense debate and follow-on research, but ultimately failed to win over the net-working community. Fifteen years later, the problemsthat motivated the active networks proposal persist.

We call for a revival of active networks. We presentNetServ, a fully integrated active network system thatprovides all the necessary functionality to be deployable,addressing the core problems that prevented the practicalsuccess of earlier approaches.

We make the following contributions. We present ahybrid approach to active networking, which combinesthe best qualities from the two extreme approaches–integrated and discrete. We built a working system thatstrikes the right balance between security and perfor-mance by leveraging current technologies. We suggestan economic model based on NetServ between contentproviders and ISPs. We built four applications to illus-trate the model.

1 Introduction

Tennenhouse and Wetherall presented the vision of anactive network architecture in their seminal 1996 pa-per [28]. They noted that growing demand for in-network services resulted in the proliferation of mid-dleboxes, overcoming “architectural injunctions againstthem.” By adopting active technologies already avail-able at end systems–mobile code between web serverand client, for example–they proposed to activate net-work nodes, making in-network computation and storageavailable to users.1 They argued that active networks notonly consolidate the ad hoc collection of middleboxesinto a common programmable node, but also accelerate

1We use the termusers broadly, referring not only end users, butalso application service providers and content providers.

the pace of innovation. The possibility of in-network de-ployment enables new network-based services, and thosenew ideas are no longer shackled by the slow pace of pro-tocol standardization.

It is remarkable that, 15 years later, their voice ringseven louder today. Middleboxes have continued to pro-liferate. NAT boxes are everywhere, from enterprise net-works to home networks. Web proxies and load bal-ancers are growing in numbers and capability, recentlycoining a new term,application delivery controller, torefer to the most sophisticated breed. Even traditionalrouter vendors are jumping in with SDKs to allow thirdparty packet processing modules [21].

The ossification of the network layer has gotten to apoint where researchers are no longer hesitant to call fora clean-slate redesign of the Internet, but we have yet tosee a clear winner with a serious prospect of adoption. Inthe meantime, content and application providers’ needfor in-network services are filled by application-layer so-lutions that can make suboptimal use of the network.Witness the emergence of the Content Distribution Net-work (CDN) industry.

The rise of CDNs has also contributed to a recenttrend: blurring of the lines between content providersand Internet service providers (ISPs). Some very largecontent providers–Google, for example–operate datacenters at Internet exchange points. Some traditionalISPs, on the other hand, are getting into CDN market–Level 3 hosting and delivering Netflix’s streaming video,for example. This trend highlights the benefit of operat-ing services at the strategic points within the network.

Despite the far-reaching vision, however, the advo-cates of active networks ultimately failed to win over thenetworking community. The biggest objections were thesecurity risk and performance overhead associated withthe extreme version of active networks where every userpacket carries code within it. Another important factor,in our opinion, was the lack of compelling use cases.

1

We call for a revival of active networks. Active net-working was ahead of its time when it was proposed,but we believe its time has arrived. We claim that thetechnology advances in the past fifteen years provides asolid ground on which we can design an active network-ing system that strikes the right balance to address bothsecurity and performance concerns. Moreover, we ob-serve that active networks present a compelling use casein today’s Internet economy.

We present NetServ, a fully integrated active networksystem that provides all the necessary functionality to bedeployable, addressing the core problems that preventedthe practical success of earlier approaches. Our contri-bution can be summarized in terms of resolving the fol-lowing three conflicts:

Integrated vs. Discrete We present a hybrid approachthat combines the best qualities from the two ex-treme approaches to active networking.

Security vs. Performance We built a working systemthat strikes the right balance between security andperformance by leveraging the current technologies.

Content provider vs. ISPs We suggest an economicmodel on top of the newly available in-network re-sources between content providers and ISPs. Webuilt four applications to illustrate the model.

We elaborate on the three conflicts in Section 2. Sec-tion 3 describes the resulting architecture meeting thegoals and challenges. Section 4 describes our implemen-tation on Linux. In Section 5, we expand on the securityissues. In Section 6, 7, 8, we describe our four sampleapplications, talk about our activities on GENI, and dis-cuss some issues regarding the deployment of the appli-cations. In Section 9, we evaluate our Linux implemen-tation. Section 10 describes the OpenFlow extension ofNetServ, which addresses the performance limitation ofour Linux implementation. Sections 11 discusses relatedwork. We conclude in Section 12.

2 Goals and Challenges

2.1 Integrated vs. Discrete

Active networking proposed two approaches to program-ming the network. In theintegrated approach, everypacket contains user code that is executed by the net-work nodes through which the packet travels. Many re-searchers attribute the ultimate demise of active networksto the security risk and performance overhead associatedwith user packets carrying code.

In the more conservativediscrete approach, net-work nodes are programmed by out-of-band mecha-nisms which are separate from the data packet path. In

other words, the discrete active network nodes are pro-grammable routers. Indeed, since the active networkproposal, the research community has seen many pro-grammable router proposals [11,18,20,22] which are ei-ther considered a platform for active networking, or atleast heavily influenced by it.

Notwithstanding the general view that associates pro-grammable routers with active networks, we do not con-sider typical programmable routers an adequate platformto realize the active network vision. Typical uses of pro-grammable routers center around the network functionsrequired by the network operators, like QoS, firewall,VPN, IPsec, NAT, web cache, and rate limiting. Thevariety and sophistication of available services on pro-grammable routers is a boon for network management,but it is far from the active network vision, where usersinject custom functionality into the network. In fact, weargue that programmable routers, despite their root in ac-tive networks, compound the problem that motivated ac-tive networks in the first place: proliferation of middle-boxes.

NetServ aims to be the vehicle to bring back the activenetworking vision, not just another programmable router.NetServ must provide a mechanism to inject user codeinto the network. At the same time, we cannot repeat thesame failure by adopting the integrated approach.

We take a hybrid approach. Like the discrete ap-proach, we separate the data path and the control chan-nel through which the network nodes are programmed.Like the integrated approach, however, it is the user whoprograms the network nodes. A user sends an on-pathsignaling message towards a destination of his interest,which will trigger the NetServ nodes on-path to down-load the user’s code module and install it dynamically.

2.2 Security vs. Performance

The user-driven software installation made security ourtop priority. Unlike previous programmable routers thatran service modules in (or very close to) kernel spacefor fast packet processing, NetServ runs modules in userspace. Specifically, user modules are written in Java andexecuted on Java Virtual Machines (JVMs). A NetServnode hosts multiple JVMs, one for each user.

Our choice of user space execution and JVM allows usto leverage the decades of technology advances in operat-ing systems, virtualization, and Java. NetServ makes useof isolation and resource control mechanisms available inall layers: OS-level virtualization, process control, Java2 security, and the OSGi component framework. We dis-cuss resource control and isolation further in Section 5.

Running service modules in user space, and inJava on top of that, inevitably raises the eyebrows ofperformance-mindedcritics. In Section 9, we explore the

2

most worrisome case, namely, a Java service module sit-ting in the fast data path, and performing deep packet in-spection (DPI) and modification. Every processed packetincurs the overhead of kernel packet filter, kernel-to-user(and back) transitions, transfer from native to Java code,and application code running in JVM. The evaluation ofour Linux-based implementation shows that the overheadis indeed significant, but not prohibitively so.

Our real defense against performance-related criti-cisms is the multi-box lateral expansion of NetServ us-ing the OpenFlow [24] forwarding engine, describedin Section 10. In this extended architecture, multipleLinux-based NetServ nodes are attached to an OpenFlowswitch, which provides a physically separate forwardingplane. The scalability of user services is no longer lim-ited to a single NetServ box.

We do not claim to have invented any of the individualtechnologies that we use for NetServ. Our challenge, andthus our contribution, lies in combining the technologiesto strike the right balance between security and perfor-mance, culminating in a fully-integrated active networksystem that can be deployed on the current Internet whileremaining true to the original active networking vision.

2.3 Content providers vs. ISPs

We identify two Internet actors that are currently in a tus-sle, and suggest a way to use NetServ to enter into aneconomic alliance. We have already noted that the linesbetween content providers and ISPs are blurring, whichhighlights the importance of occupying strategic pointsin the network. Those strategic points are often at thenetwork edge. Content providers are motivated to oper-ate at the network edge, close to end users, as evidencedby the success of CDN operators like Akamai.

The network edge belongs to a particular type ofISPs, often calledeyeball ISPs. As opposed tocontentISPs who provide hosting and connectivity to contentproviders,eyeball ISPs provide last-mile connectivity toend users. It has been noted that eyeball ISPs wield in-creased bargaining power in peering agreements becausetheyown the eyeballs [12]. We argue that their presenceat the network edge is another powerful asset. The edgerouters of eyeball ISPs, due to their proximity to endusers, are excellent candidates for NetServ nodes that canhost cached content and custom service modules for con-tent and application providers. For content providers, alarge number of NetServ nodes spread out at the networkedge would create an attractive alternative to CDNs. Foreyeball ISPs, a cluster of NetServ nodes at the networkedge provide another source of revenue.

We envision that the economic alliance between con-tent providers and ISPs will be facilitated by brokerswho aggregate resources from different ISPs, arrange re-

NetServ packet transport

Virtual execution

environment

Building block layer

Virtual execution

environment

Building block layer

Virtual execution

environment

Building block layer

Service modules Service modules Service modules

NetServ controller

Module download

Module install

Signaling message

to install module

Signaling message

forwarded to next hop

Data packets processed

by service modules

Figure 1: NetServ node architecture.

muneration, and possibly provide value-added services.This is already happening in cloud computing.

In Section 6, we describe four sample applicationsthat we have built in order to illustrate this opportu-nity. Building those applications had an interesting con-sequence on the NetServ architecture. Some of our ap-plication scenarios require that a user service modulerunning on NetServ not only perform packet processing,but also provide traditional end-to-end network services.The end result is that a NetServ application module canbe apacket processing application module that sits inthe data path, aserver application module that uses theTCP/IP stack in the traditional way, or a combination ofboth. We consider this trait yet another way we conformto the active networking vision, or possibly even extendit, as we aim to eliminate the distinction between routersand servers. User code simply runs everywhere.

3 Architecture Overview

Figure 1 depicts the architecture of a NetServ node. Theservice modules, represented as ovals, run in a virtual ex-ecution environment. The virtual execution environmentprovides a basic API as a building block layer, consistingof preloaded modules.

We took heed of Calvert’s reflection on active net-working in 2006 [10]. He noted that “late binding”–i.e.,leaving things unspecified–did not help the case. Wepicked the JVM as the execution environment for ser-vice modules to achieve service mobility and a platform-independent programming interface. Java is the naturalchoice today. No other technology matches its maturity,features, track record of large-scale deployments, exten-sive libraries and wide-spread use among developers.

The execution environments communicate with thepacket transport layer. The packet transport layer pro-vides the TCP/IP stack for server application modules.For packet processing application modules, the packettransport layer provides a mechanism to filter IP packetsand route them to appropriate modules.

3

Service Container for

Provider 1

1) User Request

3) Install Module

2) NSIS Signal

Provider 1

End Users

NetServ Router

Ne

tSe

rvC

on

tro

ller

1) User Request3) Install Module

2) NSIS Signal

Provider 2End Users

Ne

tSe

rv

Service Container for

Provider 2

Figure 2: Deploying modules on a NetServ node.

The NetServ controller downloads and installs an ap-plication module when it receives a signaling message.A user sends a signaling message towards a destinationof his interest. Every NetServ node on-path interceptsthe message, takes an appropriate action, and forwards itto the next hop.

Figure 2 places a NetServ node in a broader contextof an end-to-end service. (1) End user requests are re-ceived by a content provider’s server, triggering signal-ing from the server. (2) As a signaling message travelstowards an end user, it passes through a mixture of reg-ular IP routers and NetServ-enabled routers between thecontent provider and the user. Regular IP routers simplyforward the message towards the destination. (3) Whenthe message passes through a NetServ router, however,it causes the NetServ router to download and install anapplication module from the content provider. The ex-act condition to trigger signaling and what the moduledoes once installed will depend on the application. Forexample, a content provider might send a signal to in-stall a web caching module when it detects web requestsabove a predefined threshold. The module can then actas a transparent web proxy for downstream users. Wewill describe four specific examples of this applicationscenario in Section 6.

4 Implementation

We implemented the NetServ architecture on Linux. Wereleased source code2 in conjunction with a NetServ tu-torial we gave at the 11th GENI Engineering Conference(GEC11). We will continue to release new versions ofour software and give NetServ tutorials at future GECs.

Figure 3 describes our Linux implementation. The ar-row at the bottom labeled “signaling packets” indicatesthe path a signaling packet takes. The packet is inter-cepted by the signaling daemons, which unpack the sig-naling packet and pass the contained message to the Net-Serv controller. The controller acts on the message byissuing commands to the appropriate service containers,

2http://www.cs.columbia.edu/irt/project/netserv

NetServ

NSLP

GIST

GIST packet interception

UNIX socket

NetServ

Controller

Linux kernel

Tra

nsp

ort

la

ye

r

Service Container

Service Container

Service Container

Java OSGi

Java OSGi

Java OSGi

Server

application

modules

Con

tain

er

co

ntr

ol co

mm

and

s

Client-

Server

data packets

Forwarded

data

packets

Signaling

packets

iptables

command

Netfilter NFQUEUE #2 NFQUEUE #1

Ne

tSe

rv C

on

tro

l

Me

ssa

ge

s

Packet

processing

application modules

NS

IS s

igna

ling

dae

mo

ns

Figure 3: Linux implementation.

to install or remove a module, for example.Service containers are user space processes with em-

bedded JVMs. Each container holds one or more ap-plication modules created by a single user. The JVMsrun the OSGi module framework [4]. Thus, the appli-cation modules installed in service containers are OSGi-compliant JAR files known asbundles. The OSGi frame-work allows bundles to be loaded and unloaded while theJVM is running. This enables a NetServ container to in-stall and remove application modules at runtime.

The two circles on the upper-right service containerare server application modules, sending and receivingnormal TCP/IP packets labeled “client-server data pack-ets.” The two circles on the lower-left container arepacket processing application modules. The arrow la-beled “forwarded data packets” shows how an incomingpacket is routed from the kernel to a user space containerprocess. The packet then visits the two packet processingmodules in turn before being pushed back to the kernel.

We provide a detailed description of each part of Fig-ure 3 in the following subsections.

4.1 Signaling

We use on-path signaling as the deployment mechanism.Signaling messages carry commands to install and re-move modules, and to retrieve information–like routerIP address and capabilities–about NetServ routers on-path. We use the Next Steps in Signaling (NSIS) protocolsuite [17], an IETF standard for signaling. NSIS consistsof two layers: a genericsignaling transport layer and anapplication-specificsignaling application layer.

The two boxes in Figure 3, labeled “GIST” and “Net-Serv NSLP,” represent the two NSIS signaling layersused in a NetServ node. GIST, the General Internet Sig-nalling Transport protocol [27], is an implementation ofthe transport layer of NSIS. GIST is a soft state protocolthat discovers other GIST nodes and maintains associa-tions with them in the background, transparently provid-ing this service to the upper signaling application layer.

4

SETUP NetServ.apps.NetMonitor_1.0.0 NETSERV/0.1dependencies:

filter-port:5060filter-proto:udpnotification:

properties:visualizer_ip=1.2.3.4,visualizer_port=5678ttl:3600

user:janedoeurl:http://content-provider.com/modules/netmonitor.jarsignature:4Z+HvDEm2WhHJrg9UKovwmbsA71FTMFykVa0Y\xGclG8o=

<blank line>

Figure 4: ASETUP message.

NetServ‐aware

Application

Setup/Remove Request

Setup/Remove Response

Probe Request

Probe Response

NetServ

Router

Probe Request

Probe Response

NetServ

Router

Probe Request

NetServ

RouterEnd User

Setup/Remove ResponseSetup/Remove Response

Setup/Remove RequestSetup/Remove Request

Probe stack

Probe Response

Last node detection

Last node detection

NOT PRESENT

DOWNLOADING

ACTIVE

Probe stack

DOWNLOADING

ACTIVE

Probe stack

ACTIVE

Figure 5: Request and response exchange.

NetServ NSLP is the NetServ-specific applicationlayer of NSIS. It contains the signaling logic of NetServand relays messages to the NetServ controller. The cur-rent implementation of the NetServ signaling daemons isbased on NSIS-ka [3].

There are two kinds of NetServ signaling messages:requests and responses. Typically, a content provider’sserver sends a request toward an end user. The last on-path NetServ node generates a response to the server.

There are three types of NetServ requests:SETUP,REMOVE, and PROBE. The SETUP message is used toinstall a module on the NetServ nodes on-path. TheREMOVE message uninstalls it. ThePROBE message isused to obtain the NetServ nodes’ statuses, capabilities,and policies. Figure 4 shows an example of aSETUPmes-sage. It requests that an application module called Net-Monitor be downloaded from the given URL, installedin the packet path to process UDP packets for port 5060,and automatically removed after 3600 seconds. Ourcompanion technical report [23] contains a listing of thecurrently supported header fields in request messages.

Figure 5 shows how response messages are gener-ated at the last node and returned along the signalingpath back to the requester. The responses toSETUP andREMOVE requests simply acknowledge the receipt of themessages. A response to aPROBE request carries theprobed information in the response message. As the mes-sage transits NetServ nodes alone the return path, eachnode adds its own information to the response stack inthe message. The full response stack is then deliveredback to the requester. Figure 5 shows a response to amodule status probe being collected in a response stack.

4.2 NetServ Controller

The NetServ controller coordinates three componentswithin a NetServ node: NSIS daemons, service contain-ers, and the forwarding plane. It receives control com-mands from the NSIS daemons, which may trigger theinstallation or removal of application modules within ser-vice containers, and in some cases filtering rules in theforwarding plane.

The controller is responsible for setting up and tear-ing down service containers. The current prototype pre-forks a fixed number of containers. Each container isassociated with a specific user account. The controllermaintains a persistent TCP connection to each container,through which it tells the container to install or removeapplication modules.

4.3 Forwarding Plane

The forwarding plane is the packet transport layer in aNetServ node, which is typically an OS kernel in anend host or forwarding plane in a router. The architec-ture requires only certain minimal abstractions from theforwarding plane. Packet processing modules require ahook in user space and a method to filter and direct pack-ets to the appropriate hook. Server modules require aTCP/IP stack, or its future Internet equivalent. The for-warding plane must also provide a method to interceptsignaling messages and pass them to the GIST daemonin user space.

Currently we use Netfilter, the packet filtering frame-work in the Linux kernel, as the packet processing hook.When the controller receives aSETUP message contain-ing filter-* headers, it verifies that the destination iswithin the allowed range specified in the configurationfile. It then invokes aniptables command to install afiltering rule to deliver matching packets to the appropri-ate user space service container using Netfilter queues.

The Linux TCP/IP stack allows server modules to lis-ten on a port. The allowable ports are specified in theconfiguration file.

NetServ can use forwarding planes other than theLinux kernel. We have prototyped alternate forwardingplanes for NetServ using the Click router [22] and theOpenFlow [24] switch. We are also porting NetServ toJuniper routers using the JUNOS SDK [21]. The Clickimplementation and our approach for Juniper port are de-scribed in [23]. The OpenFlow extension for NetServ isdescribed in detail in Section 10.

4.4 Service Container and Modules

Service containers are user space processes that run mod-ules written in Java. Figure 6 shows our current imple-mentation. The service container process can optionally

5

Building block layer

Library modulesSystem modules Wrappers for native

functions

Packet

processing

application

module 1

Server

application

module 1

Command

from

NetServ

controller

Client-server

data packets

dispatcher.addPktProcessor(this);

Packet dispatcher�

Servlet API XugglerXML-RPC

� �

Packet

processing

application

module 2

NFQUEUE

Linux kernel

Building block layer

OS-level VM ( lxc Linux Container )

JVM + OSGi

Forwarded

data

packets

libnetfilter_queue

Figure 6: User space service container process.

be run within lxc [2], an OS-level virtualization technol-ogy included in the Linux kernel. We defer the discus-sion of lxc to Section 5.

When the container process starts, the container cre-ates a JVM, and calls an entry point Java function thatlaunches the OSGi framework.

The service container starts with a number of prein-stalled modules which provide essential services to theapplication modules. We refer to the collection of prein-stalled modules as the building block layer. The build-ing block layer typically includes system modules, li-brary modules, and wrappers for native functions. Sys-tem modules provide essential system-level services likepacket dispatching. Library modules are commonly usedlibraries like Servlet engine or XML-RPC. The buildingblock layer can also provide wrappers for native codewhen no pure Java alternative is available. For example,our ActiveCDN application described in Section 6.1 re-quires Xuggler [7], a Java wrapper for the FFmpeg videoprocessing library.

The set of modules that make up the building blocklayer is determined by the node operator. An applica-tion module with a specific set of dependencies can dis-cover the presence of the required modules on path us-ingPROBE signaling messages, and then include a depen-dency header in theSETUP message to ensure the appli-cation is only installed where the modules are available.We plan to develop a recommendation for the composi-tion of the building block layer.

The container process useslibnetfilter queue toretrieve a packet, which is then passed to the packet dis-patcher, a Java module running inside the OSGi frame-work. The packet dispatcher then passes the packet toeach packet processing application module in turn. Thispath is depicted by the arrow labeledforwarded datapackets in Figure 6. We avoid copying a packet whenit is passed from C code to Java code. We construct a di-rect byte buffer object that points to the memory addresscontaining the packet. The reference to this object is thenpassed to the Java code.

5 Security

In this section, we consider security risks that arise fromthe fact that multiple service containers belonging to dif-ferent users coexist in a NetServ node.

Resource control and Isolation: A single user shouldnot be allowed to consume more than his fair share ofthe system resources such as CPU, memory, disk spaceor network bandwidth. Furthermore, a user’s executionenvironment must be isolated from the others’, in orderto prevent intentional or accidental tampering.

Authentication and Authorization: A user’s requestto install or remove a module must be verified to ensurethat it is from a valid user. Installed modules are subjectto further restrictions. In particular, a packet processingmodule must not be allowed to inspect or modify packetsbelonging to other users.

5.1 Resource Control and Isolation

We have multiple layers of resource control and isola-tion in the service container. First, because the containeris a user space process, we can use the standard Linuxresource control and isolation mechanisms, such as nicevalue,setrlimit(), disk quota, andchroot.

We control the application modules further using Java2 Security [14]. It provides fine-grained controls on filesystem and network access. We use them to confine themodules’ file system access to a directory, and limit theports on which the modules can listen. Java 2 Securityalso allows us to prevent the modules from loading nativelibraries and executing external commands.

In addition, the container can optionally be placedwithin lxc3, the operating system-level virtualizationtechnology in Linux. Lxc provides further resource con-trol beyond that which is available with standard oper-ating system mechanisms. We can limit the percentageof CPU cycles available to the container relative to otherprocesses in the host system. Lxc provides resource iso-lation using separate namespaces for system resources.The network namespace is particularly useful for Net-Serv containers. A container running in lxc can be as-signed its own network device and IP address. This al-lows, for example, two application modules running inseparate containers to listen on “*:80” without conflict.At at the time of this writing, a service container runninginside lxc does not support packet processing modules.

NetServ modules also benefit from Java’s languagelevel security. For example, the memory buffer contain-ing a packet is wrapped with aDirectByteBuffer ob-ject and passed to a module. TheDirectByteBuffer

3lxc is also referred to as “Linux containers” which should not beconfused with NetServ service containers. References to containersthroughout this paper mean NetServ service containers.

6

is backed by memory allocated in C. However, it is notpossible to corrupt the memory by going out-of-boundssince such access is not possible in Java.

5.2 Authentication and Authorization

SETUP request messages are authenticated using the sig-nature header included in each message. Currently, theNetServ node is preconfigured with the public key ofeach user. When a user sends aSETUP message, it signsthe message with a private key, this signature is verifiedby the controller prior to module installation. The cur-rent prototype signs only the signaling message–whichincludes the URL of the module to be downloaded. Thenext prototype will implement signing of the module it-self. As future work, we plan to develop a third party au-thentication scheme which will eliminate the need to pre-configure a user’s public key. A clearinghouse will man-age user credentials and settle payments between contentproviders and ISPs.

Authorization is required if theSETUP message for anapplication module includes a request to install a packetfilter in the forwarding plane. If the module wants tofilter packets destined for a specific IP address, it must beproved that the module has a right to do so. The currentprototype preconfigures the node with a list of IP prefixesthat the user is authorized to filter.

Our requirement to verify the ownership of a networkprefix is similar to the problem being solved in the IETFSecure Inter-Domain Routing working group [6]. Theworking group proposes a solution based on Public KeyInfrastructure (PKI), calledRPKI. RPKI can be used toverify whether a certain autonomous system is allowedto advertise a given network prefix. We plan on usingthat infrastructure once it becomes widely available.

We also plan to support a less secure, but simpler veri-fication mechanism that does not rely on PKI. It is basedon a reverse routability check. To prove the ownershipof an IP address, the user generates a one-time passwordand stores the password on the server with that IP ad-dress. The password is then sent in theSETUP message.Before installing the module, the NetServ controller con-nects to the server at the IP address, and compares thepassword included in theSETUP message with the onestored on the server. A match shows that the user of themodule has access to the server. The NetServ node ac-cepts this as proof of IP ownership.

6 NetServ Applications

We advocate NetServ as a platform that enables contentproviders and ISPs to enter into a new economic alliance.In this section, we present four example applications–ActiveCDN, KeepAlive Responder, Media Relay, and

End user

NetServ

router

NetServ

router

Regular

router

Regular

router

Content

provider

(1) User requests video: http://content-provider.com/?file=foo

(2) Content provider sends video file

(3) Content provider sends on-path signal to deploy ActiveCDN module

NN

(4) NetServ routers download the module

(3) Content provider sends on-path signal to deploy ActiveCDN module

(6) NetServ routers with ActiveCDN reply to probe

(7) Another user requests http://content- provider.com/?file=foo

(8) Content provider finds nearby ActiveCDN node, sends redirect message

(9) User requests http://netserv1.service-provider.com/?file=foo

(10) ActiveCDN downloads the video, simultaneously serving and caching it

(5) Content provider probes for installed ActiveCDN modules

(11) ActiveCDN can also process content

Figure 7: How ActiveCDN works.

Overload Control–which demonstrate economic benefitfor both parties.

6.1 ActiveCDN

We developed ActiveCDN, a NetServ application mod-ule that implements CDN functionality on NetServ-enabled edge routers. ActiveCDN brings content andservices closer to end users than traditional CDNs. AnActiveCDN module is created by a content provider, whohas the full control of the placement of the module. Themodule can be redeployed to different parts of the In-ternet as needed. This is in stark contrast to the largelypreconfigured topology of existing CDNs.

The content provider also controls the functionality ofthe module. The module can perform custom processingspecific to the content provider, like inserting advertise-ments into video streams.

Figure 7 offers an example of how ActiveCDN works.When an end user requests video content from a contentprovider’s server, the server checks its database to deter-mine if there is a NetServ node running ActiveCDN inthe vicinity of the user. If there is no ActiveCDN nodein the vicinity, the server serves the video to the user,and at the same time, sends aSETUP message to deployan ActiveCDN module on an edge router close to thatuser. This triggers each NetServ node on-path, generallyat the network edge, to download and install the mod-ule. Following theSETUP message the server sends aPROBE message to retrieve the IP addresses of the Net-Serv nodes that have successfully installed ActiveCDN.This information is used to update the database of de-ployed ActiveCDN locations. When a subsequent re-quest comes from the same region as the first, the contentprovider’s server redirects the request to the closest Ac-tiveCDN node, most likely one of the nodes previously

7

Figure 8: Operation of KeepAlive Responder.

installed. The module responds to the request by down-loading the video, simultaneously serving and caching it.The content provider’s server can send aREMOVE mes-sage to uninstall the module, otherwise the module willbe removed automatically after a while. The process re-peats when new requests are made from the same region.

6.2 KeepAlive Responder

The ubiquitous presence of Network Address Transla-tors (NATs) poses a challenge to communication servicesbased on Session Initiation Protocol (SIP) [25]. After aSIP User Agent (UA) behind a NAT box registers its IPaddress with a SIP server, the UA needs to make sure thatthe state in the NAT box remains active for the durationof the registration. Failure to keep the state active wouldrender the UA unreachable. The most common mecha-nism used by UAs to keep NAT bindings open is to sendperiodic keep-alive messages to the SIP server.

The timeout for UDP bindings appears to be rathershort in most NAT implementations. SIP UAs typicallysend keep-alive messages every 15 seconds [9] to remainreachable from the SIP server.

While the size of a keep-alive message is relativelysmall–about 300 bytes when SIP messages are used forthis purpose, which is often the case–large deploymentswith hundreds of thousand or even millions of UAs arenot unusual. Millions of UAs sending a keep-alive ev-ery 15 seconds represent a significant consumption ofnetwork and server resources. A surprising fact is thatthe keep-alive traffic can be a bottleneck in scaling a SIPserver to a large number of users [9].

Figure 8 shows how NetServ could help offload NATkeep-alive traffic from the infrastructure of Internet Tele-phony Service Providers (ITSPs). Without the NetServKeepAlive Responder, the SIP UA behind a NAT sends akeep-alive request to the SIP server every 15 seconds andthe SIP server sends a response back. When an NSIS-enabled SIP server starts receiving NAT keep-alive traf-fic from a SIP UA, it initiates NSIS signaling in order tofind a NetServ router along the network path to the SIPUA. If a NetServ router is found, the router downloadsand installs the KeepAlive module provided by the ITSP.

After the module has been successfully installed, itstarts inspecting SIP traffic going through the router to-wards the SIP server. If the module finds a NAT keep-alive request, it generates a reply on behalf of the SIP

Figure 9: Operation of NetServ Media Relay.

server, sends it to the SIP UA, and discards the originalrequest. Thus, if there is a NetServ router close to the SIPUA, the NAT keep-alive traffic never reaches the networkor the servers of the ITSP; the keep-alive traffic remainslocal in the network close to the SIP UA.

The KeepAlive Responder spoofs the IP address of theSIP server in response packets sent to the UA. IP addressspoofing is not an issue here because the NetServ routeris on-path between the spoofed IP address and the UA.

6.3 Media Relay

ITSPs may need to deploy media relay servers to facili-tate the packet exchange between NATed UAs. However,this approach has several drawbacks, including increaseddelay, additional hardware and network costs, and man-agement overhead.

Figure 9 shows how NetServ helps to offload the me-dia relay functionality from an ITSP’s infrastructure. Thedirect exchange of media packets between the two UAsin the picture is not possible. Without NetServ the ITSPwould need to provide a managed media relay server.When a NetServ router is available close to one of theUAs, the SIP server can deploy the Media Relay moduleat the NetServ node.

When a UA registers its network address with the SIPserver, the SIP server sends an NSIS signaling messagetowards the UA, instructing the NetServ routers along thepath to download and install the Media Relay module.The SIP server then selects a NetServ node close to theUA, instead of a managed server, to relay calls to andfrom that UA.

6.4 Overload Control

SIP servers are vulnerable to overload due to the lack ofcongestion control in UDP. The IETF has developed aframework for overload control in SIP servers that canbe used to mitigate the problem [15]. Figure 10 illus-trates the scenario. The SIP server under load, referredto as the Receiving Entity (RE), periodically monitors its

8

Figure 10: NetServ as SIP overload protection.

load. The information about the load is then communi-cated to the Sending Entity (SE), which is the upstreamSIP server along the path of incoming SIP traffic. Basedon the feedback from the RE, the SE then either rejectsor drops a portion of incoming SIP traffic.

We implemented a simple SIP overload control frame-work in NetServ. When the load on the SIP server ex-ceeds a preconfigured threshold, the SIP server startssending NSIS signals towards the UAs in an attempt todiscover a NetServ node along the path and install the SENetServ module on the node. Once the module is suc-cessfully installed, it intercepts all SIP messages goingto the SIP server. Based on the periodic feedback aboutthe current volume of traffic seen by the SIP server, themodule adjusts the amount of traffic it lets through in realtime. The excess portion of incoming traffic is rejectedwith “503 Service Unavailable” SIP responses.

Without NetServ, an ITSP’s options in implementingoverload control are limited. The ITSP can put both theSE and the RE in the same network. Such configurationonly allows hop-by-hop overload control, in which caseexcessive traffic enters the ITSP’s network before it isdropped by the SE. Since all incoming traffic usually ar-rives over the same network connection, using differentcontrol algorithms or configurations for different sourcesof traffic becomes difficult.

With NetServ, the ability to run an SE implementationat the edge of the network makes it possible to experi-ment with control algorithms and configurations for dif-ferent sources of traffic. Being able to install and removea NetServ SE module dynamically makes it easy for anITSP to change the traffic control algorithm. Since theNetServ SE module is installed outside the ITSP’s net-work, excess traffic is rejected before it enters the ITSP’snetwork, protecting not only the SIP server, but also thenetwork connection.

7 NetServ on GENI

GENI [1] is a federation of many existing networktestbeds under a common management framework.GENI is comprised of a diverse set of platform resources,which are shared among many experimenters.

NetServ is becoming a resident tool of the GENI in-frastructure. NetServ’s common execution environmentcan accelerate development, deployment and testing of

experiments. NetServ’s Java-based API makes GENI agentler environment for educational use.

Figure 11: ActiveCDN demo.

We demonstrated NetServ, using two of our sampleapplications, at the plenary session of the 9th GENI En-gineering Conference (GEC9).4 Figure 11 shows thescreenshots from the ActiveCDN demo. The ActiveCDNmodule performed custom processing on cached content,watermarking a video stream with local weather infor-mation of Salt Lake City where the module was installed.We hold NetServ tutorials at GECs since GEC11.

8 Discussion

Reverse data pathThe descriptions of the NetServ applications in Sec-

tion 6 assumed that the reverse data path is the same asthe forward path. On the Internet today, however, this isoften not the case due to policy routing.

For ActiveCDN and Media Relay, this is not an is-sue. The modules only need to be deployedcloser tothe users, not necessarily on the forward data path. Themodule will still be effective if the network path from theuser to the NetServ node has a lower cost than the pathfrom the user to the server.

For KeepAlive Responder and Overload Control, themodule must be on-path to carry out its function. How-ever, this is not a serious problem in general. First, Net-Serv routers are located at the network edge. It is un-likely that the reverse path will go through a differentedge router. Even in the unlikely case that a module isinstalled on a NetServ router which is not on the reversepath, if we assume a dense population of users, it is likelythat the module will serve some users, albeit not the oneswho triggered the installation in the first place. If a mod-ule is installed at a place where there is no user to serve,it will time-out quickly.

If a reverse on-path installation is indeed required,there are two ways to handle it. First, the client-sidesoftware can initiate the signaling instead of the server.But this requires modification of the client-side software.Second, the server can use round-trip signaling. WeimplementedTRIGGER signaling message in NetServNSLP. The server encapsulates aSETUP or PROBE in a

4The 14-minute demo video is athttp://vimeo.com/16474575.

9

TRIGGER, and sends it towards the end user. The lastNetServ router on-path creates a new upstream signalingflow back to the server. This approach, however,assumes that the last NetServ node is on both forwardand reverse path, and increases the signaling latency.

Off-path signalingIn addition to on-path signaling, we envision that

certain cases of off-path signaling would be useful forsome NetServ applications. There is a proposal toextend NSIS to include epidemic signaling [13]. Theproposed extension will enable three additional modesof signal dissemination: (1) signaling around the sender(bubble), (2) signaling around the receiver (balloon),and (3) signaling around the path between the senderand receiver (hose). The bubble and balloon modeswill be useful for NetServ module deployment withinan enterprise environment. The hose mode will beuseful for the scenarios where NetServ nodes are notexactly on-path, but a couple of hops away. This modecan mitigate the aforementioned concerns about thedivergent reverse path.

9 Evaluation

In this section, we provide evaluation results for ourLinux implementation. In particular, we measure theoverhead introduced by placing packet processing mod-ules in user space JVM. First, we measure the MaximumLoss Free Forwarding Rate (MLFFR) of a NetServ nodewith a single service container. We show the overheadassociated with each layer in a NetServ node. Second,we perform a microbenchmark measurement to show thedelay in each layer. Lastly, we run 100 service containersin a NetServ node and measure the throughput and mem-ory consumption. Our results suggest that while there iscertainly significant overhead, it is not prohibitive.

9.1 Setup

Our setup consists of three nodes connected in sequence:sender, router, and receiver. The sender generates UDPpackets addressed to the receiver and sends them to therouter, which forwards them to the receiver.

All three machines were equipped with a 3.0GHz In-tel Dual Core Xeon CPU, 4 x 4GB DDR2 RAM, and anIntel Pro/1000 Quad Port Gigabit Ethernet adapter con-nected on PCIe x 4 bus which provided 8Gb/s maximumbandwidth. All links ran at 1Gb/s. We disabled Ethernetflow control which allowed us to saturate the connection.

For the sender and receiver, we used a kernel modeClick router version 1.7.9 running on a patched 2.6.24.7

Linux kernel. The Ethernet driver was Intel’s igb ver-sion 1.2.44.3 with Click’s polling patch applied. For therouter, we used Ubuntu Linux 10.04 LTS Server Edition64bit version, with kernel version 2.6.32-27-server, andthe igb Ethernet driver upgraded to 2.4.12 which sup-ports the New API (NAPI) in the Linux kernel.

9.2 Results

We measured the MLFFRs of six different configurationsof the router. Each configuration adds a layer to theprevious one, adding more system components throughwhich a packet must travel.

Configuration 1 is the plain Linux router we describedabove. This represents the maximum attainable rate ofour hardware using a Linux kernel as a router.

Configuration 2 adds Netfilter packet filtering kernelmodules to configuration 1. This represents a more re-alistic router setting than configuration 1 since a typicalrouter is likely to have a packet filtering capability. Thisis the base line that we compare with the rest of the con-figurations that run NetServ.

Configuration 3 adds the NetServ container, but withits Java layer removed. The packet path includes the ker-nel mode to user mode switch, but does not include aJava execution environment.

The packet path for configuration 4 includes the fullNetServ container, which includes a Java execution en-vironment. However, no application module is added tothe NetServ container.

Configuration 5 adds NetMonitor, a simple NetServapplication module with minimal functionality. It main-tains a count of received packets keyed by a 3-tuple:source IP address, destination IP address, and TTL.

Configuration 6 replaces NetMonitor with the Keep-Alive module described in Section 6.2. KeepAlive exam-ines incoming packets for SIPNOTIFY requests with thekeep-aliveEvent header and swaps the source and des-tination IP addresses. For the measurement, we disabledthe address swapping so that packets can be forward tothe receiver. This test represents a NetServ router run-ning a real-world application module.

Figure 12(a) shows the MLFFRs of five differentrouter configurations. The MLFFR of configuration 1was 786kpps, configuration 2 was 641kpps, configura-tion 3 was 365kpps, configuration 4 was 188kpps, andconfiguration 5 was 188kpps.

The large performance drop between configurations2 and 3 can be explained by the overhead added by akernel-user transition. The difference between configu-rations 3 and 4 shows the overhead of Java execution.There is almost no difference between configurations 4and 5 because the overhead of the NetMonitor module isnegligible.

10

0

100

200

300

400

500

600

700

800

900

0 100 200 300 400 500 600 700 800 900

For

war

ding

rat

e [k

pps]

Input rate [kpps]

Conf 1: Plain LinuxConf 2: Linux with packet filter

Conf 3: NetServ Container with Java removedConf 4: NetServ Container with no moduleConf 5: NetServ Container with NetMonitor

(a) Configuration 1 to 5 with 64B packets.

0

100

200

300

400

500

0 100 200 300 400 500

For

war

ding

rat

e [k

pps]

Input rate [kpps]

Conf 1: Plain LinuxConf 2: Linux with packet filter

Conf 3: NetServ Container with Java removedConf 4: NetServ Container with no moduleConf 5: NetServ Container with NetMonitorConf 6: NetServ Container with KeepAlive

(b) Configuration 1 to 6 with 340B packets.

Figure 12: Forwarding rates of the router with different configurations.

The dips in the curves for configurations 3 through5 are the result of switching between the interrupt andpolling modes of the NAPI network driver in Linux. Seeour technical report [23] for details.

Figure 12(b) shows the repeated measurement but with340B packets, in order to compare them with configura-tion 6. For configuration 6, we created a custom Clickelement to send SIPNOTIFY requests, which are UDPpackets. The size of the packet was 340B, and we usedthe same SIP packets for configurations 1 through 5.

The MLFFR of configuration 1 was 343kpps,configuration 2 was 343kpps, configuration 3 was213kpps, configuration 4 was 117kpps, configuration 5was 111kpps, and configuration 6 was 71kpps.

There was no difference between the performance ofconfigurations 1 and 2. The difference between configu-rations 2 and 3 is due to the kernel-user transition. Thedifference seen between configurations 3 and 4 is dueto Java execution overhead. Both of these were previ-ously seen above. Again, there is almost no differencebetween configurations 4 and 5. The difference betweenconfigurations 5 and 6 shows the overhead of KeepAlivebeyond NetMonitor. There is a meaningful difference be-tween the modules because the KeepAlive module mustdo deep packet inspection to find SIPNOTIFY messages,and further, we made no effort to optimize the matchingalgorithm.

Figure 13 shows our microbenchmark result. It com-pares delays as a packet travels through each layer ina NetServ node. The first bar shows only the delay inLinux kernel (configuration 1 in our MLFFR graphs),the second bar adds the delay from the kernel packetfilter (configuration 2), and the third bar shows the de-lays in all layers up to the KeepAlive module (configura-tion 6). The second bar represents the delay experiencedby packets transiting a NetServ node without being pro-cessed by a module. We note that the additional overhead

0

10

20

30

40

50

60

70

80

90

100

Plain Linuxforwarding

Unprocessed packetsin NetServ

Processed packetsin NetServ

Del

ay [m

icro

seco

nd]

Linux kernelPacket filter

NetServ container - C layerNetServ container - Java layer

NetServ KeepAlive module

Figure 13: Microbenchmark.

compared to the first bar, plain Linux forwarding, is verysmall. The third bar, representing the full packet pro-cessing overhead, shows a significant amount of delay,as expected.

The overhead is certainly significant. Packets pro-cessed by the KeepAlive module achieve only 20% ofthroughput and incur 92microsecond delay, compared tounprocessed packets. However, we make a few obser-vations in our defense. The KeepAlive throughput of71kpps is on par with the average traffic experienced bya typical edge router [8]. Our tests were performed onmodest hardware, and more importantly, a packet pro-cessing module would only be expected to handle a smallfraction of the total traffic. Our Linux implementation,thus, is quite usable in low traffic environments. And, aswe argued before, the OpenFlow extension in Section 10provides a solution for high traffic environments.

Lastly, we observe the behavior of a NetServ noderunning many containers. Incoming traffic is equallydistributed to each container. Figure 14 shows the to-tal throughput and memory consumption as we increase

11

20

40

60

80

100

120

140

160

20 30 40 50 60 70 80 90 100 2

4

6

8

10

12

14

16T

hrou

ghpu

t [kp

ps]

Mem

ory

usag

e [G

B]

Number of NetServ containers

Total peak throughputTotal throughput at saturation

Memory usage

Figure 14: NetServ node with many containers.

the number of containers in a NetServ node. The to-tal throughput gradually decreases, indicating the over-head of running many containers. The line labeled “peakthroughput” is the maximum throughput reported just be-fore the NetServ node started experiencing packet loss asthe input rate increased. The line labeled “throughput atsaturation” is the throughput when it plateaued againstthe increasing input rate. The overhead of running manycontainers, again, exacerbates the difference. Mem-ory consumption is proportional to the number of con-tainers. Each container consumes about 110MB whenthe KeepAlive module is busy processing packets at thepeak throughput. Each container contains a JVM, OSGiframework, and a collection of building block modules.Figure 14 shows that a NetServ node scales reasonablywell as we increase the number of containers in it.

10 OpenFlow Extension

The Linux-based implementation that we described inthe previous section has a limitation in terms of perfor-mance. Multiple layers present in our execution envi-ronment introduce significant overhead when a packet issubject to DPI, as our evaluation will show in Section 9.A more serious limitation is the fact that the scalability islimited to a single Linux box, even when no packet pro-cessing is performed. In general, a general-purpose PCcannot match the forwarding performance of a dedicatedrouter. This makes our Linux implementation unsuitablefor high traffic environment.

We can address this limitation by offloading the for-warding plane onto a physically separate hardware ele-ment, capable of forwarding packets at line rate. Thehardware device must also provide dynamically instal-lable packet filtering hooks, so that the packets that needto be processed by NetServ modules can be routed appro-priately to one or more NetServ nodes which are attachedto the hardware device.

OpenFlow Switch

Flow Table

MAC

src

MAC

dst

IP

src

IP

dst

TCP

sport

TCP

dportAction

**10.0.0.1***Output

port 1

OpenFlow

Controller

1st pktsPKT

(4) (1)

(3)

(2)

subseq. pkts

dst: 10.0.0.1

1 pkts

dst: 10.0.0.1PKT

(5)10.0.0.1

PKTPKTPKT

Figure 15: How OpenFlow works.

The OpenFlow programmable switch architecture pro-vides exactly the capabilities that we need. In this sec-tion, we briefly explain the OpenFlow architecture, anddescribe our prototype implementation of the OpenFlowextension of NetServ. At the time of this writing, the pro-totype is at an early stage where we have proved that ourapproach works. We expect to have a beta release by thetime this paper is published.

10.1 OpenFlow Overview

An OpenFlow switch [24] is an Ethernet switch with itsinternal flow table exposed via a standardized interfaceto add and remove flow entries. The OpenFlow Con-troller (OFC), typically a software program running ona remote host, communicates with the switch over a se-cure channel using the standard OpenFlow Protocol. Anentry in the flow table defines a mapping between a setof header fields–MAC/IP addresses and port numbers,for example–and one or more associated actions, suchas dropping a packet, forwarding it to a particular porton the switch, or even simple modifications of headerfields. When a packet arrives at an OpenFlow switch,the switch looks up the flow table. If an entry match-ing the packet header is found, the corresponding actionsare performed. If no entry matches the packet header,the packet is sent to the remote OFC, which will decidewhat to do with the packet, and also insert an entry intothe switch’s flow table so that subsequent packets of thesame flow will have a matching entry.

Figure 15 illustrates this process. (1) A packet des-tined for 10.0.0.1 arrives at an OpenFlow switch, whichcontains no matching entry in its flow table. (2) APacketIn message is sent to the OFC. The OFC, afterconsulting its routing table, determines the switch portto which the incoming packet should be output. (3) TheOFC sends aFlowMod command to the switch to adda flow table entry. (4) The command also include an in-struction to forward the incoming packet, which has beensitting in a queue waiting for the verdict from the OFC.The packet goes out to the destination. (5) All subse-quent packets destined for 10.0.0.1 match the new flowtable entry, so the packets are forwarded by the hardware

12

OpenFlow Switch

Flow Table

OpenFlow

Controller

OSGi

NetServ

Controller

NetServ Node

Packet

Processing

Module

Linux kernelSETUP

signaling

message Port 2

(1)

(4)PKT

1st packet

Dst: 10.0.0.1

Subsequent

packets

Dst: 10.0.0.1

Flow Table

MAC

src

MAC

dst

IP

src

IP

dst

UDP

sport

UDP

dportAction

2222*10.0.0.1*33:4411:22Output

port 2

2222*10.0.0.1*55:6633:44Output

port 1

Port 1

PKTPKTPKT

(5)(3)

(4)

(8)

(2)

(6)(9)

(7)

10.0.0.1

Figure 16: NetServ with OpenFlow extension.

switch at line rate without incurring the overhead of mak-ing a round trip to OFC.

10.2 NetServ on OpenFlow

NetServ on OpenFlow integrates the two technologiesin two ways. First, one or more NetServ nodes are at-tached to an OpenFlow switch. From NetServ’s point ofview, the OpenFlow switch provides a common forward-ing plane for multiple NetServ nodes. From OpenFlow’spoint of view, the NetServ nodes are external packet pro-cessing devices. (The OpenFlow paper [24] envisionssuch devices based on NetFPGA.)

Second, the OpenFlow Controller (OFC) is now im-plemented as a NetServ module. As such, the OFC canbe dynamically installed, updated, or moved to anothernode. Furthermore, there can be many OFCs, one peruser, or even one per application. In conjunction withFlowVisor–a special purpose OFC that acts as a transpar-ent proxy for a group of OFCs–NetServ-based OFCs willopen up interesting possibilities like an in-network ser-vice that reconfigures network topology as needed. Fur-ther exploration is planned as future work.

Figure 16 shows what happens when a packet destinedfor 10.0.0.1 is being processed by a NetServ node at-tached to an OpenFlow switch. First of all, the OFCrunning in NetServ sends a proactiveFlowMod com-mand to direct all NSIS signaling messages to the Net-Serv controller. (1) When the NetServ controller receivesa SETUP message–thanks to the proactive flow tableentry–it installs the requested packet processing applica-tion module. In addition, the NetServ controller tells theOFC that an application module has requested a packetfilter, and the OFC remembers the fact in preparation forincoming packets, but it does not add a flow table entryat this point. (2) A packet matching the filtering rule ofthe NetServ application arrives. There is no flow tableentry for it, so it goes to the OFC. The OFC in NetServ,before it begins its usual OFC work of consulting its rout-ing table, notices that the packet matches the application

filtering rule that the NetServ controller has told the OFCearlier. (3) The OFC translates the NetServ application’spacket filter into a flow table entry, and injects it into theOpenFlow switch so that the packet will be routed to theNetServ node. (4) The packet is delivered to the NetServnode, and then to the appropriate application module. (5)After processing the packet, the NetServ node sends itback to the switch. At that point, however, the packetmust go back to the OFC, so that it can find its switchingdestination. When the OFC sees the packet the secondtime around, it knows that it has been processed by aNetServ module (from the input switch port), so it goesstraight to the normal OFC work of consulting its routingtable. (6) The OFC injects another flow table entry in or-der to output the packet. (7) The packet goes to the des-tination. (8) Subsequent packets matching the NetServapplication’s filtering rule are now routed directly to theNetServ node without going to the OFC first (because ofthe flow table entry added in step (3).) (9) And whenthose subsequent packets come back from the NetServnode to the switch, there is the flow table entry added instep (6) to guide them to the correct output port.

Having a separate hardware-based forwarding planeeliminates the performance problem for the packets thatdo not go through a NetServ node. For the packetsthat need to go through NetServ, the OpenFlow exten-sion does not reduce individual packet processing time,but we can increase the throughput by attaching multi-ple NetServ nodes. Different flows can be assigned todifferent NetServ nodes, or depending on applications, asingle flow can use multiple NetServ nodes.

11 Related Work

Many earlier programmable routers focused on provid-ing modularity without sacrificing forwarding perfor-mance, which meant installing modules in kernel space.Router Plugins [11], Click [22], PromethOS [20], andPronto [18] followed this approach. As we noted before,NetServ runs modules in user space.

LARA++ [26] is similar to NetServ in that the mod-ules run in user space. However, LARA++ focuses moreon providing a flexible programming environment bysupporting multiple languages, XML-based filter spec-ification, and service composition. It does not employ asignaling protocol for dynamic code installation.

The Million Node GENI project [5], which is a partof GENI, provides a peer-to-peer hosting platform usingPython-based sandbox. An end user can contribute re-sources from his own computer in exchange for the useof the overlay network. The Million Node GENI focuseson end systems rather than in-network nodes.

Google Global Cache (GGC) [16] refers to a setof caching nodes located in ISPs’ networks, provid-

13

ing CDN-like functionality for Google’s content. Net-Serv can provide the same functionality to other contentproviders, as we have demonstrated with ActiveCDNmodule.

One of the goals of Content Centric Networking(CCN) [19] is to make the local storage capacity of nodesacross the Internet available to content providers. CCNproposes a replacement of IP by a new communicationprotocol, which addresses data rather than hosts. Net-Serv aims to realize the same goal using the existing IPinfrastructure. In addition, NetServ enables content pro-cessing in network nodes.

12 Conclusion

We call for a revival of active networks. We present Net-Serv, a fully integrated active network system that pro-vides all the necessary functionality to be deployable,addressing the core problems that prevented the practi-cal success of earlier approaches.

We present a hybrid approach to active networking,which combines the best qualities from the two extremeapproaches–integrated and discrete. We built a work-ing system that strikes the right balance between securityand performance by leveraging current technologies. Wesuggest an economic model based on NetServ betweencontent providers and ISPs. We built four applications toillustrate the model.

Acknowledgements

The authors would like to thank Roxana Geambasu,Volker Hilt, Zoran Despotovic, Wolfgang Kellerer,Mauro Femminella and Gianluca Reali for feedback anddiscussion. We also thank Eric Liu, Raynald Seydoux,Abhishek Srivastava and Nathan Miller for their assis-tance with implementation. This work was supported inpart by NSF grant NSF-CNS #0831912.

References

[1] GENI. http://www.geni.net/.

[2] lxc Linux Containers.http://lxc.sourceforge.net/.

[3] NSIS-ka. https://projekte.tm.uka.de/trac/NSIS/wiki/.

[4] OSGi Technology. http://www.osgi.org/About/Technology/.

[5] Seattle, Open Peer-to-Peer Computing.https://seattle.cs.

washington.edu/html/.

[6] Secure Inter-Domain Routing (sidr).http://datatracker.ietf.org/wg/sidr/charter/.

[7] Xuggler. http://www.xuggle.com/xuggler/.

[8] Princeton University Router Traffic Statistics.http://mrtg.net.princeton.edu/statistics/routers.html, 2010.

[9] S. A. Baset, J. Reich, J. Janak, P. Kasparek, V. Misra, D. Ruben-stein, and H. Schulzrinne. How Green is IP-Telephony? InTheACM SIGCOMM Workshop on Green Networking, 2010.

[10] K. Calvert. Reflections on Network Architecture: an Active Net-working Perspective.ACM SIGCOMM Computer Communica-tion Review, 36(2):27–30, 2006.

[11] D. Decasper, Z. Dittia, G. Parulkar, and B. Plattner. RouterPlugins: A Software Architecture for Next-Generation Routers.IEEE/ACM Transactions on Networking, 8(1):2–15, 2000.

[12] P. Faratin, D. Clark, P. Gilmore, S. Bauer, A. Berger, and W. Lehr.Complexity of Internet Interconnections: Technology, Incentivesand Implications for Policy. InTPRC, 2007.

[13] M. Femminella, R. Francescangeli, G. Reali, and H. Schulzrinne.Gossip-based Signaling Dissemination Extension for Next Stepsin Signaling. Technical Report, paper under review (available at:http://conan.diei.unipg.it/pub/nsis-gossip.pdf).

[14] L. Gong. Java 2 Platform Security Architecture.http://download.oracle.com/javase/1.4.2/docs/

guide/security/spec/security-spec.doc.html.

[15] V. Gurbani, V. Hilt, and H. Schulzrinne. SIP Overload Control.Internet-Draft draft-ietf-soc-overload-control-01, 2011.

[16] J. M. Guzman. Google Peering Policy. http:

//lacnic.net/documentos/lacnicxi/presentaciones/

Google-LACNIC-final-short.pdf, 2008.

[17] R. Hancock, G. Karagiannis, J. Loughney, and S. Van den Bosch.Next Steps in Signaling (NSIS): Framework. RFC 4080, 2005.

[18] G. Hjalmtysson. The Pronto Platform: a Flexible Toolkit for Pro-gramming Networks Using a Commodity Operating System. InOPENARCH, 2000.

[19] V. Jacobson, D. Smetters, J. Thornton, M. Plass, N. Briggs, andR. Braynard. Networking Named Content. InCoNeXT, 2009.

[20] R. Keller, L. Ruf, A. Guindehi, and B. Plattner. PromethOS: ADynamically Extensible Router Architecture Supporting ExplicitRouting. InIWAN, 2002.

[21] J. Kelly, W. Araujo, and K. Banerjee. Rapid Service Creationusing the JUNOS SDK.ACM SIGCOMM Computer Communi-cation Review, 40(1):56–60, 2010.

[22] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F. Kaashoek.The Click Modular Router.ACM Transactions on Computer Sys-tems, 18(3):263–297, 2000.

[23] J. W. Lee, R. Francescangeli, W. Song, J. Janak, S. Srinivasan,M. Kester, S. A. Baset, E. Liu, H. Schulzrinne, V. Hilt, Z. Despo-tovic, and W. Kellerer. NetServ Framework Design and Imple-mentation 1.0. Technical Report cucs-016-11, Columbia Univer-sity, May 2011.

[24] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar,L. Pe-terson, J. Rexford, S. Shenker, and J. Turner. OpenFlow: En-abling Innovation in Campus Networks.ACM SIGCOMM Com-puter Communication Review, 38(2):69–74, 2008.

[25] J. Rosenberg, H. Schulzrinne, G. Camarillo, A. Johnston, J. Pe-terson, R. Sparks, M. Handley, and E. Schooler. SIP: SessionInitiation Protocol. RFC 3261, 2002.

[26] S. Schmid, J. Finney, A. Scott, and W. Shepherd. Component-based Active Network Architecture. InISCC, 2001.

[27] H. Schulzrinne and R. Hancock. GIST: General Internet Sig-nalling Transport. RFC 5971, 2010.

[28] D. L. Tennenhouse and D. J. Wetherall. Towards an ActiveNet-work Architecture.ACM SIGCOMM Computer CommunicationReview, 26(2):5–17, 1996.

14