A Verification ServiceArchitecture for the Future Internet

Ahmet Can Babaoglu, Rudra DuttaDepartment of Computer Science, North Carolina State University

Email: acbabaog@ncsu.edu, rdutta@ncsu.edu

Abstract—In this paper, we propose a service architecturefor verification, a necessary component of a choice-based econ-omy architecture of the future Internet. Such a verificationarchitecture motivates and addresses the task of enabling usersto verify, or obtain third-party verification of, whether theservice components offered by various network service providerswere responsible for meeting or failing to meet overall serviceexpectations. To be useful, such an architecture must allow viablebusiness propositions for each of the principals, and providemeaningful results at reasonable cost and overhead. We articulatethe architectural decisions, requirements, roles and interfaces.We then describe a proof-of-concept prototype realized in NS-3. Finally, we make observations contrasting our results withprevious approaches and discuss the implementation challengesto realize this work in practice.

I. INTRODUCTION

Even as the current Internet enables a range of servicesand distributed applications that grow ever broader and morevareigated, several limitations of its architecture have becomeapparent as billions of humans and devices are connectedthrough it. This has caused a resurgence in interest in archi-tectural research to address evolutionary paths for the Internetin recent years. National research directive and funding or-ganizations have created programs encouraging such researchin various contexts, including the Future Internet Architecture(FIA) program of the US National Science Foundation [1].

One key challenge is the discrepancy between the mecha-nisms by which technology is deployed in the Internet and thebusiness models surrounding these processes. ChoiceNet [2],one of the projects being conducted under the FIA umbrella,and that our research group is contributing to, attempts toaddress this challenge. Briefly, the ChoiceNet project proposesthe introduction of architectural entities into the current ar-chitecture of the Internet to enable an “economy plane” thatallows the presentation of competing offerings for variousnetworking services, the formation of contracts for the variousservices that make up the entirety of a user’s network needs,and tracking the performance of each provider in meetingtheir parts of the contracts. In Section IV, we provide a moredetailed overview of ChoiceNet.

Contracts are meaningless without the ability to at leastverify, and possibly enforce, compliance. From the serviceprovider’s point of view, such verification and enforcementoften comes down to access or admission control of somesort. However, from individual or small customers’ point ofview, there is little or no granularity in the ability of obtaininformation about whether the contracted service was obtained,and if not, which one of the many service providers involved

in providing the customer’s experience are to blame. In somecases, a customer might be able to directly measure quantitiesof interest that provides the answers, but in many cases, theservice experienced by the customer is composed out of severalcomponent services provided by distinct providers. In theChoiceNet view, the customer makes separate contracts witheach such provider, and needs monitoring information for eachservice – some of which can only be obtained at points ofpresence inside the network cloud, where the customer doesnot have presence. This points to the need of a specific classof service providers: independent third-party verifiers. Suchverifiers would have distributed infrastructure and points ofpresence, but instead of selling transport service, would sellmonitoring service to customers.

In this paper, we articulate, within the ChoiceNet frame-work, the architectural roles, entities and interactions neces-sary to establish a verification plane for ChoiceNet, or anyChoiceNet-like architecture. Wherever possible, we call outsimilarities of existing tools or approaches with the entitieswe envision. We defer a broader discussion of related work toSection IV in order to be able to refer to these architecturalentities or concepts when doing so. We start by briefly dis-cussing the goals of architectural design, and the main pointsof ChoiceNet along with some of the related measurementtools and systems in the literature. In Section III, we describeour proposed architecture. Finally in Section V, we discussthe results of a prototype run on NS3 simulator as a proofof concept, and contrast it with existing methods; and discussimplementation challenges.

II. ARCHITECTURAL CONTEXT

A. Goals in Architectural Design

The Internet is a large system, with multiple providers ofequipment and services, with many entities playing many roles,some of which mirror others partially or wholly. It has becomeclear in recent decades that the tremendous success of the Inter-net is due to the possibility of many viable value propositionsto exist in this space, such that many different businesses canprofitably provide parts of the overall functionality. In doingso, often the interests of the various players come in conflictwith each other, though at the highest level they may be in asymbiotic relationship; in [3], the authors call these “tusslesin cyberspace”, and consider the function of archiecture tobe to provide the space in which such necessary tussles mayproceed, without restricting or biasing the outcome.

In functional terms, the purpose of architectural design isto articulate the necessary entity interactions required for the

978-1-4673-5775-3/13/$31.00 ©2013 IEEE

desired functionging of the whole system. The entity rolesmay be thought of containers of functionality, which arespecified by the architecture. Various roles may be fulfilled bydifferent participating business entities. The mechanism thateach participant decides to realize within their entity may bedifferent, but the interactions between them must be specifiedby the architecture, and the interfaces for such interactionsmust be standardized. Thus the identification of the roles andthe interfaces constitute what Clark et al. have referred to asthe “cut-points of the architecture”. For a diverse, collaborativeand dynamic enviroment such as the Internet, it is desirableto postulate a role only when the absence of any entity inthat role would leave some essential function unfulfilled in theecosystem. The goal of the architecture, then, is to identify aminimal set of roles and interfaces amongst them, such thatroles that are likely to be filled by different business entitiesin the real world are separated, there is a value propostionfor each role, and their interactions are clearly defined andspecified.

In what follows, we use the term “role” for an independentservice provider, and the term “entity” for units of functionalitywithin roles. First we describe the broader architectural visionof ChoiceNet. We then describe the verification architecture,keeping in mind that the architecture must not mandate heavy-weight interactions that themselves impose a large overhead ofcommunication on the network (though specific realizationsmay do so). It is important to note that a particular businessmay fulfill more than one of the roles below. Similarly, partic-ular business entities may carry out the mandated interactionsusing the interfaces we specify below, or other mechanismsthat they agree on, even paper contracts or phone calls (as is thecase for similar interactions today). However, by standardizingsuch interactions in the architecture, we seek to lower thebarrier for businesses that do not already have existing businessrelationships with other parties to enter into interactions withothers. Hopefully this aids the entry of innovators into themarketplace.

B. ChoiceNet

The success of the current Internet is driven very largely bythe economic interaction of many entities, acting as customersand providers of various hardware and software services fornetwork functions and computing and storage functions, allmediated by economic interactions. However, this economicreality is not represented in any of the network software orprotocol interactions, thus the economic incentive for providinginnovative services is detached from the network interaction.No valuable network function can be assumed to be free,or made available as a public service. In monetory transac-tions for services, competition improves the collective benefit.Competition is more effective when the marketplace is larger.A small marketplace can become a buyers market or sellersmarket more easily, and also because a larger variety ofbuyers means a larger variety of producers and sellers can besustained. In particular, small producers, who in a marketplaceof only a few large buyers will either become captive to oneof them or be driven out, may flourish in a large variegatedmarketplace containing small buyers who can be directly soldto. These reflections prompt the goal of ChoiceNet to introducearchitectrual entities into the Internet to enable fine-graineconomic interactions.

The ChoiceNet architecture provides a common platformfor service providers to advertise their services and for cus-tomers to easily discover, negoitate and pay for these services.The component empowering the advertisements for servicesis called the “marketplace” where service providers registertheir services and customers discover them via querying themarketplace. Once a customer decides the service to purchase,further ChoiceNet interaction between customer and providercreates a contract, with the customer receiving a token ofsome form. These interactions constitute what we term the“Economy Plane” of ChoiceNet, and are all paralleled by real-world interactions that take place, but outside the networkarchitecture, today. Subsequent to this, the customer may usethe token to obtain the desired service in the same manner aswould take place in the current Internet - we refer to this asthe “Use Plane”. Such services could be for endpoint services,path or pathlet services, or even in-network processing ser-vices. However, in the ChoiceNet view, the customer wouldundertake individual contracts with each of the providers ofsuch services along the complete path to compose the entireservice, and would be financially rewarding each provider, thuscreating incentive for innovation at all such service points,endpoints, paths, and intermediate processing/storage services.In reality, automated software acting on behalf of the userwould undertake these decisions, translating high-level goalsof user experience into low-level ones.

III. VERIFICATION SERVICE ARCHITECTURE

By connecting the user directly to each separate serviceprovider involved in delivering an overall service experienceto the user, and by enabling the user to contract with each suchprovider individually, ChoiceNet empowers the user to makefine-grain choices, and reward providers who perform wellwith continued contracts, while voting with their wallet againstthose that do not – i.e. choosing alternate service offeringsfrom other providers. However, for this, the user must be ableto discern which service provider is to blame if the user’soverall satisfaction metric is not met. If a user is not satisfiedwith the quality of experience in viewing streaming video, theyneed to know if the fault lies with the player or other localsoftware, their home network, their home Internet service, anyof the several provider hops their video traffic passes throughin sequence, or the video streaming service at the far end.This will generally require measurements to be made insidethe network cloud. Since the individual consumer does nottypically have presence except at the endpoints, this pointsto the potential of a specific class of service provider roles,independent third-party verifiers, and the need of architecturalmechanisms specific to the collection and providing of suchverification information. In this section, we state the require-ments and design decisions for such a candidate verificationarchitecture, followed by role descriptions and interactionsbetween these roles.

In what follows, we stress the role of the user (or softwareagent acting on behalf of the user) in choosing the serviceproviders by referring to this role as “chooser”. We take anexample scenario in which the chooser obtains service fromtwo bandwidth providers, NSP1 and NSP2, such that SP1carries the packets of the chooser to a Network Access Point,and NSP2 carries it from there to SP, the endpoint of the traffic

flow, which provides some computation or data service to theuser.

A. Architectural Roles

Figure 1 shows the roles of our verification architecture,where the dashed red arrows represent the service setup inter-actions (economy plane interface as yellow rectangles) and thegreen dotted arrows represent the measurement data transfer(use-plane). In addition, the thick blue solid line represent theservice received by the chooser from SP (use-plane interfaceas triangles). Moreover, horizontally and vertically dashedinterfaces symbolizes the ChoiceNet customer and providerinterfaces respectively. Also, rectangles with “S” and “R”denote the MSP sender and receiver interfaces respectively.Because MSP1 and MSP2 have very similar structures, MSP2is represented as a box.

The Marketplace

Chooser App

Mechanism

S

Analysis Logic Manager

R

Manager

Mechanism

S

MP1 MP2 MP3

SP App

Mechanism

Wrapper

SComposition Service

MSP2S

MSP1

MASP1

MCSP1

Chooser

SP

Wrapper

Wrapper

Use plane

Economy plane

Verification Plane

Fig. 1: Verification Architecture Roles

Measurement Service Provider (MSP): As the core role ofthe verification service architecture, an MSP is an independentprovider responsible for producing and delivering the mea-surement data (MD) relevant to the chooser. The role itselfconsists of several sub-entities as shown in Figure 1. An MSPutilizes one or more “Measurement Points (MP)” to producethe MD. An MP is the actual measurement equipment locatedinside the network, preferably at a point of presence or InternetExchange Point (IXP), typically with passive measurementcapability to capture chooser packets. Deploying MPs at IXPsis attractive because this requires no extra support or trustfrom any network service provider and yet the system canhave access to the ingress and outgress traffic of hundreds ofproviders [4] at a single location.

Once a packet is captured, the production of MD dependson the selected “MSP Mechanism (MM)”. An MSP can offer

several such MMs as part of the service advertisement, and thesame MM may need to be used at multiple MSPs to compose averification service. Choosers might prefer different MMs fordifferent applications and several MMs may run concurrently.Although MP and MM are logically separate entities, due toperformance reasons, it is suitable to have them as differentprocesses in the same device, or even have MMs executedas simple functions in MP. A simple MM example can bea “throughput calculation mechanism”, which calculates thenumber of bytes received within a second for the desired flow,but more complex mechanisms can be envisaged as shown inSection ??.

As part of low-level detail abstraction, An MP never di-rectly interacts with the customer. Instead, an “MSP Wrapper”entity provides the MD in a “service-specific” representation.Also the use plane for this service can optionally have aspecific interface, called “MSP Interface”, at sender as “S”interface and at the receiver as “R” interface respectively.The type and format of the messages exchanged by this MSPInterface are negotiated by the ChoiceNet Interface, includingthe encryption keys to maintain secure communication.

In Figure 1, each role also has a “manager” sub-entity,responsible for general administrative tasks such as configura-tion, logging, payment, identity, registry or advertisements, aswell as being a central entity for forwarding user requests toappropriate entities.

Measurement Composition Service Provider (MCSP): Inthe current Internet, a user flow typically goes through morethan one network service provider (NSP) before reachingits destination, therefore they may be several MSP choices.Having several such MSP alternatives with various capabitiliesto choose from, makes verification service composition achallenge by itself. Therefore an MCSP role makes suchservice compositions convenient for customers. An MCSP firstnegotiates with various MSPs to setup the desired measurementservice with appropriate metrics, mechanisms, MD represen-tation. MCSP then finds an appropriate analyzer (describedbelow) for verification. Having found the entities, MCSPreturns a “recipe” to the customer. Having got the recipe, thecustomer can now communicate with the providers to make aservice contract.

It is important to note that MCSP is a special case ofservice composition, a general feature of ChoiceNet. That is,it can be realized as part of a general ChoiceNet servicecomposition role, instead of a separate entity in Figure 1.Also, it may very well be implemented as a software in cus-tomer’s computer, or alternatively, as part of a large provider(say provider X) which also has MSP functionality. The lastscenario allows “MSP peering”, where such a provider X cancontract with another MSP, in a ChoiceNet customer/providerfashion, to broaden its measurement capability geographicallyand to offer advertisements especially desirable for customerscommunicating over long distances. In that case, a customermay only be negotiating with provider X which may indeedhave multiple providers performing as subcontractors. Further-more, a hierarchy of MSPs is possible, where a provider X mayhave contracts with many geographically distant small MSPsto increase its competitiveness. We anticipate our work is todesign the architecture enabling such possible scenarios andlet the market decide for feasible practices.

Measurement Analysis Service Provider (MASP): The rawMD itself is not generally useful to the customer until it isanalyzed, which makes MASP a crucial role for verification.Having MD as input, MASP typically applies statistical meth-ods (similar to the measurement tools in Section IV) to provideuseful information to the customers. However, we anticipate avariety of “detail levels” can be desirable for distinct customersand their applications. While some applications or users mayonly ask for a “yes/no” reply for each service provider, othersmight want more in-depth analysis. Therefore MASP shouldsupport various levels of detail. Additionaly, as part of servicenegotiation, a MASP optionally indicates a receiver, “R”.Similar to MCSP above, a MASP can also be implementedas a software in customer’s computer or as a part of ageneral analysis provider, or alternatively as part of a largemeasurement provider that has MSP or MCSP.

Chooser and SP: In some cases (for example, dynamicallychanging throughput measurement, described in Section ??),a MASP may require the MD input from the chooser and SP (aservice provider that users get end-point service such as videostreaming) in order to compare them with MD collected fromMSPs. In that case, chooser and SP3 act similar to MSPs fortheir flow and should run a measurement mechanism, possiblyas a plug-in.

Other Business Roles: A viable business proposition foran entity exists if its role requires some specific stock-in-trade (equipment, right-of-way, intellectual property on customalgorithms, etc.), and adds some value for another entity thatcan only be produced with the stock-in-trade. Though a fulldiscussion is out of scope of this paper, we can show notonly that such value propositions exist for each of the aboveentities, but for other innovative roles. Such roles could be thatof a “Measurement Metric Converter Service” which helpsthe chooser convert high-level metrics such as “satisfactorySD video quality” to “≤ 5 ms jitter and ≥ 200 Kbps overany 100 ms” or similar low level ones. ChoiceNet requiresan identification framework that we have not discussed; inthe context of measurement, providing proxy identificationfor measurements to preserve privacy of a monitoring entitymay be a value added service. An “NSP reviewer service”can collect long-term performance data by performing variousactive measurements at various network conditions, and sell“consumer reports”. Thus there are many more business nichesin this ecosystem than the principal roles above.

B. Interactions and Interfaces

Having described the roles, we now turn our attentionto the interfaces shown in Figure 1, which play a criticalrole in ensuring that customers and providers understand eachother. Because setting up a verification service involves severalstages, we first describe the general information exchangestructures, followed by the set of interactions to perform theservice. It is important to note that the architectural roles wediscussed earlier, also use the native ChoiceNet services andthus their interactions, such as marketplace queries, paymenttransactions, authentication messages. However, these interac-tions are not the focus of our work so we prefer explainingthe information exchanges only related to our work. The focusof this work is to articulate the necessary interactions. Toillustrate them, we use a standard XML interface. The XML

samples below are intended as examples only; as we remarkin Section IV, an exhaustive taxonomy or ontology is a topicfor further research. Due to the space limitations we onlygive some of XML samples exchanged by different entities,rather than exhaustively showing all the messages that couldbe exchanged in our scenario.

Flow is defined by the endpoints in Listing 1, whereendpoints are represented by the five tuple (source IP address,source transport port, destination IP address, destination trans-port port and the transport protocol). While this is the mostcommon way to define a flow, the extensible nature of thearchitecture allows different flow definitions, without changingthe architecture itself. For example, IP version 6 can easilybe defined by only changing the version attribute value. Infact, the architecture even allows aggregated flows, such asflows based on MPLS or VLAN tags or IP address prefixes.Furthermore, we envision that this extensibility facilitatescompatibility with Future Network Architectures.

Listing 1: Flow definition1 <flow>2 <endpoint t y p e ="source">3 <ip v e r s i o n ="4">chooser IP</ip>4 <port t y p e ="tcp">chooser port</port>5 </endpoint>6 <endpoint t y p e ="destination">7 <ip v e r s i o n ="4">service provider IP</ip>8 <port t y p e ="tcp">service provider port</port>9 </endpoint>

10 </flow>

Path is defined by the endpoints and a number of serviceproviders in between. The endpoint definition discussed as partof the flow definition, ingress and engress can be defined byprotocols other than IP, if negotiating parties agree. Metricscan be used to represent a constraint or a measurement request.Mechanism can be defined by its name and input parametersto be used. Wrapper is essential for 3rd parties to interpretthe meaning of MD sent from several MSPs. Analysis resultsmust be easily consumable by the customers regardless ofthe mechanisms or wrapper formats. Preference defines thecriterias of the customer in a request to a MCSP, such as thelowest price.

Phase 1: The chooser first searches for a verification serviceand MCSP from the marketplace as shown in Message 1 inFigure 2. The chooser then sends “flow”, “path”, “metric”with constrainsts, “mechanism”, “wrapper”, “analysis” and“preference” information to MCSP represented by Message2 in Figure 2.

Phase 2: The inner workings of MCSP is not the scope ofthis work. However, MCSP can query the marketplace as wellas its known providers to get information. What matters to usis the “recipe”, i.e., the list of providers with their interfacesto realize the service, returned to the chooser as shown inMessage 3 in Figure 2.

Phase 3: Having a recipe, the chooser now makes individualcontracts with MASP, SP, MSPs. The first provider to makecontract is the MASP because it will provide the interface thatother providers need to send MD to. The chooser message in-cludes the “flow”, “path”, “metric” for analysis, “mechanism”,“wrapper”, “analysis” as shown in Message 4 in Figure 2.MASP1 replies with an “MSP Interface” accompanied by

Chooser MCSP1 MSP1 MSP2 MASP1 SP

3

Co

mp

osi

tio

nC

on

trac

t Se

tup

Marketplace

12

4

67

9

13

10

1415

12

18Res

ult

s

5

8

11

1617

Fig. 2: Time-Sequence Diagram for Interactions between enti-ties. Each number M represents a message, which is referredas Message M in this report.

credentials for each entity. That is precisely the reason whyMessage 4 includes 4 arrows. The chooser now makes contractwith SP, MSP1 and MSP2 as shown in Messages 5, 8 and 11in Figure 2. Having their internal setup done, SP and MSPstry to connect to MASP1 with these credentials as representedin Message 6, 9, 12 in Figure 2. Upon success, each will senda “ready” message to the chooser as shown in Message 7, 10,13 in Figure 2. Note that the exact order of these messagesdoes not matter, but once all “ready” messages are received,it signals to the chooser that the system is now ready forverification service.

Phase 4: The chooser starts use-plane interaction, and in thisscenario, we are sending packets from the chooser to SP.Chooser, MSP1, MSP2 and SP now sends MDs to MASP1as shown in Messages 14, 15, 16, 17 in Figure 2 respectively.Listing 2 shows a sample MD based on the marker mechanismwith incomplete information such as hardware or software tags.However, we emphasize that a complete MD specification isa future work.

Listing 2: A sample measurement data1 <md name="markerPacketMeasurementData" v e r s i o n ="1.0">2 <experiment i d ="45632fd"> . . . </experiment>3 <node i d ="5472473453"> . . . </node>4 <mechanism name="marker" v e r s i o n ="1.0"> . . . </mechanism>5 <element t y p e ="receivedBytes">6 <value t y p e ="bytes" >58460</value>7 . . .8 </element>9 </md>

Phase 5: Having collected the MD, MASP1 now providesa sample output to the chooser as shown in Message 18 inFigure 2 and it is defined by the “result for verification”message in Listing 3.

Listing 3: A sample verification result1 <result name="measurementResult" v e r s i o n ="1.0">

2 <provider t y p e ="nsp" i d ="nsp1" owd="yes" t h r ="yes"/>3 <provider t y p e ="nsp" i d ="nsp2" owd="no" t h r ="yes"/>4 <provider t y p e ="nsp" i d ="nsp3" owd="yes" t h r ="yes"/>5 </result>

IV. RELEVANT PRIOR WORK

A great deal of research work exists in passive and ac-tive monitoring of the Internet that are relevant to the theVerification Service framework we envision, and fit naturallyinto measurement and analysis service provider roles. In thissection we briefly discuss some of these relevant approaches.

Because network measurement was not a priority in theinitial Internet design goals [5], support for measurement isarchitecturally minimal. Utilizing control protocols such asICMP or hacking IP TTL field for probes represets impor-tant work but of limited power. Measuring the end-to-endperformance of applications has been the primary concern forendnode users. Even with the limited support from the architec-ture and current practical problems, combining the active andpassive measurement techniques with statistics has providedsignificant insight about the behaviour of Internet traffic suchas path stability, delay characteristics, link bandwidth, bottle-neck bandwidth, available bandwidth [6]–[8]. These tools havehad to depend on unrealistic simplifed assumptions (symmetricpaths, symmetric RTT, simplified queueuing model), and havenot provided feedback to customers about the performanceexperienced by individual flows.

To address the need for more systematic, scalable and accu-rate measurements for the large Internet infrastructure, novelnetwork measurement architectures [9]–[11] were proposed,but none addressed a realistic business model beneficial forall entities, and most did not design a complete co-dependentecosystem. To the best of our knowledge, the PerfSONARproject [11] is the only one that attempts a task closest to archi-tecting the complete ecosystem we see as necessary. Its SOAbased approach improves robustness and flexibility by inde-pendent entities each providing different services, including ameasurement layer to obtain actual measurements; the servicelayer to mediate measurement delivery as well as discovery,security; and end-user tools to visualize the measurement data.It also uses existing tools such as NetFlow or SNMP. Weenvisage that a realistic realization of our architecture would beable to reuse as much of existing technology as possible. In ourenvisioned architecture, perfSONAR entities and mechanismscan participate seamlessly as measurement service providers,or various components thereof, and the IPFIX standard (andits close predecessor, Cisco’s NetFlow) are of similar interest.

However, a complete verification architecture (as opposedto measurement architecture such as perfSONAR) requires theadditional key characteristics of (i) measurement compositionservice, to know what to measure to verify a given usersatisfaction, (ii) measurement analysis, to perform the actualverification, (iii) integration with an economic marketplaceframework, to enable a thriving ecosystem to come into beingthrough mutual benefit.

A common definition for measurement metrics is providedby the IP Performance Metrics (IPPM) [12], which we adopt;however the topic of a semantically enriched, flexible and ex-tensible representation of measurement metrics remains worthy

of further exploration. The emergence of Software DefinedNetwork technologies such as OpenFlow shows on-demandflow setup with integrated statistics collection at intermediatenodes, as we postulate, is not realistically unachievable.

V. PROOF OF CONCEPT

We realized a prototype in NS-3 for our verification archi-tecture, to first show a simple and then complicated experi-ment, where the entities are decoupled and are collaboratingthrough the ChoiceNet interfaces. The purpose is not to designan optimal system in any sense, but to show that even asimple and straightforward mechanism can provide meaningfulfeedback to the user, simply by allowing the user to forma partnership with the MSP, and access information withinthe network cloud. We first introduce our novel yet simplemechanism, called the “marker mechanism” with a discussionon its benefits, followed by the experiment topology and thecomparison of its results with existing measurement tools.Next, we give a more complicated simulation that apportionsthe overall jitter contribution of each provider in a videostreaming case, in order to show how the power of separationof roles achieves a harder verification task.

1) The Marker Mechanism: As the name implies, the“marker” mechanism is based on marking some of thechooser’s packets, while the chooser is sending a stream ofdata traffic from the destination. Note that this is a case wherethe chooser runs a plug-in agent of the measurement serviceprovider, as mentioned Section in III. The pseudocode ofthe mechanism is shown in Algorithm 1, where each markedpacket has a sequence number to distinguish that marker packetand in our simulations, the sequence number is stored at theapplication payload of UDP packets. When a measurementpoint (including chooser and SP) captures the initial markerpacket (say sequence number 1), it records the timestamp ast1 and starts adding packet lengths of the packets belongingto that flow, received after this initial marker packet. Whenthe chooser sends the next marker packet at time tc, each MP(including chooser and SP) records the timestamp of this newpacket as t2, calculates the number of bytes recorded, called Bbetween t1 and t2, and then sends an MD to MASP with thefive tuples; the sequence number, t1, t2, B and MP ID. MASP(the analyzer) can then calculate the throughput as (B/(t1−t2))and OWD (one way delay) as (t2− tc) for each MP. However,it is up to MASP to do any additional calculation to performadvanced analysis.

Under perfect time synchronization, no packet loss, noqueuing delay and no out-of-order packets, the marker mech-anism should give 100% accurate throughput because it willcalculate the same exact number of bytes at each MP betweent2 and t1. Also, perfect synchronization naturally gives theperfect one-way-delay at each MP. We observed this expectedphenomenon in our simulations. We also note that losing aregular (non-marker) data packet does not affect the accuracyof the mechanism, because MPs are still able to correctlycount the number of bytes received between marker packets.In a typical 1% packet loss scenario, losing a marker packetis relatively rare. Thus, the effect of low packet loss rate onthe overall system is also low. However, marker packets canget lost and in such cases, an MP will keep counting thenumber of bytes until the next marker packet arrives. When it

Algorithm 1 MARKER(flow f )

while (p = recvFromFlow(f )) doif p.isMarker() then

if p.isFirstMarker() thenf [cT ime] ← GetTimeNow()f [lastSeqNum] ← 0f [rxBytes] ← 0

else if f [lastSeqNum] < p.seqNum() thenf [pT ime] ← f [cT ime]f [cT ime] ← GetTimeNow()f [rxBytes] ← f [rxBytes] + p.packetSize()send(mp id, p.seqNum(), f [rxBytes], f [cT ime],f [pT ime])f [rxBytes] ← 0f [lastSeqNum] ← p.seqNum()

elsef [rxBytes] ← f [rxBytes] + p.packetSize()

end ifelsef [rxBytes] ← f [rxBytes] + p.packetSize()

end ifend while

receives the next marker packet, the MP will then send MD asshown in Algorithm 1. Now the analysis (MASP) provider caninterpret the loss of a marker packet by the sequence numberdifference and can average the throughput accordingly. Thesame holds also for multiple marker packet losses. Similarly,out-of-order regular data packets do not affect the accuracyof measurement and in some cases, where the marker packetsthemselves arrive out-of-order, MP simply discards the smallerone by keeping a counter of the maximum sequence numberreceived as shown in Algorithm 1. Variation in queueing delayaffects the throughput because it causes some packets to arrivelater than normal (reducing throughput) and others to arrivequicker and causing peaks in throughput. However, its effectis relatively low especially in low granularity high throughputmeasurements. Combined with one way delay measurement,MD of marker mechanism can reveal which provider is causingthe queueing delay as we discuss in the simple experiment. Inaddition, this mechanism highly flexible because the markingrate can be adjusted based on the chooser preference, thethroughput, granularity level without any requirement on thetiming or packet size. Furthermore marker information (suchas the sequence number) can be inserted into the data packetsas well.

2) Simple Throughput Measurement: Figure 3 shows ourNS-3 simulation topology, where the SP (end service provideras discussed in Section III) sends variable-sized UDP packetswith varying intervals to the chooser through NSP1, MP1,NSP2, MP2 and NSP3 nodes. MP1 and MP2 are the measure-ment points capturing all of the packets on the way, runningthe measurement mechanism and sending MDs to MSP1. MP1and MP2 links never get congest and add no delay. MSP1then sends the MD to the chooser. MASP is also located atthe chooser and receives MD from the chooser, MSP1 andSP. BG1 and BG2 send TCP and UDP background traffic toBG3 and BG4 through SP1, and BG4 also sends traffic to thechooser through NSP2 and NSP3.

NSP 3

SP

BG 1 BG 2 BG 3 BG 4

Chooser

MP 1NSP 1 NSP 2 MP 2

MSP 1

10 Mbps2 ms

20 Mbps40 ms

20 Mbps30 ms

5 Mbps5 msMD Traffic

UsePlane

Fig. 3: NS-3 Topology. The dashed boxes represent MSP andMP entities, whereas oval BG traffic is denoted by the dashedarrows.

The results for throughput and one way delay are shownin Figure 4 and Figure 5 respectively, where our network has1% packet loss for the path and 10ms average queueing delayper packet. The SP’s sending rate is varying (not constant bitrate) and that is why, it is also an important input parameterfor throughput verification (unlike existing measurement toolsdiscussed in Section IV), which the makes the problem morechallenging. As shown in Figure 4, even a mechanism assimple as the marker mechanism achieves a high accuracy.We also note the two interesting observations; i) in Figure 4at t = 6, the MP1 throughput is higher than the SP’s sendingrate. This is due to the number of temporarily queued packetsat NSP1 after t = 4.7. Those queued packets are later sentby NSP1 and are counted as part of the next measurement,showing that queueing took place within that time interval.ii) in Figure 5 (not belonging to the same measurement) wecan observe a peak in MP2 delay at t=4 is due to queueingdelay of NSP2, while there is no change at MP1 measurements,so NSP1 is not to blame in that time interval, whereas att=20, we have introduced an additional background trafficto NSP1 from BG1 and BG2, and that is why, the SP flowis affected significantly, whereas the contribution of NSP2 islow, therefore NSP1 is to blame within that time interval. Wecan also observe that NSP3 has not introduced any signicantperformance problem during this simulation.

500

510

520

530

540

550

560

570

580

590

600

1.1 2.4 3.6 4.7 6.0 7.2 8.4 9.5 10.7 11.9Time (s)

SP

MP1

MP2

Chooser

Throughput (Kbps)

Fig. 4: Throughput Results vs. Time at each MP, where eachcluster represents the same marker packet measured at eachentity with the time axis denoting the time SP sent it.

0

50

100

150

200

250

0 4 8 12 16 20 24 28 32 36 40Time (s)

MP1

MP2

Chooser

One way delay (ms)

Fig. 5: One-way-delay vs. Time Results, where each data pointrepresents a marker packet measured at each entity with thetime axis denoting the time SP sent it.

3) Overhead and Delay Benefits: MD transmission shouldintroduce low overhead on the network. It might appear thatMD being collected all over the network and being sentback to an analysis provider (MASP), which further sendsthe analyzed report back to the chooser, may impose severelyincreased overhead and delay. We provide the arguments toshow that this is not necessarily true, and in fact the overheadmay be far lower with the proposed architecture. Assume ascenario where a path consisting of M=5 NSPs, and SP issending data at 2Mbps. Including SP and the chooser, thereare M + 2=7 MD producers and with 1 in 100 markingrate, we approximately send 5 marker packets per second.Conservatively speaking, independent marker packets (with200 bytes data) add ∼10Kbps extra network load at each MPand including flow’s marker packet to the 7 MD makes soin total it makes 80Kbps extra load on the whole network.Comparison studies [7], [13] show that measurement toolssuch as Pathchar, Cprobe or Pathload typically use severalMbps. This mechanism also consume at least one order ofmagnitude less than the more conservative tools such as spruceor pathchirp. In the following subsection, we describe a morecomplicated simulation to illustrate a harder verification task.

4) Jitter Apportionment for Video Streaming ExperienceQuantification: We now show a sophisticated simulation byusing the same exact MSP, mechanism and topology as theprevious experiment, but a more complex analysis service(MASP) is used in a video streaming scenario with differenttraffic pattern. Video streaming is commonly used today butthe video experience of a user can vary greatly and thereis no easy way to apportion the blame for unsatisfactoryexperience among multiple bandwidth providers. We take jitteras a commonly accepted metric for video experience and usepublicly available MPEG-4 video traces to develop a simplevideo streaming sender and receiver program in NS3 where,the artificial video player at the receiver requires at least 25frames in its buffer to play the “video”. If there are less than25 frames in the buffer, then the video player “freezes”, i.e.,waits for the buffer to fill up and retries to “play” at the nextsecond. Additionally, if a frame or a part of the frame is lost,the frame is discarded, recorded as a “glitch” and it is notretransmitted. Note that in this experiment, all losses are dueto queue buffer overflow.

0

100

200

300

400

500

600

700

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30Time (s)

MP1

MP2

Chooser

Glitch

Freeze

Jitter atChooser

One way delay (ms)

Fig. 6: OWD, Jitter and User Experience Quantification, wherethe time axis denotes the time SP sent a marker packet andthe vertical lines represent “freeze” or “glitch” occurrences.

0

100

200

300

400

500

600

4 8 10 12 15 17 20 22 25 28

Delay (ms)

Time (s)

MP1 OWD

MP2 OWD -MP1 OWD

Chooser OWD -MP2 OWD

Fig. 7: Additional delay per provider. For example, “MP2-MP1” represent the delay difference between MP2 and MP1for each marker packet received.

Figure 6 shows the one-way-delays measured for eachmarker packet at MP1, MP2 and the chooser respectively, aswell as the jitter (packet delay variation) experienced at thechooser by taking the delay difference between two consec-utive marker packets, as described in [14]. While these areimportant metrics, our goal in this experiement is to quantifythe user’s experience, which in this case is the number offreezes and the glitches that reduce the video quality expe-rience with respect to time, and more importantly apportionthe blame among multiple providers. Until t = 5, the packetsgo through smoothly and the video player at receiver buffersthe incoming frames. However, after t = 5, we inject extratraffic entering NSP1, so NSP1 experiences queueing delaysincreasing the jitter, and we observe glitches as “X” marks onX axis, with their respective vertical lines in Figure 6. Havingseveral glitches, the buffer finally experiences an underflowand video player “freezes” at t = 8 and t = 10. At t = 12,we remove that additional traffic, so the load is again lowtill t = 17. At t = 17, we inject extra traffic to NSP2 andthis additional traffic goes to the chooser through NSP3. Notethat MP2 does not measure a variation in delay, whereas thechooser experiences huge delays and jitters, causing againglitches followed by a freeze. This means, NSP2 easily handles

TABLE I: Jitter Apportionment percentages per provider foreach freeze

Time interval NSP1 NSP1 % NSP2 NSP2 % NSP3 NSP3 %Freeze 1 (t=6 - 10) 27.42 96.7% 0.06 0.2% 0.88 3.1%Freeze 2 (t=8 - 12) 28.74 98.5% 0.003 0.0% 0.42 1.5%

Freeze 3 (t=17 - 21) 18.02 27.7% 0.14 0.2% 46.96 72.1%Freeze 4 (t=24 - 28) 8.86 14.0% 0.10 0.2% 54.42 85.8%

Overall 16.84 44.6% 0.08 0.2% 20.82 55.2%

the extra load, whereas NSP3’s bottleneck link is causing thecongestion and is to blame in such a scenario. After t = 26,all background traffic is removed, so the user experience goesback to normal. Furthermore, Figure 7 shows the delays addedby each NSP during the same experiment. It clearly showsNSP2 adds almost no delay under any circumstances, whileNSP1 is to blame for the problems between t = 5 and t = 12.However, we can easily observe that NSP1 is not to blamefor the problems encountered between t = 17 and t = 26, soNSP3 is to blame for the delay between that time interval.

While this case study clearly shows the power of suchan architecture, the customer might not be interested in in-terpreting the figures, but instead might just prefer a simpleexplanation for each NSP’s performance. MASP is the entityresponsible for this task and we now provide such an expla-nation using the same MD used in Figure 6 and 7. We firstdenote the “contributing delay” in Equation 1, where dki is theone-way-delay (owd) measured at node k for a marker packetwith sequence number i and the contributing delay cdki refersto the owd that is measured for node k, on top of the measuredowd value at the previous node k−1, for a marker packet withsequence number i. Also the first node is the sender whichwe assume introduces no delay difference, i.e., cd1i = d1i .Similarly, the “contributing jitter” cjki in Equation 2 is thenthe difference between the last two cd values, that we use for“jitter apportionment” calculation.

cdki = dki − dk−1i (1)

cjki = cdki − cdki−1 (2)

The simplest approach is to take the mean of the contributingjitter for the marker packets received at each provider, denotedby mcjk at node k in Equation 3. Note that, because jitter isthe difference between two packets, we do not count the firstpacket and so we have n−1 values. Also we take the absolutevalue of cjki so that positive and negative jitter values do notcancel each other.

mcjk =

∑ni=2(

∣∣cjki ∣∣)n− 1

(3)

We have shown the values and percentages per providers foreach freeze time interval. Note that different approaches suchas standard deviation of jitter and maximum jitter can be alsocalculated, but due to space limitations, we do not show thoseresults.

Below we describe a few representative issues in consider-ing transitioning to a real-world realization of this system. Firstand foremost, it may appear that wirespeed measurement atIXPs is unrealistic; however, deep packet inspection is carriedout routinely by some providers, here we simply suggestthat the specialized equipment to do so may be suitable

for supporting an additional business model (MSP), not justinternal monitoring for the provider. Using optical splitters,the OCXMON [9] project showed that passive monitoringat high-speed optical links is indeed possible, as long as adecade ago. Today, DAG cards [15] offer packet capturing at10GigE or SONET network interfaces. In order to providethe synchronization between MPs, the chooser and SP, ahighly-accurate (less than millisecond) clock synchronizationtechnology is necessary. On WANs, NTP may have around 10millisecond error. Thus, we envision GPS-based solutions [10]are worth using despite the installation costs.

Some mechanisms may utilize “marker packets” that re-quires identifying each packet uniquely. Even though a uni-directional flow is identified by the five tuple (IP src/dst,transport src/dst port and protocol), it is harder to uniquelyidentify a flow’s packets. In theory, the IP Identification fieldcan be employed however, its platform-dependent implementa-tions and 16-bit field makes it undesirable packets in practice.One way is to add a sequence number to application field atUDP packets, or utilize that of TCP packets.

VI. SUMMARY AND FUTURE WORK

We believe a system that explicitly makes various choicesavailable to end-users will encourage the emergence of highquality and variety of innovative services in the Internet, andverification is an essential part for rewarding providers whosatisfy customer expectations. Gaining from past measurementsystems, tools and experiences, we proposed a measurementservice architecture for verification, with its components andinterfaces. Because the services are usually provided via col-laboration of several network providers, this service orientedarchitecture has an emphasis on verifying the quality of serviceat each provider and finding out the provider which causedthe performance problem if a service quality is not met. Ourproof-of-concept simulation demonstrates the ease with whichsignificantly useful monitoring data can be supplied to theuser to “know what happened”, once a distributed verificationservice architecture is postulated. Our ongoing work is todevelop a complete realistic prototype, on the GENI testbed orsimilar, to realize this measurement system. There are manychallenges in measurement service description, composition,and analysis that are opened up by this architecture, and canprovide subject for meaningful future research.

The authors would like to thank the PIs of the ChoiceNetproject, Drs. Tilman Wolf, Ilia Baldine, Kenneth Calvert, JimGriffoen, Anna Nagurney, George Rouskas, and collaboratorDr. Shu Huang, for their valuable feedback. This work wassupported by NSF NeTS award 1111276.

REFERENCES

[1] NSF Future Internet Architecture Project. http://www.nets-fia.net/.[2] Tilman Wolf, James Griffioen, Kenneth L. Calvert, Rudra Dutta,

George N. Rouskas, Ilia Baldine, and Anna Nagurney. Choice as aprinciple in network architecture. SIGCOMM Comput. Commun. Rev.,42(4):105–106, August 2012.

[3] David D. Clark, John Wroclawski, Karen R. Sollins, and Robert Braden.Tussle in cyberspace: defining tomorrow’s internet. In Proceedings ofSIGCOMM ’02, pages 347–356, New York, NY, USA, 2002. ACM.

[4] Bernhard Ager, Nikolaos Chatzis, Anja Feldmann, Nadi Sarrar, SteveUhlig, and Walter Willinger. Anatomy of a large european IXP. InProceedings of ACM SIGCOMM 2012, pages 163–174. ACM, 2012.

[5] D. Clark. The design philosophy of the DARPA internet protocols.SIGCOMM Comput. Commun. Rev., 18(4):106–114, August 1988.

[6] Van Jacobson. pathchar − a tool to infer characteristics of Internetpaths. Slides available from ftp://ee.lbl.gov/pathchar/, April 1997.

[7] Kevin Lai and Mary Baker. Measuring link bandwidths using adeterministic model of packet delay. SIGCOMM Comput. Commun.Rev., 30(4):283–294, August 2000.

[8] Vinay J. Ribeiro, Rudolf H. Riedi, Richard G. Baraniuk, Jiri Navratil,and Les Cottrell. pathchirp: Efficient available bandwidth estimationfor network paths. In Proceedings of PAM 2003.

[9] T. McGregor, H.-W. Braun, and J. Brown. The NLAMR networkanalysis infrastructure. Communications Magazine, IEEE, 38(5):122–128, may 2000.

[10] Matthew J. Zekauskas Sunil Kalidindi. Surveyor: An Infrastructure forInternet Performance. In IETF, 1999.

[11] Andreas Hanemann, Jeff W. Boote, Eric L. Boyd, Jerome Durand,Loukik Kudarimoti, Roman Lapacz, D. Martin Swany, Szymon Trocha,and Jason Zurawski. PerfSONAR: a service oriented architecture formulti-domain network monitoring. In Proceedings of ICSOC’05, pages241–254. Springer-Verlag, 2005.

[12] V. Paxson, G. Almes, J. Mahdavi, and M. Mathis. RFC 2330:Framework for IP performance metrics, May 1998.

[13] Daniele Croce, Marco Mellia, and Emilio Leonardi. The quest forbandwidth estimation techniques for large-scale distributed systems.SIGMETRICS Perform. Eval. Rev., 37(3):20–25, January 2010.

[14] C. Demichelis and P. Chimento. IP Packet Delay Variation Metric forIP Performance Metrics (IPPM), November 2002.

[15] Endace DAG Cards Website. http://www.endace.com/endace-dag-high-speed-packet-capture-cards.html.