Distributed Databases for Challenged NetworksVinoth Chandar, Ashwini Athalye

Most delay tolerant networking research has focussedon developing routing protocols that optimize data deliveryof a single data unit from a source to potentially multipledestinations. However, distributed applications built on top ofsuch intermittently connected networks, typically involve morecomplex data access mechanisms. Distributed databases arewidely used to capture such interactions. Thus, exploring thefeasibility of building a distributed database system for DTNsis an intriguing research problem. Applications can benefitimmensely from the resulting location transparency and datadecoupling.

The paper proposes a distributed database system for suchnetworks. The key contributions of this work are 1) practicalheuristics for query scheduling 2) a prereplication schemethat reduces the cost of on-demand retrieval by actively pre-caching data. To our knowledge, this is the first work toexamine these issues in an unified framework. We describethe results of several simulations conducted on a wide varietyof artificial and real world traces.

I. INTRODUCTION

DTN is a category of networks that is characterized byintermittent connectivity, irregular connectivity patterns,unpredictable communication opportuntity and diversedevices. As explained in [1], these types of networks arebecoming important with the pervasiveness of wirelesstechnology. For example, the campus wide deploymentof wireless technology coupled with the large number ofusers having laptops/cell phones with wireless capability,provides enormous opportunities for exchange of ’good-to-know’ information amongst students. Such good-to-knowinformation can substantially augment the quality of decisionmaking in daily life. In the past, there have also been manyreal world DTN deployments like Zebranet[11], DakNet[12]and DieselNet[13]. Zebranet tries to study the movementpatterns and the behaviour of zebras, by attaching sensornodes to the Zebras. Daknet is one of the earliest practicalDTN systems, that provides Internet connectivity to ruralvillages in India. DieselNet is a DTN infrastructure deployedover the bus transport system in Amherst. These practicalsystems have clearly proved the utility of delay tolerantnetworking.

Thus, the wide applicability of DTNs call for a deeper lookinto the needs of the applications that can be supported overit. [2] stresses on the need to develop robust mechanismsfor DTN based collaborative applications to improve datapropagation rates in realistic deployments. Using thisinformation as a guiding light, we propose mechanisms

for query processing in a distributed Database for DTNenvironments. The benefits that an application can enjoy fromsuch a system are location transparency and data decoupling.Achieving location transparency in a DTN setting obviouslyresults in more flexible applications.

To our knowledge, no previous work has examined thisproblem of distributed query processing in a completelydecentralized DTN environment. This work tries to put intoperspective the issues that are to be addressed in order tobuild a practical system, over which multitude of interestingapplications can be written. The problem requires additionalmechanisms to improve latency involved in answering queries,due to the inherently time varying nature of the links. Such ascheme is vital to ensure the practical usefulness of the system.

The paper is organized as follows. Section 2 details therelevant background and the other related research in thisarea. Section 3 presents the motivations and challenges tosolving this problem.Section 4 discusses the problem of queryscheduling in DTNs. Section 5 presents the prereplicationscheme proposed. Section 6 and 7 contain details about meta-data exchanges involved and the packet scheduling criteriainvolved. Section 8 presents detailed results from simulationsof the scheme under various scenarios. This is followed by ashort observations section. Section 10 concludes after outliningsome of the future directions that we intend to pursue.

II. BACKGROUND AND RELATED WORK

Traditional query processing schemes in distributeddatabases construct an execution plan for an input query basedon the sites participating in the query and the size of thedata set that needs to be moved from one site to anotherfor the execution of the query. Algorithms that optimize forresponse time typically do so by reducing the amount of datashipped between sites, to execute the query. These schemes[5] however assume a fixed network topology which meansthat there are statically defined paths between nodes and thatthey are always up. This is contrary to what DTNs possess.In a DTN, simple optimization for communication cost maynot minimize the response time, since there can be scheduleswhich involve earlier contacts between sites, leading to lowerdelays, even with larger data shipping. Hence query schedulingis consequently more complex for such an environment. Thisis the focus of the research conducted in this paper.

ICEDB [8] proposes a system, where in users query a webportal, connected to devices on the road. Devices are sensornodes that build databases with the information of interest. Theportal is responsible for sending the user queries and getting

the results back from the devices. The queries are annotatedwith priorities and the devices schedule results to be movedto the portal based on the priorities. This work illustratesthe utility of using database queries to retrieve content fromintermittently connected nodes, since the application can workwith a simpler abstraction of a database table. [6] builds afederated database over Bluetooth. However, it assumes thatthere is no mobility and hence, does not apply to most practicalscenarios. [9] is a mobile surveillance system, where busesupload spatially and temporally tagged images to a base stationplaced in the bus depot. These data stores are then queriedfrom a central cloud server.

A survey of the related work gives insight into somepractical aspects of DTNs. [7] formally analyzes the problemof DTN routing at various degrees of knowledge of the futureand the network. MaxProp[10] is a DTN routing protocol thatbuilds relative meeting probabilities and uses it to forwardpackets to a destination. RAPID [3] is a DTN routing protocolthat can be optimized specifically for a routing metric such asworst-case delay, average delivery delay. In this paper DTNrouting has been formulated as a resource allocation problemunder a realistic scenario where both storage and bandwidthare limited. The selected routing metric is used to prioritizea packet at each hop and locally optimize the delivery ofpackets in the order of the change in their utility function.Another interesting work on DTN routing [4], DPSP, is tunedfor applications that use a publish scheme to advertise content,on topics and the nodes subscribe to different topics. Thisscenario does not require either the content provider or thecontent subscriber to have knowledge about each other. Also,the knowledge given out by the publisher and the subscriber ismade use of to effectively allocate the resources and to preventflooding of the network with replicated packets.

To summarize, the approach to query processing in DTNshas been largely centralized i.e one central server holds thenode-to-data mapping and uses it to perform query scheduling.Most practical DTN routing mechanisms work with heuristicsand rely on empirical results to prove their usefullness, in anattempt to keep the problem tractable.

III. MOTIVATION AND CONTEXT

DTNs offer opportunities to develop some exciting dis-tributed applications. For example, a newcomer to Austinmight want to know about good places to dine, live musicevents etc. A distributed peer to peer application can helphim to query the people around for this information, evenwithout Internet connectivity. Typically, this good-to-knowinformation is not very time sensitive and hence, can bedelivered over DTNs. Existing routing protocols in DTNstypically operate with a known source and destination. Theapplication developer needs to schedule packets to differentdestinations, according to the needs of the application. Theproblem becomes worse in case of open networks, where notevery node knows each other, even though unknown nodesmay carry relevant information. Thus, the application needs to

track the nodes that carry interesting information, increasingthe complexity of the application.

Hence, if applications query using an indirection of databasetables, which are then mapped to the nodes that containfragments of the global information, it can lead to reducedoverhead for applications and seamless integration of differentapplications to work with each other. Moreover multipleapplications can be built on top of this layer of abstractionsince they no longer need to have node specific information.In essence, building a distributed database system over DTNswould foster the development of myriad of interesting appli-cations, that in turn enrich the practical benefits of conductingdelay tolerant networking research.

IV. QUERY PROCESSING

In conventional distributed databases, selectivity factors areused to perform reductions on JOIN operations and optimizefor the response time of the query. Computing the selectivityfactor is a linear time operation that can be performed instan-taeneously. However, the traditional schemes break in a DTNsince there is no way of maintaining selectivity factors in aDTN. Most traditional schemes reduce the response time byreducing the amount of data shipped across the network. Sincethe data size shipped, directly contributed to the delay involvedin processsing the query , in a DTN setting, our approachlargely pursues this path.

We work with approximations of contact times/probabilitiesthrough metadata exchanges between the nodes. Deadlines areuseful in identifying the time until the data is valid. Thiscorrelates closely with the type of ’good-to-know’ informa-tion that the system intends to work with. Queries can betagged with deadlines to indicate the amount of time that theresults will be useful for. We also intend to reduce the queryprocessing delay, by sending results as and when intermediateresults are available, without waiting for the completion of theentire query, that potentially provides additional results. Theduplicate results can be filtered out through a delta detectionmechanism at either the query initiator or the intermediatenodes, using standard techniques.

We assume that applications that talk to each other haveknowledge about the database tables. The knowledge aboutthe horizontal fragmentation, including cardinalities, can bebuilt based on metadata exchanges. The packets that aresent through the network can be either those carrying thequery or packets that carry the results. We divide the setof queries that are issued by applications into Unions andJoins. Unions consist of queries on single tables that arehorizontally fragmented across various nodes. Unions aresimply sent to all the nodes that the query initiator knowsof, which contain the target query. Joins consist of queries onmultiple tables that can be horizontally fragmented and haveat least one common attribute. Joins require a much morecomplicated query scheduling mechanism. Joins, which arecostly operations even in a conventional distributed databasesystem, are far more expensive in DTN environments.

For distributed databases, a number of factors contributeto heuristics that determine how the query gets processed.These are typically the cardinality of a database at a node,the size of a tuple in the database, and together the twonumbers give the size of the data that needs to be movedbetween sites for complex queries. Query processing in DTNwould require heuristics to be designed such that they takeinto account, packet scheduling criteria for DTN as well asdatabase query optimization parameters. In this paper, wediscuss two heuristics for scheduling Joins 1)Heaviest site2)Nearest site.

A. Optimal scheme

Assuming the knowledge of contact times and contactdurations of the nodes helps in improving the performanceof routing mechanisms [7]. In many real life scenarios, suchaccurate contact information can be obtained, such as scheduleof students in a campus. But, when this information is notavailable,statistical approximations can be used in their place,based on contact histories. Even with this complete informa-tion, computing an optimal path is NP Hard [3].Given that theDTN routing problem, even with a perfect Oracle is NP Hard,the problem of scheduling the queries, which involves movingrelevant data to the processing site in the best possible manneralso becomes NP hard, since the query processing problem iscomposed of independent subproblems of the routing problem.

The non availability of selectivity factors, precludes anychances of reducing joins at intermediate sites. Also, withthe prereplication scheme discussed in section 6, even floatingapproximate and possibly stale values of the selectivity factorsin the network might not perform well, since the prereplicationwould invalidate the selectivity factors very quickly due toaddition of new content at the node. Moreover, selectivityfactors are very specific to a particular query. For a selectivityfactor to be useful, the new query needs to match very closelyto the original query. Hence, the relevant data must be broughtto a single common processing site, for query evaluation. Inthis paper, we primarily focus on choosing this processing site.Hence, we try to approach the problem using heuristics, localoptimizations, and techniques from existing work on DTNrouting and distributed database query optimization.

B. Heaviest site

This heuristic tries to optimize for lower delays, by movingthe data to the site which possesses the largest amount ofdata relevant to the query. The heaviest site is chosen as theprocessing site. Note that the processing site is one amongthe sites that have the relevant data. The intuition is that if theheaviest site is prevented from shipping its data elsewhere,the lesser bandwidth consumption would result in significantsavings in the response time.

Q : set of all nodes that are relevant tothis query.

T : set of tables that are required.Pseudo-code:For each node q in Q:

For each t in T:B[q] += cardinality(t)

Processing node = i , such thatB[i] = max(B[q]), for all q in Q.

C. Nearest site

This heuristic picks the site which all of the relevantsites (sites that house the relevant data) are more likely tomeet. Note that this processing site may not be amongst therelevant sites. The relevant sites may accumulate data at someintermediate node, that every node is most likely to meet.This heuristic tries to directly optimize for the time spent ingetting the required data to the processing site. The meetingprobabilities used in MaxProp[10] are used directly to computethe nearest site.

N : set of nodes known to current nodeQ : set of all nodes that are relevantto this query.

C[n] : cummulative meeting probabilityof nodes in Q to each node n in N

Pseudo-code:For node n in N:

For each q in Q:C[n] += meetingprobability(q,n)

Processing site = i such thatC[i] = max(C[n] ), for all n in N

V. PREREPLICATION

As mentioned earlier, the cost of on-demand retrieval ismuch higher in a delay tolerant network, since the links maynot become available again for a long time. In such cases,actively moving content to nodes that are likely to be involvedin relevant queries, would help reduce the latency involved inanswering the queries. Each node builds a popularity indexfor table combinations that are involved in the queries that itsees. Nodes also build the popularity index through exchangeof this information amongst themselves. Thus, a global pop-ularity metric is built for all the table combinations. Over alonger duration of system operation, this metric captures theprobability that a future query will be generated for each ofthe table combinations. Let us define the servability, of a nodeas the amount of popular content possessed by the node. Whentwo nodes meet, they exchange more popular content amongstthemselves.

When a node meets another,

* Updates popularities from other node

* Computes a Integer Knapsack to deter-mine the transfers that are to bemade to increase servability

* Utility of transfering a table frag-ment to another node is the amountof increase in servability due tothat transfer

* Cost of transfer is the bandwidthconsumed

* The maximum cost that can be incurr-ed is total bandwidth of the conta-ct Schedules the transfers that itneeds to make to the other node

A. Returning results

As more content is prereplicated, the more popular contentis widely spread across the nodes in the network. As a result,the number of relevant sites in a join query increases withthe amount of prereplication. Hence, the latency involved inprocessing the join also increases prohibitively. To keep thelatency low, two mechanisms can be used to return the results.

• Best : The processing site waits for all table fragmentsfrom relevant sites to arrive, to begin processing the query

• Fast: The processing site processes the query and sendsthe result out, as soon as atleast one portion of all thetables involved in the join are available.

The two mechanisms are a tradeoff between accuracy ofresults (best) and response time for the query (fast). Weconsidered two mechanisms at the two extremes for simplicityof analysis. Obviously, an intermediate scheme that balancesbetween the two can also be used.

VI. METADATA EXCHANGES

The unstructured nature of DTNs requires that a mechanismbe in place for each node to either have a global view of thenodes in the system or have some way of updating its localinformation based on interactions with other nodes. The latterinvolves learning state of neighbors by exchanging relevantmetadata between nodes in contact. Gradually as more andmore node exchanges happen, the metadata grows and getsreplaced with newer state.

The following meta data are exchanged• Meeting probability, as described earlier• List of tables possessed by each node, and their cardinal-

ities• Popularity indexes

VII. PACKET SCHEDULING

The packet scheduling scheme used must reflect the goalsof the database layer above. It needs to decide on specificper-packet and per-packet-per-hop metrics to replicate packetsin the network. We chose replication over forwarding sincereplication has better chances of delivering the packets quickly,even though it involves more resources. The following sum-marizes the packet scheduling scheme used in the paper.

When node A meets node BExchange metadataExchange directly deliverable content

ordered by deadlineFor all packets p in buffer:

if B has higher meeting probabilityto dest[p] than A:

Pick p for transferSend all picked packets in buffer order

-ed by deadlineExchange prereplication content.

Note that the prereplication content is sent at the last, ineffect , using only any amount of spare bandwidth that maybe available. Thus, prereplication is used as an add-on and itdoes not affect the data traffic.

VIII. EVALUATION

We have implemented our scheme on a DTN simulatorcalled [14] The ONE [see figure 1]. This Simulator providesthe following desirable features which led us to eventuallychoosing it for our project:

• Various Routing mechanisms like Epidemic Routing,MaxProp Routing, Spray and Wait etc. are supported.

• Different mobility models for simuating movement ofnodes.

• A visualization tool to view node movement as well asdata exchanges.

Fig. 1. Screenshot of ONE Simulator

We have implemented our scheme on top of the MaxPropRouter, provided by the simulator. The router maintains meet-ing probabilities of each node with other nodes it meets, whichis useful for nearest site selection.

As mentioned earlier, there is no existing work in thisspace to compare our scheme against. Hence we have tried toevaluate our scheme by testing it against different movementmodels. We have also varied the number of nodes in thenetwork to see the effect of node size on the performanceof the proposed scheme. Through our evaluation, we hope toget an insight into the goodput of the scheme as well as itsapplicability to different DTN scenarios.

We have analysed our scheme against 5 movement models:• Random Way Point: This movement model captures

completely randomized node movement.• Shortest Path Based: This model simulates the movement

of people. The nodes move along fixed paths, however

their movement along these fixed paths is random andbased on certain points of interest.

• Map Based: This movement model simulates DTN wherenodes have periodicity in their movement. We haveutilized the movement of trams in Helsinki as part ofthis model.

• DieselNet: This is a bus system that has been deployedby UMass consisting of 34 buses.

• ZebraNet: This is a trace of a network of 5 zebras whichhave sensors deployed on them.

The first 3 movement models are provided by the simulator.Due to simulation time constraints, we have performed anal-ysis on these models with 20 and 40 nodes. The remaining2 movement models - DieselNet and ZebraNet are real traceswhich were converted into a format that was acceptable to thesimulator. Hence we have been able to run our scheme onsimulation models as well as real traces.

A. Metrics

To evaluate the performace of our scheme, we consider thefollowing metrics.

• Percentage of answered queries: This metric was choosento get the throughput of the system for the total offorward(path along which query is sent) and reversepaths(path along which query result is sent).

• Percentage of Unions: This metric gives the peformancefor single table queries.

• Percentage of Joins: This metric gives the peformancefor multi table queries. This metric is especially usefulto verify our scheme since we expect performance toimprove for joins, especially with our processing siteselection scheme and the prereplicaton scheme.

• Average query latency: This metric is again importantto understand the behavior of our scheme. We expectour scheme to reduce the query latency i.e. decrease thedelay between the time when the query was issued by anode and the time that it received the results. The averagequery latency also provides a useful insight into the kindof deadlines that each DTN scenario is typcially able tohandle.

B. Scenarios

We present the results of the metrics introduced in theprevious section, for three scenarios.

• No repl - This represents the behavior of the system, withno prereplication, ’best’ return of results and heaviestsite query processing. This serves as the vanilla scenarioagainst which we compare the next two scenarios.

• Repl(Heavy,Fast) - This involves prereplication, ’fast’return of results and heviest site query processing.

• Repl(Near,Fast) - This is same as Repl(Heavy,Fast) ex-cept that nearest site query processing scheme is used.

C. Random Way point

The random way point model simulates randomly movingnodes. Although, it is a unrealistic mobility model since most

real world events are roughly deterministic, it serves as a goodstarting point for our analysis. The graphs in figure 2,3 conveyresults generated for 20 and 40 nodes in a Random Way Pointmodel.

Fig. 2. Query processing throughput for Randomly moving nodes

Fig. 3. Latency for Randomly moving nodes

D. Map based Tram movements

This model simulates the movement of trams in Helsinki,using WKT files that were generated from a real GIS system.It simulates periodicity in meeting patterns. Figures 4,5 showthe performance of the scheme for 20 and 40 people in thesystem.

E. Shortest path People movement

With the popularity of portable devices like hand held PDA’sand cell phones, DTNs with such devices as nodes is anotherpotential scenario. Hence the movement patterns for peoplegive an idea of connectivity in such an environment. Thescheme was evaluated for 20 and 40 nodes in the system [seefigure 6,7].

Fig. 4. Query processing throughput for Tram system

Fig. 5. Latency for Tram system

Fig. 6. Query processing throughput for People interactions

F. DieselNetDieselNet is a realtime trace which exemplifies the appli-

cability of our scheme for a real world scenario. The traces

Fig. 7. Latency for People interactions

were generated by contacts between metro buses, in Amherst.The number of nodes in the DieselNet traces is 34. Figures8,9 present the results of the simulation run for about a week.

Fig. 8. Query processing throughput for DieselNet

G. ZebraNet

Monitoring wildlife and natural phenomena is another in-teresting area for DTNs. It also presents a good case, wherea query processing scheme could be invaluable. ZebraNet isa real world deployment of sensors on 5 zebras. See figure10,11 for results.

IX. OBSERVATIONS

In general, Pre-replication provides improvements in latencyas well as the success ratio of the queries.However, the extentof benefit varies from one movement model to another.

• The pre-replication benefits are around 30-40% for suc-cess rate.

• Benefits for latency are around 30-50%.

Fig. 9. Latency for DieselNet

Fig. 10. Query processing throughput for ZebraNet

Fig. 11. Latency for ZebraNet

• In most cases the Nearest site heuristic performs betterthan Heaviest Site in terms of sucess rate as well as

latency.

X. CONCLUSION AND FUTURE WORK

This is the first work to address the problem of queryscheduling in a completely decentralized and adhoc , delaytolerant network. This is a very interesting research area withenormous scope for improvement. We intend to pursue furtherresearch along these lines.

• Building a DB Aware packet replication scheme withmulticast capability would help improve performancesince the query issuals are inherently multicast traffic.

• If the intermediate nodes cache previous results, queriescould be answered from these caches, improving latency.However, this would involve complex tradeoffs in buffermanagement. Also, the applications should be willing toaccept some degree of staleness in the results.

• Given that many real world DTN environments have nearperiodic events, a local prereplication scheme could beused wherein the nodes perform the prereplication basedon the queries that they answer and route. This is differentfrom the current scheme where the global popularities areused.

• A very important contribution would be to improve theheuristics used for query scheduling. Ways to reducemulti table joins, parallelly in this environment, is a openproblem.

• In our scheme we were not able to use bandwidth betweennodes as a metric since the simulator doesnot provide away to extract contact durations between nodes. Lookinginto this aspect can yield better more practical heuristics.

• With better simulation results, the system needs to be im-plemented on real hardware to exemplify its real benefits.

The work identifies an area that has not received dueattention before. We have proposed query scheduling andprereplication mechanisms to enable applications to managedata in a flexible manner, in a DTN setting. The simulationresults prove the validity of the approaches, followed in thepaper. We strongly believe that such a distributed solution,will really foster the growth of killer applications that canpopularize the notion of DTNs.

XI. REFERENCES

[1] Kevin Fall, Delay Tolerant Network Architecture forChallenged Internets, Sigcomm 03, August 25-29[2] Xuwen Yu and Surendar Chandra, Delay tolerantcollaborations among campus-wide wireless users, IEEEInfocom 2008[3]Aruna Balasubramanian, Brian Neil Levine and ArunVenkataramani, DTN Routing as a Resource AllocationProblem, Sigcomm 07, August 27-31[4] Janico Greifenberg and Dirk Kutscher, EfficientPublish/Subscribe-based Multicast for OpportunisticNetworking with Self Organized Resource Utilization,IEEE 2008[5] Donald Kossmann,The State of the Art in DistributedQuery Processing, ACM 2001

[6] Hassan Artail, Haidar Safa and Manal Shihab,Implementation of a Federated Database on Bluetooth-Enabled Mobile Devices, ICPS 2008, July 6-10, ACM 2008[7] S. Jain, K Fall, R Patra, Routing in delay tolerant network,ACM SIGCOMM 2004, pg 145-158[8] Yang Zhang, Bret Hull, Hari Balakrishnan and SamuelMadden,ICEDB: Intermittently connected continuous queryprocessing[9] Stewart GreenHill, Svetha Venkatesh, Distributed queryprocessing for mobile surveillance, Proceedings of the 15thinternational conference on Multimedia, 2007[10] J. Burgess, B. Gallagher, D. Jensen, and B. N. Levine,MaxProp: Routing for Vehicle-Based Disruption-TolerantNetworks. In Proc. IEEE Infocom, April 2006[11] Philo Juang and Hidekazu Oki and Yong Wang andMargaret Martonosi and Li-shiuan Peh and Daniel Rubenstein,Energy-Efficient Computing for Wildlife Tracking: DesignTradeoffs and Early Experiences with ZebraNet,ASPLOS-Xconference,San Jose,Oct 2002[12]Pentland, A.; Fletcher, R.; Hasson, A., ”DakNet:rethinking connectivity in developing nations,” Computer ,vol.37, no.1, pp. 78-83, Jan. 2004[13] UMass DieselNet Project:http://prisms.cs.umass.edu/dome/umassdieselnet[14] The ONE Simulator:http://www.netlab.tkk.fi/tutkimus/dtn/theone/