Saving Space and Time Using Index Merging

Lubomir Stancheva,, Grant Weddellb

aComputer Science Department, Indiana University - Purdue University Fort Wayne,USA

bDavid R. Cheriton School of Computer Science, University of Waterloo, Canada

Abstract

Managing digital information is an integral part of our society. Efficient ac-cess to data is supported through the use of indices. Although indices canreduce the cost of answering queries, they have two significant drawbacks:they take additional storage space and their maintenance can become a bot-tleneck. We address these challenges by introducing search data structuresthat reduce the need for storing redundant data among indices. Our ex-perimental results with the main-memory version of these data structuresshow that our approach can reduce by half the storage space and can im-prove performance, where the highest performance improvement is achievedfor workloads with high update ratios. Our experimental results with thesecondary-storage version of the data structures shows that our approachproduces a solution that can outperform both IBM DB2 and Microsoft SQLServer on the popular TPC-C workload.

Key words: Index Structures, Data Storage Representations, IndexMerging

Email addresses: stanchel@ipfw.edu (Lubomir Stanchev),gweddell@uwaterloo.ca (Grant Weddell)

Preprint submitted to Data and Knowledge Engineering August 4, 2010

1. Introduction

Efficient access to vast amount of data, whether stored on secondary diskor kept in main memory, is possible through the use of cleverly designed aux-iliary data structures, such as indexes, hash tables, and materialized views.Although these data structures can improve access time exponentially, theyrequire additional storage space and additional maintenance overhead. Thispaper shows how index structures can be merged by reducing the redundantdata among them. This results not only in savings in storage space, but alsoimproved performance due to reduced maintenance overhead.

Consider a workload of ten queries defined over the same table. An indexadvisor, such as the one available in IBM DB2 ([16]), Microsoft SQL Server([2]), and Oracle ([10]), can suggest creating an index for each query in orderto improve performance. This will result in substantial increase of storageoverhead. Moreover, every update to the table needs to be accompaniedwith updates to each of the ten indices, which can become a performancebottleneck. Conversely, if some of the suggested indices are merged in a waythat reduces redundant data among them, then less storage space will beneeded and updates will be faster because fewer copies of the same data willhave to be refreshed.

Index merging is a difficult problem and most commercial database man-agement systems (DBMSs) provide only limited support ([5]). The reason isthat, except for the most trivial cases, it is impossible to perform index merg-ing in a way that preserves the set of queries that can be answered efficientlywithout creating instances of data structures that are more evolved than atraditional index. For example, consider the indices X1 = 〈R1, 〈A asc〉〉 1,X2 = 〈R2, 〈A asc〉〉, and X3 = 〈R3, 〈A asc〉〉. Moreover, suppose that allthree tables have the same set of attributes and R2 and R3 are disjoint mate-rialized views that contain a subset of the elements of R1. One may identifythe indices X2 and X3 as redundant because they contain a subset of thedata that is already stored in X1. However, removing the two indices willprevent us from efficiently answering the query “select * from R2 where

A > 5” because index X1 cannot be used to efficiently enumerate the ele-ments of R2. In this paper we show how the three indices can be merged intoa single extended index2 on the elements of R1 that can efficiently answer all

1This denotes an index on the table R ordered by the attribute A is ascending order.2An extended index is a tree index that has additional capabilities that allow for data

2

the queries that the initial tree indices can answer.Different index structures have been known for more than forty years. For

example, AVL trees were first introduced in 1962 (see [1]), while B+ treeswere introduced in 1972 (see [4]). However, few studies have considered thepossibility of merging indices in order to eliminate redundant data. Two ofthe few exceptions are [9] and [5], which consider merging indices wheneverthey have attributes in common. However, unlike our approach, the papers’approach can lead to exponential degradation in query performance. Thereason is that a query that has a worst-case logarithmic time-bound againstone of the indices to be merged can have linear worst-case complexity againstthe merged index. This happens when the merged index no longer efficientlysupports one or more of the initial queries.

In the paper we examine how indices can be merged in a way that doesnot affect the space of queries that can be efficiently answered. The goal is topreserve the initial intention of the index advisor about the queries that needto run efficiently. This restriction results in less index merging opportunitiescompared to the approach taken in [9, 5]. However, our experimental evalua-tion shows that our approach can work well in practice, resulting in a signifi-cant reduction of storage cost and improved performance for workloads withwith significant update ratio (e.g., above 10% updates). The main-memoryexperimental results in the paper are based on Arne Andersson trees (or AAtrees - see [3]), while the secondary-storage experiments use the B+ treeimplementation of the commercial system that is being compared.

1.1. Our Approach

We adopt the model where the physical design advisor of a DBMS pro-duces a set of parameterized simple SQL queries (or sSQL queries for short- see Table 3 for a formal definition) over existing base tables and newlyrecommended materialized views. Such an output can be produced, for ex-ample, by examining the operation tree of the queries in the workload andidentifying leaf subtrees for which the benefit of creating an index and/ora materialized view outweighs the cost of maintenance (see [12] for details).Alternatively, if the what-if query optimizer model presented in [2] and [16]is applied, then the what-if query optimizer can easily estimate the cost ofa SQL query based on information about the cost of sSQL queries. Finally,

compactness, where the precise definition will be presented in Section 3.

3

(name) (query)

Q1

select *

from Employeewhere SSN = :P

Q2

select *

from Departmentwhere depName = :P

Table 1: Breakup sSQL queries

even if the physical design tuning is done online without user input (see [6]),the same set of requirements can be identified.

In order to demonstrate how sSQL queries are created, consider the fol-lowing query and assume that every employee works for a single department.

select *from Employee e, Department dwhere e.depName = d.name and e.SSN = :P

In the query, :P is used to denote a parameter, SSN is a key for the tableEmployee, and depName is a key for the table Department. When construct-ing a query plan for answering the query, the query engine can determinethat the queries shown in Table 1 are needed to efficiently support the initialquery. In this case, our system will take the sSQL queries Q1 and Q2 asinput. Using these sSQL queries, the original query can be answered usingthe following query plan.

Employee e = Q1(:P );Department d = Q2(e.depName);return join(e, d);

We assume that the query optimization that selects the sSQL queries hap-pens outside our system and is done by an external tool. Note that the inputto our system includes sSQL queries rather than indices because sSQL queriescarry more detailed information. For example, the index 〈R, 〈A asc, B asc〉〉does not tell us whether the order of the attributes can be swapped withoutsacrificing the logarithmic time-bound for object retrieval. In particular, theorder of the attributes is not important if the sSQL query “select * from

R where A = :P1 and B = :P2” generated the index, but it is important

4

Selector

sSQL queries

Index Merger extended indices

workload: SQL queries and updates schema information and

statistics

Index and Materialized View

materialized

views

Figure 1: Physical design advisor architecture

if the SQL query “select * from R order by A asc, B asc” generatedthe index.

The core of the paper is the Index Merger module shown in Figure 1. Ittakes as input sSQL queries defined over base tables and newly introducedmaterialized views, schema information, and statistics on the current stateof the database. The result of the module is a set of extended indices and amapping between each input sSQL query and the extended index that canefficiently answer it. Statistical information is used to estimate the size of anextended index.

The first step of our algorithm is to create a Parameterized Access Re-quirement Type (PART) for each input sSQL query, where a PART representsa set of extended indices. In particular, for an input sSQL query Q, we willcreate the PART P that corresponds to a set of extended indices that canefficiently answer Q by doing a single index scan, where we require that theextended indices are minimal in the sense that they cannot be simplified andstill effetely answer the query Q. The second step is to merge PARTs thatrepresent indices that have a non-empty intersection. The merging has theproperty that the initial queries can be efficiently answered by each extendedindex that is represented by the created PART. The final step in both algo-rithms is to create an extended index for each of PARTs that is constructedin the previous step, where the extended indices that are anticipated to beof the smallest size given the available statistics are selected.

5

grade

{disjoint,complete}

{disjoint,complete}

Person

name

Customer

Manager

TraineeEmployee

salary

Worker

completionLevelbalance

wageGrade

Figure 2: Example database schema

(name) (query)

Q1

select *

from Personorder by name asc

Q2

select *

from Customerwhere name = :P1

order by balance asc

Q3

select *

from Traineewhere completionLevel = 1order by name asc, grade asc

Table 2: Three example queries

1.2. Motivating Example

We next demonstrate our approach on the UML class diagram shown inFigure 2. It depicts a typical company that has customers, employees, andtrainees that are disjoint and every employee is a manager or a worker, butnot both. We have picked a UML class diagram rather than a relationalschema as out data schema in order to simplify the presentation. However,the algorithm works on any relational or object-oriented database schema.For example, the relational database that corresponds to the example schemacan contain the materialized views Customer, Employee, and Trainee definedover the base table Person and the materialized views Manager and Workerdefined over the materialized view Employee. If the proper integrity con-straints are specified, then our algorithm can be applied on this modifiedrelational database schema. It is also the case that inheritance between

6

classes (or set containment between tables in the relational case) is not re-quired in order to reap the full benefit of our algorithm. For example, itmay be the case that the initially data schema can contains only the tablePerson, where the other entities are generated as the result of the queries inthe workload (for example, a query that asks about persons of type traineecould have generated the materialized view Trainee).

Suppose that the workload consists of the example queries shown in Ta-ble 2. Note that queries Q1 and Q2 follow the sSQL syntax, while query Q3

can be rewritten as a sSQL query over the materialized view VT with thefollowing underlying query.

select *from Traineewhere completionLevel = 1

We next describe an extended index that can efficiently answer all threequeries. The search tree for the index will contain nodes that contain pointersto Person objects (that is, one can perceive this index as a secondary index onthe Person table). Since the class Person contains objects of four differenttypes (that is, Customer, Manager, Worker, and Trainee), the index willcontain pointers to objects of four different types, which demonstrates thepolymorphic property of an extended index.

The nodes in the search tree of the index are first ordered relative to theattribute name in ascending order and next relative to the derived attributeA, where A = 0 for Customer objects, A = 1 for Employee objects, andA = 2 for Trainee objects. Next, the ordering depends on the value for theattribute A. Nodes that point to objects that have the same value for thename attribute and for which A = 0 are ordered relative to the attributebalance in ascending order. Alternatively, nodes that point to objects thathave the same value for the name attribute and for which A = 2 are orderedrelative to the attribute grade in ascending order. The presented orderingdemonstrates the branching order property of an extended index.

Lastly, we are going to add a marker bit to each node of the search treeof the index. The bit of a node will be set exactly when the node or oneof its descendent nodes contains a pointer to a Trainee object for whichcompletionLevel = 1.

The extended index that was described can be used to efficiently answerquery Q1 from Table 2 by performing an in-order traversal of the search tree.

7

This will result in computing the correct query result because the nodes ofthe index are ordered relative to the attribute name.

Query Q2 from Table 2 can be efficiently answered by first finding theleft-most node in the search tree that points to a Customer object with thegiven name. This can be done efficiently because nodes for people with thesame name are ordered relative to their type. The query result consists of theobjects pointed to by sequential nodes in the search tree starting with thefound node, where the terminating condition is reaching a node that pointsto an object that is not a Customer or that has a name that is different fromthe specified name.

Query Q3 can be efficiently answered by performing an in-order traversalof the marked nodes of the search tree. In particular, any subtree with aroot node that is not marked can be pruned-out because such a subtreecannot contain a pointer to a Trainee object for which completionLevel = 1.Conversely, any subtree that has a root node that is marked will contain apointer to an object from the query result and therefore needs to be examined.Note that the resulting objects will be in the correct order because the nodesin the search tree that point to Trainee objects are first ordered relative toname in ascending order and then relative to grade in ascending order.

1.3. Paper Outline and Contributions

Chapter 2 presented related research, while Chapter 3 outlines relevantdefinitions. The most significant contributions of the paper are presented inthe next three chapters.

1. We present the notion of an extended index, which is a novel datastructure that can contain data from several indices in a way thatreduces redundancies - see Section 4.

2. We show how extended index can be merged - see Section 5.

3. We present experimental results that show how our approach to in-dex merging can decrease storage overhead and speedup updates - seeSection 6.

Chapter 7 summarizes our results and outlines directions for future research.

8

2. Related Research

Two papers that address the problem of index merging are [9] and [5].Given the indices 〈R, 〈A〉〉 and 〈R, 〈B〉〉, the algorithms in both papers candecide to merge them into the index 〈R, 〈A,B〉〉. This differs from our ap-proach in two ways: (1) indices rather than sSQL queries are part of theinput and (2) the exponential time capabilities of the initial indices are notpreserved. For example, if all the objects in R have different values for A,then the new index can take linear time to answer the query “select * from

R where B = :P”. However, this query can be efficiently answered by thesecond input index. The reason the papers’ approach cannot overcome thisshortcoming is because they do not explore index merging techniques thatgenerates index structures that are different from traditional indices. We,on the other hand, explore extended indices, which have the polymorphic,branching order, and marker bit properties. To summarize, our approach isorthogonal to the approach presented in the two papers and can be appliedin combination with them.

Merging artifacts and the order of performing the merging procedure hasbeen considered for different applications. For example, [8] examines howimage regions can be merged and how the order of merging can affect thefinal result. However, their approach is not applicable here because indicesare simpler artifacts than images and more precise merging techniques canbe developed.

Other ways to reduce index maintenance cost have been explored. Forexample, [11] explains how to reduce index maintenance cost by using adap-tive indexing. Adaptive indexing allows updates to be propagated fast, butindex retrieval produces “candidate” and not “certain” query results. An-other proposal is [13]. It is one of a sequence of papers on cracked databases,where updates are applied just before the required data is queried. We be-lieve that this work is orthogonal to our proposal. An interesting approachis described in [6]. It shows how physical design selection can be done onthe fly while the database is queried and updated rather than being invokedmanually. Again, we believe that this work complements this paper.

It is important to note that this paper does not address the problemof automatic physical design (see [2, 5, 7]). Our approach assumes thataccess requirements (described as sSQL queries and materialized views) havealready been determined before our algorithm is applied.

9

3. Definitions

In this section we describe the database schema type, the query language,and the problem that we are solving. Note that we fix the database schematype in order to increase the readability of the paper, where our algorithmcan be applied to both a relational and object-oriented database schema.

3.1. Database Schema

Throughout the paper we will use the unqualified term table to refer toboth base tables and materialized views, where a base table defines the setof objects that are instances of a particular class. We will use T to denotea base table, V to denote a materialized view, R to denote a table, and Σto denote a database schema. We use the letters A and B to refer to tableattributes and attr(R) to refer to the attributes of the table R.

We require that every table has the system attribute ID that uniquelyidentifies a database object. The non-system attributes of a table are eithernon-reference and are of one of the predefined types (e.g., integer, string,etc.), or are reference and store the ID of an object in the database. Werequire that all reference attributes are not null and refer to an existingobject, that is, we impose a foreign key constraint on reference attributes.

We will refer to a materialized view that is defined using a query of thefollowing type as a simple materialized view.

select A1, . . . , Aafrom T as twhere γ(t)

In the above query γ is used to denote an efficient predicate, where theprecise definition follows.

Definition 3.1 (efficient predicate). An efficient predicate γ over a tableR has the property that it can be decided in O(|def(γ)| · |t|) time whether thepredicate holds for an object t ∈ R.

Note that throughout the paper we use | · | to denote the size of theenclosed component and |def (·)| to denote the size of its definition. Thepredicate (t.name =“John” and t.salary > 200000) is an example of anefficient predicate. It can be used to define a simple materialized view withthe following underlying query.

10

select ∗from Manager as twhere t.name = “John” and t.salary > 200000

Simple materialized views are important because they reuse the ID at-tribute of the underlying tables over which they are defined. For example,in the above materialized view managers named John that make more thantwo hundred thousand dollars can be identified as such (e.g, by connectingthem in a linked list or creating an index on them) without the need to createadditional records for them.

3.2. The Query Language

We assume that the the input queries to the Index Merger module (seeFigure 1) are sSQL queries – see Table 3. In the table we have used [·] to de-note an optional component and dir to denote asc or desc. The restrictionsfor sSQL queries prevents ordering on reference attributes. This is reasonablebecause the value of a reference attribute depends on the the internal imple-mentation of the system and should not be relied on by external users. Therestriction also enforces partial-match attributes to be non-ID. A query inwhich one of the partial-match attributes is an ID attribute can be answeredby executing a query of the third type followed by a predicate check on theresulting object. We have chosen this sSQL syntax because it restricts inputqueries to single table queries that can be efficiently answered using a singleindex. Adhering to the SQL standard, we will use “select *” to denoteselecting all the attributes of a table.

3.3. The Problem

We next define the characteristics of an efficient query plan.

Definition 3.2 (efficient plan for a query). Consider a SQL query Q andthe corresponding access plan QP . Assume that the size to encode a valuefor each of the attributes of the database schema is constant. Then the queryplan QP is efficient exactly when it returns each object of the query result in

O(|def (Q)| · (m∑i=1

log(|Ri|))) time, where {Ri}mi=1 are the the tables that are

referenced in Q.

11

(type) (query)

(1)

select B1, . . . , Bb

from R[where A1 = :P1 and . . . and Al = :Pl][order by Al+1 dirl+1, . . . , Aa dira]

(2)

select B1, . . . , Bb

from Rwhere A1 = :P1 and . . . and Al = :Pl and Al+1 between :Pl+1 and :Pl+2

[order by Al+1 dirl+1, . . . , Aa dira]

(3)select B1, . . . , Bb

from Rwhere ID = :P1

{Ai}li=1 are non-ID attributes and {Ai}ai=l+1 are non-reference attributes.

Table 3: The three sSQL query types

In the paper we will describe the design of the Index Merger module (seeFigure 1). One possible optimization criteria is that the size of the producedextended indices should be as small as possible subject to the constraintthat each input sSQL query should have an efficient plan based on one ofthe produced extended indices. (The supplied statistical information canbe used to approximate the size of an extended index.) Alternatively, theproblem can be formulated as finding the physical design that fits in theavailable storage space and that can efficiently supports as many of the inputqueries as possible. For conciseness, in the paper we skip the details of thealgorithms for solving the two optimization problems and concentrate on theindex merging procedure.

4. Physical Design Model

We next present the notion of object ordering, followed by the formalsyntax and semantics of an extended index.

Definition 4.1 (object ordering). For a table R, an object ordering isdefined using the syntax 〈R, 〈A1 dir1, . . . , Aa dira〉〉, where A1, . . . , Aa aredistinct attributes of the table R. It denotes an ordering of the objects inthe table R, where the objects are first ordered relative to the value of A1 inascending order if dir1 = asc and in descending order if dir1 = desc, nextrelative to the value of the attribute A2 in direction dir2 and so on.

12

Sometimes, when the table on which an ordering is applied is clear fromthe context, we will skip the table name from the syntax of an object ordering.Also, note that we will use R1 op R2 to denote πID(R1) op πID(R2), whereop ∈ {⊂,⊆,≡,∪,∩} and ≡ is used to denote the set equivalence operator.

Definition 4.2 (syntax of an extended index). An extended index X isrepresented by a pair 〈{γ1, . . . , γm}, Gt〉. We will refer to γ̄ = {γ1, . . . , γm} asthe γ-condition of X and write γ(X). When the γ-condition is missing, thetrivial γ condition that consists of the empty set is assumed. The second ar-gument Gt is a rooted tree with sibling ordering (that is, the children of a par-ent node are ordered) and node labels that are of the form 〈R, 〈A1, . . . , Aa〉〉.We will refer to this tree as the description tree of the index. For a la-bel 〈R, 〈A1, . . . , Aa〉〉, we will refer to R as the table of the node and writetable(n) and to 〈A1, . . . , Aa〉 as the ordering label of the node and writeL(n). We require that the predicates {γi}mi=1 are efficient predicates over thetable R that is the table of the root node of the description tree. We imposethe following additional restrictions on Gt.

1. Let the node n with table R be the parent of the nodes 〈n1, . . . , nk〉 withtables 〈R1, . . . , Rk〉, respectively. Then the following rules should holdfor any instance of the tables:

(1) Ri ⊆ R for 1 ≤ i ≤ k,

(2) Ri ∩Rj = ∅ for 1 ≤ i 6= j ≤ k,

(3)k⋃i=1

Ri = R, and

(4) attr(R) ⊆ attr(Ri) for i = 1 to k.

2. If the node n1 is an ancestor of the node n2, then the ordering labels ofn1 and n2 do not share attributes in common.

For convenience, we introduce several node labeling functions. We willuse label(n) to denote the label of a node n. We also define L↓ recur-sively as follows: for a leaf node n: L↓(n) = label(n) and for a non-leafnode n with label 〈L〉 and ordered children n1, . . . , nk we defined L↓(n) =〈L, [L↓(n1), . . . ,L↓(nk)]〉. Since the string L↓(nr), where nr is the root nodeof tree, completely describes a tree, we will refer to it as the tree’s stringdescription and we will sometimes represent a tree by its string description.

13

We next recursively define the function L↑, which returns the extended la-bel of a node. For the root node of the tree nr, we define L↑(nr) = label(nr).For a non-root node n with label 〈L〉 and parent node n′ with extended label〈L′〉 we defined L↑(n) = 〈L′, L〉. Informally, the extended label of a node isa listing of the labels for the nodes in the path that starts at the root nodeof the tree and ends at the node.

Consider the extended index from our running example created in Sec-tion 1.2. It will have the syntax: 〈{γ},Person, 〈name〉, [〈Customer , 〈balance〉〉,〈Employee, 〈〉〉, 〈Trainee, 〈grade〉〉]〉, where γ(t) is true exactly when t is aTrainee object with completionLevel = 1. (For now, it should be clear thatthis extended index satisfies Definition 4.2, where the meaning of this ex-tended index will become clear after we present the semantics of an extendedindex.)

Definition 4.3 (semantics of an extended index). The extended index〈{γ1, . . . , γm}, Gt〉 is implemented by a search tree. If n1, . . . , nk are the leafnodes in Gt and they have tables 〈R1, . . . , Rk〉, respectively, then the searchtree contains data pointers to the objects of the tables {Ri}ki=1. For each noden of Gt, we next define an ordering function Or, where the elements in thesearch tree will be ordered relative to the order Or(nr) and nr is the root ofGt.

If n is a leaf node and L(n) = 〈A1, . . . , Aa〉, then we define Or(n) =〈A1 asc, . . . , Aa asc〉. If n is a non-leaf node with children 〈n1, . . . , nk〉,table(n) = R, and L(n) = 〈A1, . . . , Aa〉, then we define Or(n) to be thefollowing ordering. (Nodes that are indistinguishable relative to this orderwill be ordered relative the attribute ID of the objects they point to.)

1. The objects are first ordered relative to the object ordering 〈A1 asc, . . . , Aaasc〉.

2. Next, if two or more objects have the same value for the attributes{Ai}ai=1, then they are ordered relative to the attribute A in ascendingorder, where t.A = i if and only if t ∈ table(ni) for 1 ≤ i ≤ k.

3. Finally, if two or more objects have the same value for the attributes{Ai}ai=1 and for the attribute A, then they are ordered relative to Or(ni),where i is the common value for the attribute A.

If the index has a non-trivial γ-condition of the form {γ1, . . . , γm}, thenwe will associate with each node in the search tree m marker bits. The jth

14

marker bit of a node is set exactly when the node or one of its descendantsin the search tree contains a data pointer to an object for which γj holds(1 ≤ j ≤ m).

Figure 3 shows the extended index for our running example. The γ-condition of an extended index contains one predicate for each marker bitthat needs to be created. In the example case, the marker bit allows us to ef-ficiently find Trainee objects for which completionLevel = 1. The descriptiontree of an extended index defines the branching order of the index. In ourexample, people with the same name are ordered relative to their type andfurther different ordering is defined for customers, employees, and trainees.The tables of the leaf nodes in the description tree define the objects that willbe pointed-to by the index. In our example, the extended index will pointto the Customer, Employee, and Trainee objects. The tables for the non-leafnodes are simply defined as the union of the tables of the child nodes. In ourexample, in order for the extended index to be valid, the table Person mustbe the union of the tables Customer, Employee, and Trainee.

< {Trainee(t) and t.completionLevel=1}, <Person,<name>> >

<Customer, <balance>> <Employee,<>> <Trainee, <grade>>

Figure 3: Example extended index

Before defining the interface of an extending index (that is, the methodsthat it can efficiently supports), we introduce several intermediate definitionsand lemmas.

Definition 4.4 (extended γ-condition). The extended γ-condition of anextended index X is {TRUE} ∪ {FALSE} ∪ {

⋃∅6≡γ̄⊆γ(X)

∨γ∈γ̄

γ}. We will write

γe(X) to denote it.

We next present a lemma that explains the meaning of an extended γ-condition.

Lemma 4.5 (meaning of an extended γ-condition). Let X be an ex-tended index. Then γ ∈ γe(X) if and only if the value of the marker bit for

15

γ in every node of the search tree for X, if such a marker bit hypotheticallyexisted, can be computed as a function of the values of the other marker bitsfor that node.

Proof: See [14].As an example, if an extended index contains the γ-conditions Trainee(t)

and t.completionLevel = 1, then the γ-condition Trainee(t) ∨ t.completion-Level = 1 will be in the extended γ-condition of the index. The reason isthat if a node in the search tree has one of the marker bits for the first twoγ-conditions set, then the marker bit for the third γ-condition will also beset and therefore the third marker bit does not need to be stored explicitly.

Definition 4.6 (rooted path in a tree). A rooted path in a tree is anypath in the tree that starts at the root of the tree. We will denote by rp(Gt)the set of all rooted paths in the tree Gt.

Definition 4.7 (clustering property). Let K = 〈n1, . . . , nk〉 be a rootedpath in the description tree of the extended index X and let the attributes inL↑(nk) be 〈A1, . . . , Ab〉 in this order. Given an integer a, 1 ≤ a ≤ b, we willsay that K is clustered relative to the integer a if and only if at least one ofthe following conditions holds.

1. k = 1.

2. If nr is the node in K with the biggest subscript for which L↑(nr) con-tains exclusively attributes from the set {Ai}ai=1 in its ordering label,then either r = k or r = k − 1.

The following lemma explains why the above property is called the clus-tering property.

Lemma 4.8 (meaning of the clustering property). Let X be an exten-ded index and K = 〈n1, . . . , nk〉 be a rooted path in description tree of theextended index. Let table(nk) = Rk and 〈A1, . . . , Ab〉 be the attributes inL↑(nk) in order of appearance. Then the Rk objects that have the same valuefor the attributes {Ai}ai=1 (a ≤ b) are sequential in an in-order traversal ofthe index (that is, clustered together) if and only if K is clustered relative tothe integer a.

16

Proof: See [14].In the description tree of Figure 3, consider the left most rooted path that

goes through the Person and Customer tables. This path is clustered relativeto the integers 1 and 2 because the Customers that have the same value forthe attribute name are clustered together, as are the Customers that have thesame value for the attributes name and balance. Conversely, if the node forthe Customer table had two children nodes with tables wealthyCustomers andaverageCustomers, then the rooted path that reaches the wealthyCustomersnode will not be clustered relative to the integer 1 because wealthy customerswith the same name are going to be separated in groups relative to theirbalance and therefore not clustered together.

Definition 4.9 (interface of an extended index). An extended index Xwith description tree Gt supports the operations that are listed bellow. Forthe last three operations a common pre-condition is the existence of a nodenr in Gt with table Rr. We will refer to the path in Gt that starts at the rootand ends at nr as K = 〈n1, . . . , nr〉. We also assume that the attribute inL↑(nr) are 〈A1, . . . , Ab〉 in this order and a is an integer between 1 and b.Note that the next and the two search methods return NULL when the objectwe are searching for does not exist.

• insert(reference p):

– pre-condition: The object with ID p belong to the table of one ofthe leaf nodes in Gt.

– action: A node that points to this object is inserted in X.

• delete(reference p):

– pre-condition: There exists a node that points to the object withID p in X.

– action: Deletes this node from X.

• reference exact search(table Rr, param P1, . . . , param Pa, directiondira+1, . . . , direction dirb, efficient predicate γ):

– pre-condition: Either K is clustered relative to the integer a andγ ∈ γe(X) or there exists a γ′ ∈ γe(X) that has the property thatγ′(t) = TRUE if and only if t ∈ γ(Rr) for any object t.

17

– return value: Let O be the object ordering 〈Aa+1 dira+1, . . . , Abdirb〉. This method returns the ID of the first object t in X, relativeto the order O, for which the following hold.

(a) t ∈ Rr.

(b) γ(t) holds.

(c)a∧i=1

(t.Ai = Pi).

• reference closest search(table Rr, param P1, . . . , param Pa, direc-tion dira, . . . , direction dirb, efficient predicate γ):

– pre-condition: Either K is clustered relative to the integer a− 1and γ ∈ γe(X) or there exists a γ′ ∈ γe(X) that has the propertythat γ′(t) = TRUE if and only if t ∈ γ(Rr) for any object t.

– return value: Let O be the ordering 〈Aa dira, . . . , Ab dirb〉. Thismethod returns the ID of the first object t in X, relative to the orderO, for which the following hold.

(a) t ∈ Rr.

(b) γ(t) holds.

(c)a−1∧i=1

(t.Ai = Pi).

(d) t.Aa > Pa when dira = asc and t.Aa < Pa when dira = desc.

• reference next(table Rr, param P1, . . . , param Pa, direction dira+1,. . . , direction dirb, reference p, efficient predicateγ):

– pre-condition: There exists an object t ∈ Rr with ID p pointed toby a node in X. Also, either K is clustered relative to the integera and γ ∈ γe(X) or there exists a γ′ ∈ γe(X) that has the propertythat γ′(t) = TRUE if and only if t ∈ γ(Rr) for any object t.

– return value: Let O be the ordering 〈Aa+1 dira+1, . . . , Ab dirb〉.The method returns the ID of the first object t′ ∈ Rr after the

object t, relative to the order O, for which γ(t′) anda∧i=1

t′.Ai = Pi

both hold.

The preconditions in the last three methods guarantee that either theobjects in the query result are clustered together in the index or they can be

18

retrieved efficiently by accessing only the marked nodes. As a consequence,the following theorem holds.

Theorem 4.10 (interface of an extended index). If the sizes of the ob-jects that are indexed in the extended index X with description tree Gt arelimited by some constant, then each method of the interface of X takesO(log(|X|) · |def(X)|) worst-case time.

Proof: See [14].We next present several definitions and a theorem that describe the set of

sSQL that can be efficiently answered using an extended index under certainassumptions.

(type) (query)

(1)

select B1, . . . , Bb

from R[where A1 = :P1 and . . . and Al = :Pl][order by Al+1 dirl+1, . . . , As dirs]

(2)

select B1, . . . , Bb

from Rwhere A1 =: P1 and . . . and Al = :Pl and Al+1 between :Pl+1 and :Pl+2

[order by Al+1 dirl+1, . . . , As dirs]

(3)select B1, . . . , Bb

from Rwhere ID = :P1

1. 〈R1, . . . , Rk〉 are the tables of 〈n1, . . . , nk〉, respectively.

2. R is the base table Rk (in which case γ = TRUE) or a materialized viewwith query “select * from Rk where γ(Rk)”.

3. 〈A1, . . . , As〉 are the first s attributes in this order from L↑(nk).

4. Either K is clustered relative to l and γ ∈ γe(X) or there exists aγ′ ∈ γe(X) that has the property that for an object t, γ′(t) = TRUE ifand only if t ∈ γ(Rk).

Table 4: Critical queries for a rooted path K = 〈n1, . . . , nk〉 of X

19

Definition 4.11 (queries of a rooted path of an extended index). LetX be an extended index. Table 4 shows the set of sSQL queries that we willassociate with a rooted path K of the index. We will denote this set asQK(X).

The above definition describes the set of sSQL queries that can be effi-ciently answered by the part of an extended index that is associated with arooted path of its description tree. The third condition of Table 4 requiresthat the query can refer only the attributes along the path, while the fourthcondition guarantees that the query can be answered efficiently.

Definition 4.12 (critical queries supported by an extended index).Let X be an extended index. We define the set of critical queries efficientlysupported by X to be all queries that have the following properties.

1. They are sSQL queries that reference a table that has one of the fol-lowing properties

a) the table appears in the string description of X,

b) the table is a materialized view that has an underlying query of theform “select * from R where γ(R)”, where γ ∈ γe(X) and Rappears in the string description of X,

c) the table is a union all of disjoint tables (that is, no overlappingobjects with the same ID value) described in (a) and/or (b).

2. They can be efficiently answered using X.

We will refer to this query set as Q(X).

Note that we define R1 union all R2 as all the objects in R1 followedby the objects in R2, where we require that R1 and R2 do not share objectsin common and no constraint is specified on the ordering of the result.

We next define the operator cl(Q̄) that describes the queries that can becreated from the queries Q̄ by merge sorting the results.

Definition 4.13 (cl(Q̄)). cl(Q̄) ≡ {Q|∃{Q1, . . . , Qk} ⊆ Q̄, Qi(D)∩Qj(D) =∅ for any database instance D and 1 ≤ i 6= j ≤ k and Q ≡ merge sort(Q1, . . . ,Qk, O)}, where the method merge sort returns the objects “Q1 union all

. . . union all Qk” in the order O assuming the results of {Qi}ki=1 are sortedin this order.

20

Theorem 4.14 (efficiently supported queries by an extended index).Let X be an extended index. Then cl(

⋃K∈rp(X)

QK(X)) ⊆ Q(X). More-

over, if Q ∈ Q(X), then there exists a query Q′ equivalent to Q and Q′ ∈cl(

⋃K∈rp(X)

QK(X)).

Proof: See [14].The above theorem states that the queries that are efficiently supported

by an extended index are the queries that are associated with the differentrooted paths of the description tree of the index. The cl operator is intro-duced to describe that a query that asks for the union of the results of severalqueries that can be efficiently answered can also be efficiently answered bythe extended index by merging the already sorted results.

5. Index Merging

5.1. Syntax and Semantics of a PART

Our strategy for index merging is to first calculate the set of extendedindices that can be efficiently answered by each input sSQL query. In orderto do so, we need the definition of a Parameterized Access Requirement Type(PART), which describes a set of extended indices.

Definition 5.1 (PART syntax). A PART P is defined by the pair 〈γ̄, GT 〉,where γ̄ is a set of efficient predicates and Gt is a description tree with noorder defined on node siblings, where every node has a label with the followingsyntax.

node label ::= 〈R, 〈L〉〉 | 〈R, 〈〉〉L ::= E | F | E, L | F , LE ::= {A1, . . . , Aa}F ::= A | A,F

In the above grammar, we require that a > 0.

Given a node label 〈R, 〈L〉〉, we will refer to the E parts of L as E-components and to the F parts – as F -components. We will refer to R asthe node’s table and write table(n) to denote it, to L as the node’s orderinglabel and write L(n) to denote it. Similarly, we will refer to γ̄ as the PART’sγ-condition and write γ(P) to denote it. We will use |L| to denote the numberof attributes referenced in an ordering label.

21

The meaning of an E-component is that the order of the attributes is notimportant. For example, in the query : “select * from R where A= :P1

and B= :P2, the attributes in the resulting index can be ordered relative toA, B or relative to B, A. Conversely, the order of the attributes that definethe ordering of the index that efficiently support the query “select * from

R order by A, B” is important.Informally, a PART represents a set of extended indices that can be cre-

ated from it by fixing the order of the attributes in the E components, remov-ing the curly brackets in the ordering labels of the nodes, fixing the siblingorder in the tree, and adding additional nodes when needed to satisfy con-dition 1.3 of Definition 4.2. Note that the curly brackets are used to denotethat the ordering of the attributes inside them is not fixed. In order to for-mally define the semantics of a PART, we present the following intermediatedefinitions.

Definition 5.2 (permutations for a PART ordering label). Let L bean ordering label of a PART node. We will use Π(L) to denote the result ofthe nondeterministic procedure of permutating the attributes in some of theE-components and then splitting the e-components. We will call Π a completepermutation if it converts L into an expression in which all E-componentsare of size at most 1. We will use Πc to denote a complete permutation.

For example, if “L = A,B, {C,D,E}”, then “A,B, {C,E}, {D}” is onepossible value for Π(L), while “A,B, {E}, {C}, {D}” is an example result ofa complete permutation.

Definition 5.3 (permutation for a PART). A permutation Π for a PARTP permutates the E-components of some of the ordering labels of the PART.A complete permutation Πc applies complete permutations to all orderinglabels.

Definition 5.4 (fixed PART). A fixed PART has no E-components of sizegreater than one in its ordering labels.

Definition 5.5 (fixed PART ⇒ extended index). Let P be a fixed PART.This PART can be used to create an extended index with the same γ-condition.In order to construct the description tree of the index, apply the followingnondeterministic procedure.

22

1. Fix the order of the node siblings.

2. Convert all ordering labels to F -components by removing the curlybrackets around the E-components.

3. If {ni}ki=1 are the children of the node n andk⋃i=1

table(ni) ⊂ table(n),

then add a new child node to n with table R that includes the objectsthat are in table(n) but not in table(ni) for i = 1 to k. Include emptyordering label for the new node.

Definition 5.6 (valid PART). A PART P is valid if and only if for ev-ery complete permutation Πc of P, Πc(P) is converted into a valid extendedindex by any application of the nondeterministic procedure described in Def-inition 5.5.

Definition 5.7 (cover of a set of indices). Let X̄ be a set of extended in-dices. Then cover(X̄) = {X|∃X ′ ∈ X̄,Q(X ′) ⊆ Q(X)}.

Informally, a cover of a set of indices is the set of all indices that canbe used to efficiently answer the sSQL queries that the initial extended canefficiently answer.

Definition 5.8 (semantics of a fixed PART). Let P be a fixed PART.Then we will use X(P) to denote the set of extended indices that P can beconverted into using the nondeterministic procedure from Definition 5.5. Wewill also use Xc(P) to denote the set cover(X(P)).

Definition 5.9 (semantics of a PART). Let P be a PART. Then we willuse X(P) to denote the set of extended indices

⋃Πc

X(Πc(P)), where Πc varies

over all valid complete permutations for P. We will also use Xc(P) to denotethe set cover(X(P)) and refer to is the set of extended indices representedby the PART.

Going back to our motivating example from Section 1.2, the queries fromTable 2 will generate the PARTs shown in Table 5. Note that all PARTs inTable 5 are fixed PARTs, where P = 〈Customer , 〈{name, balance}〉〉 is anexample of a PART that is not fixed and that will be generated by a querythat is equivalent to the following query.

23

(name) (query) (PART)

Q1

select *

from Personorder by name

P1 = 〈Person, 〈name〉〉

Q2

select *

from Customerwhere name = :P1

order by balance

P2 = 〈Customer , 〈{name}, balance〉〉

Q3

select *

from Traineewhere completionLevel = 1order by name asc, grade asc

P3 = 〈VT , 〈name, grade〉〉

Table 5: Three example critical queries and the corresponding PARTs

select *from Customerwhere name = :P1 and balance = :P2

As Definition 5.3 suggests, a complete permutation will convert P in eitherthe fixed PART 〈Customer , 〈name, balance〉〉 or the fixed PART 〈Customer ,〈balance, name〉〉.

The merging step will merge the PARTs from Table 5 into the PART P =〈{γ},Person, 〈name〉, [〈Customer , 〈balance〉〉, 〈VT , 〈grade〉〉]〉, where γ(t) holdsfor a Person object if and only if t is also an object in VT (that is, t is a Traineeobject for which completionLevel = 1).

The final step of our algorithm is converting the created PART into anextended index. The extended index X = 〈{γ},Person, 〈name〉, [〈Customer ,〈balance〉〉, 〈VT , 〈grade〉〉, 〈VR, 〈〉〉]〉 is one possible index, where VR containsthe Person objects that are not in the tables Customer and VT .

We next formally describe the PART creation and merging steps of ouralgorithm.

5.2. Step 1 - Converting Extended Indices into PARTs

Table 6 shows the PARTs that will be produced for each type of sSQLquery (see Table 3). Note that for a sSQL query of Type 3 we do not needto create a PART because we assume the capability of efficiently retrievingan object from its ID (that is, the ID can be the address of the object in the

24

(query type) (PART)(1) 〈R, 〈{A1, . . . , Al}, 〈Al+1, . . . , Aa〉〉〉(2) 〈R, 〈{A1, . . . , Al}, 〈Al+1, . . . , Aa〉〉〉(3)

Table 6: The PARTs for the three sSQL query types

main-memory case or a mapping hash table can be created in the secondary-storage case).

We next present a theorem that states that the created PARTs indeedrepresent the set of extended indices that can be used to efficiently answerthe original queries.

Theorem 5.10 (correctness of PART creation). If the mapping from Ta-ble 6 is applied on the sSQL query Q to generate the PART P, then Xc(P)contains exactly the set of extended indices that can be used to efficientlyanswer Q.

Proof: See [14].

5.3. Step 2 – PART Merging

In this section, we present a procedure for merging PARTs (that is, setsof extended indices). In order for a set of PARTs to be mergeable, it mustbe the case that the intersection of the extended indices that the PARTsrepresent is not empty, that is, the PART merging will be beneficial.

5.3.1. Merging Ordering Labels

A procedure for merging PARTs requires a way for merging the orderinglabels in their description trees. We next present an algorithm that does thisand we use ⊕ to refer to this operation. Informally, the operation succeedswhen one of the ordering labels is a prefix of the other under some permu-tation. The result of the merge is an ordering label that is a permutation ofboth input ordering labels. A formal definition of the ⊕ operation follows,where ◦ is the string concatenation operation and b·c is the operation thatremoves the curly brackets from empty E-components.

25

L1⊕L2 =

L2 if L1 = ε;

L1 if L2 = ε;

E1 ◦ (L′1 ⊕ (bE2 − E1c ◦ L′2)) if L1 = E1 ◦ L′1, L2 = E2 ◦ L′2, E1 ⊆ E2;

E2 ◦ ((bE1 − E2c ◦ L′1)⊕ L′2) if L1 = E1 ◦ L′1, L2 = E2 ◦ L′2, E2 ⊆ E1;

A ◦ ((bE1 − {A}c ◦ L′1)⊕ L′2) if L1 = E1 ◦ L′1, L2 = A ◦ L′2, A ∈ E1;

A ◦ (L′1 ⊕ (bE2 − {A}c ◦ L′2)) if L1 = A ◦ L′1, L2 = E2 ◦ L′2, A ∈ E2;

A ◦ (L′1 ⊕ L′2) if L1 = A ◦ L′1 and L2 = A ◦ L′2;

UNDEFINED otherwise.

Note that in the above pseudo-code we have used E to denote a non-emptyset, that is, a string of type {. . .} and A to denote a single attribute. Also, wehave used E1−E2 to denote the string that corresponds to the set difference ofthe two sets of attributes. Note as well that the “⊕” function is partial, thatis, not every two ordering labels are mergeable. Since “⊕” is commutative

and associative, we will usek⊕i=1Li to denote L1⊕ (L2⊕ (. . . (Lk−1⊕Lk) . . .)).

We next present an intermediate definition and a theorem that describes theproperties of the ⊕ operation.

Definition 5.11 (compatible attribute orderings). Let {Li}ki=1 be k or-dering labels that are F -components. We will say that the attribute orderingsdefined by {Li}ki=1 are compatible if and only if for all 1 ≤ i, j ≤ k either Liis a prefix of Lj or Lj is a prefix of Li or Li and Lj are the same.

Theorem 5.12 (correctness of the ⊕ operation). 1. Let {Li}ki=1 be

k ordering labels. Ifk⊕i=1Li returns Lk+1 6= UNDEFINED, then:

a) Let {Πci}ki=1 be complete permutations that convert {Li}ki=1 into

the F -components {L′i}ki=1. If the attribute orderings defined by{L′i}ki=1 are compatible, then there exists a complete permutationΠck+1 that converts Lk+1 into L′k+1 and the attribute orderings de-

fined by {L′i}k+1i=1 are compatible.

b) Let Πck+1 be a complete permutation for Lk+1 and L′k+1 = Πc

k+1(Lk+1).Then there exist complete permutations {Πc

i}ki=1 that convert {Li}ki=1

into the F -components {L′i}ki=1, where the attribute orderings de-fined by {L′i}k+1

i=1 are compatible.

26

2. Ifk⊕i=1Li returns UNDEFINED, then there do not exist complete permuta-

tions {Πci}ki=1 that convert {Li}ki=1 into {L′i}ki=1, respectively, such that

the attribute orderings defined by {L′i}ki=1 are compatible.

Proof: See [14].For example, the result of merging the ordering labels “{name, balance}”

and “balance” will be “balance, name”. In the first ordering label the orderof the attributes is not fixed, while the second ordering label forces balanceto come first.

5.3.2. Merging PARTs

We define a simple PART as follows.

Definition 5.13 (simple PART). A PART is simple if it has the formatshown in Table 6.

From the definition it directly follows that Step 1 of our algorithm pro-duces only simple PARTs. We next formally define when two PARTs aremergeable.

Definition 5.14 (PART merging). We will say that the PARTs {Pi}ki=1

are mergeable into the PART P if and only if the following conditions hold.

1.k⋂i=1

Xc(Pi) 6= ∅ and Xc(P) =k⋂i=1

Xc(Pi).

2. If {Ri}ki=1 are the tables of the root nodes of the description trees of{Pi}ki=1, then for every i, 1 ≤ i ≤ k there exists j, 1 ≤ j ≤ k (j 6= i)such that Ri ∩Rj 6= ∅.

The first rule in the above definition guarantees that the new PART willrepresent exactly the indices that are common to all of the original PARTs.As expected, PARTs that do not share indices in common are not mergeable.The second rule guarantees that only PARTs that represent indices withcommon data will be merged.

We next present two methods: table PART merge and gamma PART merge

that perform two different kind of PART mergings. Both method merge anarbitrary PART with a simple PART. The first method adds a new node tothe first PART, while the second method does not. [14] shows that the twoPART merging methods are sound and complete.

27

The method table PART merge(P1,P2) merges an arbitrary PART P1

with a simple PART P2 - see Algorithm 1. The method adds a new nodewith table R to P1 when possible in order to create the resulting PART andreturns UNDEFINED otherwise. If the node insertion can be done in such away so that the label of the inserted node is empty, then the method alsoreturns UNDEFINED (this is the case at Line 9 of the pseudo-code). Thereason is that the created PART will be equivalent to the PART without theinserted node in which a γ-condition is added to the PART, and thereforethe gamma PART merge method can be applied.

Algorithm 1 table PART merge(PART P1, PART P2)

Require: the path 〈n1 = 〈R1, 〈L1〉, . . . nk = 〈Rk, 〈Lk〉〉 (denoted as K) isa rooted path in the description tree of P1,γ̄1 is the γ-condition of P1

P2 = 〈R, 〈L〉〉, R ⊂ Rk, and R′ ∩R = ∅ for every table R′ that is a tableof a child node of nk in the description tree of P1

1: L′ ← L1 ◦ · · · ◦ Lk2: if L⊕ L′ = UNDEFINED then3: return UNDEFINED

4: end if5: L′′ ← L⊕ L′6: γ′ ← t ∈ R7: compute {L′i}ki=1 and Lk+1 subject to L′1 ◦ · · · ◦ L′k ◦ Lk+1 = L′′ and|L′i| = |Li|

8: if |L| ≤ |L′| then9: return UNDEFINED

10: end if11: compute L1 and L2 subject to L1 ◦ L2 = L and |L1| = |L′|12: if is E component(L1) or |L1| = 0 then13: γ ← TRUE

14: else15: γ = γ′

16: end if17: return substitute(P1, γ̄1 ∪ γ, n1 = 〈R1, 〈L′1〉〉, n2 = 〈R2, 〈L′2〉〉, . . . ,

nk = 〈Rk, 〈L′k〉, [〈R, 〈Lk+1〉〉]〉);

The method substitute(P , γ̄, n1 = . . . , nk = . . .) returns the result ofsubstituting the γ-condition in P with γ̄ and the node ni in P with the valuethat is specified (i = 1 to k). In order for this method to be well defined, it

28

must be the case that for i = 1 to k either ni is a leaf node in the descriptiontree of P or the tree for Pi is in the form of a directed path. The methodis E component(L) returns TRUE exactly when L is an E-component.

The method table PART merge adds a new node to P1 with table R.When the conditions in Line 12 of the pseudo-code is true, then the rootedpath in the new PART that ends at the inserted node, which we will denoteas K ′, is clustered relative to the integer l, where l is the position of the lastattribute in L that is part of L1. In this case, a γ-condition does not need tobe added to the new PART because the extended indices that it representswill be able to efficiently answer the query that generated P2 without theγ-condition (see Theorems 4.14 and 5.9). In the other case, we will add theappropriate γ-condition. The labels along the path K ′ are also modified toreflect the result of applying the “⊕” operation between the ordering labelsof K and the ordering label L.

The pseudo-code for the method gamma PART merge is shown in Algo-rithms 2 and 3. It merges an arbitrary PART P1 with a simple PART P2.The method tries to do so without introducing new nodes in P1 and with-out changing the tables of the nodes in P1. When this cannot be done, themethod returns UNDEFINED. Note that P is a PART, R̄ is a set of tables (ini-tially the empty set), and γ′ is an efficient predicate, where all three variablesare global variables for both methods.

The presented pseudo-code covers two cases when the merging of the twoPARTs will be successful.

1. The table for P2, which we refer to as R, is a non-strict subset ofa table of a leaf node n′ in P1 and the ordering label of the singlenode in P2, which we refer to as L, is compatible with L↑(n′). In theresulting PART the ordering labels along the complete path that endsat n′, which we will refer to as K, will be changed to L⊕ L↑(n′). Thepredicate γ′ will be added to the γ-condition of the resulting PARTexactly when either the table for n′ is different than the table R or thetwo tables are the same but K is not clustered relative to the size ofthe single E-component in P2.

2. R is a non-strict subset of the tables of several leaf nodes of P2. Anal-ogous to the previous case, the predicate γ′ will be added to the γ-condition of the resulting PART when R is a strict subset of the tablesR̄ or when the clustering property is not satisfied for one of the rele-

29

Algorithm 2 gamma PART merge(PART P1, PART P2)

Require: n1 = 〈R1, 〈L1〉〉 is the root node of the description tree of P1,γ̄1 is the γ-condition of P1, and P2 is the PART 〈R, 〈L〉〉, where L ={A1, . . . , Al}, Al+1, . . . , Aa

1: mergeable ← true2: γ′ ← (t ∈ R)3: if R 6⊆ R1 then4: return UNDEFINED5: end if6: if L⊕ L1 = UNDEFINED then7: return UNDEFINED8: end if9: L′ ← L⊕ L1

10: if ((is leaf(n1)) or (|L| ≤ |L1|)) then11: if R = R1 then12: γ ←true13: else14: γ ← γ′

15: end if16: return substitute(P1, γ̄1 ∪ {γ}, n1 = 〈R1, 〈L′〉〉)17: end if18: compute L1 and L2 subject to L1 ◦ L2 = L′ and |L1| = |L1|19: P ← substitute(P1, γ̄1, n1 = 〈R1, 〈L1〉〉)20: for n′ ∈ children(n1,P1) do21: mergable ← mergable ∧ recursive PART merge(n′, R, L2, l)22: end for23: if mergable and R ⊆

⋃R′∈R̄

R′ then

24: if R ⊂⋃

R′∈R̄R′ then

25: P ← substitute(P1, γ̄1 ∪ {γ′}, n1 = 〈table(n1),L(n1)〉)26: end if27: return P28: end if29: return UNDEFINED

vant paths. The ordering labels in P2 will be recalculated in analogousfashion to the first case.

30

Algorithm 3 recursive PART merge(node n, table R, label L, int l)

Require: the node n has the syntax 〈R1, 〈L1〉〉1: if R ∩R1 = ∅ then2: return true3: end if4: mergable←true5: if L⊕ L1 = UNDEFINED then6: return false7: end if8: L′ ← L⊕ L1

9: if is leaf(n) or |L| ≤ |L1| then10: P ← substitute(P , n = 〈R1, 〈L′〉〉)11: R̄← R̄ ∪R1

12: K ← the rooted path in P that ends at n13: n1 ← the root node of P14: if K is not clustered relative to l then15: P ← substitute(P , γ(n1) ∪ {γ′}, n1 = 〈table(n1),L(n1)〉);16: end if17: else18: compute L1 and L2 subject to L′ = L1 ◦ L2 and |L1| = |L1|19: P ← substitute(P1, n = 〈R1, 〈L1〉〉)20: for n′ ∈ children(n) do21: mergable = mergable ∧ recursive PART merge(n′, R, L2, l)22: end for23: end if24: return mergable

The following theorem holds.

Theorem 5.15 (correctness of the PART merging algorithm). Supposethat the sSQL queries {Qi}ki=1 generate the simple PARTs {P}ki=1. Let P bethe PART produced by applying a combination of the two merge methods.

Then Xc(P) ≡k⋂i=1

Xc(Pi).

Proof: See [14].

31

6. Experimental Evaluation

6.1. Main Memory

We conducted experimental results using the example database schemashown in Figure 2. The code was written in Java and executed on a SonyVAIO VGN-NW150J laptop running the Windows OS.

First, the database was populated with 130650 customer, 3146 managers,6500 workers, and 10400 trainees. The data was uniform. For example,there were 676 distinct customer names and 201 different balances (from -100 to +100) for each customer. We next created a workload that consistedof queries and updates, where the update ratio (that is, the ratio of thenumber of single object updates to the number of single object updates plusthe number of retrieved objects in retrieval queries) was set. We examinedtwo cases: the naive approach, where an index was created for each query,and the merge approach, where indices were merged into extended indices.In the naive approach, the created index was an AA tree. The results ofthe experiment are shown in Figure 4. The horizontal axis is the updateratio, while the vertical access is the time in milliseconds. The workloadconsisted of 2000 operations, where the type of operation to execute waschosen randomly based on the update ratio. For example, when the updateratio was 0.1, 181 updates and 1819 retrieve operations were performed onaverage. All the input queries were sSQL queries. The workload was ran1000 times for each distinct update ratio and the average over all runs wasrecorded. As the update ratio increases, update time will increase and querytime will decrease. Therefore, we included only the total performance time,which shows how the update ratio influences the performance.

As the figure shows, index merging is beneficial for workloads with highupdate ratios. The reason is that less duplicate information needs to be re-freshed after every update. However, index merging is only slightly beneficialfor workloads with low update ratios. The reason is that index merging con-tributes to the creation of bigger indices and searching in them can be slower.At the same time, index merging can still be beneficial for workloads withlow update ratios when storage is scarce. For example, as Figure 5 shows,index merging reduced storage by a factor of two. In the figure, the verticalline represents the total size of the extended indices in the two cases in MBs.The saving in space is the result of avoiding unnecessary data replication.

We also ran experiments without creating any indices for workloads withhigh update ratios. In particular, we stored the tables as linked lists. As

32

56 5963 66

5662

70

80

9298

108

118

60

80

100

120

140

with merging

3742 44

4852

56 5963

4756

62

0

20

40

60

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

without merging

Figure 4: Execution time in milliseconds for different update ratios

Figure 5: Comparison in MBs of the total sizes of the two sets of extended indices

expected, the transactions ran much slower. We executed the same workload1000 times and the average run time was 12.6 seconds for update ratio of0.9. This is about 100 times slower than using an index. This shows that,even for workloads with high update ratios, creating an index is beneficialbecause an index scan significantly outperforms sequential search.

6.2. Secondary Storage

Figure 6 shows out experimental results with secondary storage. In theexperiments, the indices and materialized views that were suggested by IBMDB2 and Microsoft SQL Server for the TPC-C workload benchmark ([15])with ten warehouses were compared to performing the queries over mergedextended indices. The merged indices were manually calculated. Specifically,the sSQL queries and materialized views that are input to our algorithm were

33

manually created based on the SQL queries in the workload. That is, wemanually broke TPC-C queries that are not sSQL queries into sSQL queries.Fortunately 92% of the TPC-C workload can be directly expressed usingsSQL queries. The index merging algorithm was also applied manually (im-plementing it and evaluating its performance is a topic for future research).We applied the approximate algorithm from [14] to selected the PARTs tobe merged. The algorithm runs in quadratic time relative to the number ofinput queries. For the TPC-C workload, the approximate algorithm foundthe indices of the smallest size to create.

The left side of Figure 6 shows the size of the different auxiliary datastructures in MBs, while the right side of the figure shows performance eval-uation of running the TPC-C workload for two hours under four differentscenarios. In particular, both the Microsoft SQL Server and DB2 were runwith the physical design suggested by their own physical design advisors andwith the merged extended index that were created by applying the paper’salgorithm. All four experiments used the indices on key attributes, whichare automatically created by both commercial DBMS systems. Since IBMDB2 and Microsoft SQL Server cannot directly use extended indices, the dis-played experimental results were approximated. For example, the overheadof storing and processing marking bits was calculated by adding extra bitattributes for the marking bits. Similarly, since both commercial systemscannot process indices with branching order, the performance was evaluatedon equivalent indices of the same size without the branching order. Lastly,the polymorphic property of an extended index was evaluated by evaluatingthe performance of indices on the different object types. The improvement inperformance is due to the fact that TPC-C is an update intensive OLTP work-load. Therefore, when redundant data is eliminated through index merging,less work needs to be done to refresh the indices after every update.

7. Conclusion

We presented a novel data structure called extended index. It can usedto save space and speed up updates for workloads with significant updateratios. This is done by identifying certain redundant data among indicesand eliminating the redundancy. We showed theoretically and validated ex-perimentally the benefits of our approach. Areas for future research includeimplementing our index merging algorithm and evaluating its performance.

34

2010 30

MB

400

350

thro

ughp

ut (

tran

sact

ions

/min

ute)

time(minutes)

100

150

200

250

300

50

50 7060 80 90 100 110 12040

key indices

90 63

258

200

861

Main Memory Size

���������������������������

���������������������������

����������������

����������������������������

������������

������������

������������

����������������

����������������

������������

������������

����������������

�����������������������

���������������

������������������������

������������������������

����������

����������

������������������������������������

������������������������������������

db2 indices

db2

SQL Server

proposed with db2

proposed with SQL Server

Auxiliary Storage Size

206SQL Server

db2

SQL Server indices

my indices and views

Figure 6: Throughput with different indices

8. Summary of Notation

cl(·) merge sorting function - see Definition 4.13

cover(·) finds the cover of a set of indices - see Definition 5.7

dir constant in the set {asc, desc}

E an expression of the form {A1, . . . , Aa}, where a ≥ 1

F an expression of the form A1, . . . , Aa, where a ≥ 1

Gt tree

K path in a tree

L node’s label or a substring of a node’s label

P query parameter

Q(·) set of queries represented by the enclosed component

QP query plan

35

R relation

V materialized view

X extended index

P PART

L(n) ordering label of the node n

L↑(·) node labeling function – see page 13

L↓(·) node labeling function – see page 13

γ efficient predicate

γ(·) result of applying the predicate γ over the enclosed component; resultis TRUE of FALSE

Π(·) permutation, where the actual semantics depends on the type of theenclosed component (see Definitions 5.2 and 5.3)

| · | size of the enclosed component

|def (·)| size of the definition of the enclosed component

ID an object identifier

References

[1] Adelson-Velskii, G. M. and Landis, E. M. 1962. An Algorithm for theOrganization of Information. Soviet Math. Doklady 3, 1259–1263.

[2] Agrawal, S., Chaudhuri, S., and Narasayya, V. 2000. Automated Se-lection of Materialized Views and Indexes for SQL Databases. VLDB , 496–505.

[3] Andersson, A. 1993. Balanced search trees made simple. Workshop onAlgorithms and Data Structures, 60–71.

[4] Bayer and McCreight. 1972. Organization and Maintenance of Large Or-dered Indexes. Acta Informatica 1, 3.

[5] Bruno, N. and Chaudhuri, S. 2005. Automatic Physical Database Tuning:A Relaxation-based Approach. SIGMOD 2005 , 227–238.

36

[6] Bruno, N. and Chaudhuri, S. 2007. An online approach to physical designtuning. ICDE 2007 , 826–835.

[7] Bruno, N. and Chaudhuri, S. 2010. Constrained Physical Design Tuning.The VLDB Journal 19, 1, 21–44.

[8] Calderero, F. and Marques, F. 2010. Region Merging Techniques Us-ing Information Theory Statistical Measures. Transactions on Image Process-ing 19, 6, 1567–1586.

[9] Chaudhuri and Narasayya. 1999. Index Merging. ICDE , 296–303.

[10] Dageville, B. 2004. Automatic SQL Tuning in Oracle 10g. VLDB , 826–835.

[11] Dittrich, J.-P., Fisher, P. M., and Kossmann, D. 2005. AGILE: Adap-tive indexing for context-aware information filters. ACM SIGMOD , 215–226.

[12] Finkelstein, S., Schkolnick, M., and Tiberio, P. 1988. PhysicalDatabase Design for Relational Databases. ACM Transaction on Database Sys-tems 13, 1 (March), 91–128.

[13] Idreos, S., Kersten, M., and Manegold, S. 2007. Updating a CrackedDatabase. ACM SIGMOD , 413–424.

[14] Stanchev, L. and Weddell, G. 2009. Saving Space and Time Using IndexMerging for Main-Memory Databases. IPFW Computer Science DepartmentTechnical Report 2008-2,http://www.cs.ipfw.edu/reports/2008/report2.pdf .

[15] Transaction Processing Performance Council. TPC-C OLTP. TransactionProcessing Performance Council, http://www.tpc.org.

[16] Valentin, G., Zulian, M., Zilio, D. C., Lohman, G., and Skelley,A. 2000. DB2 Advisor: An Optimizer Smart Enough to Recommend its OwnIndexes. ICDE , 101–110.

37