Modeling and Managing Content Changes in Text Databases

Panagiotis G. Ipeirotis∗ Alexandros Ntoulas† Junghoo Cho† Luis Gravano∗

Technical Report CUCS-030-04Computer Science Department

Columbia University

Abstract

Large amounts of (often valuable) information are storedin web-accessible text databases. “Metasearchers” pro-vide unified interfaces to query multiple such databases atonce. For efficiency, metasearchers rely on succinct statisti-cal summaries of the database contents to select the best da-tabases for each query. So far, database selection researchhas largely assumed that databases are static, so the associ-ated statistical summaries do not need to change over time.However, databases are rarely static and the statistical sum-maries that describe their contents need to be updated pe-riodically to reflect content changes. In this paper, we firstreport the results of a study showing how the content sum-maries of 152 real web databases evolved over a period of52 weeks. Then, we show how to use “survival analysis”techniques in general, and Cox’s proportional hazards re-gression in particular, to model database changes over timeand predict when we should update each content summary.Finally, we exploit our change model to devise update sched-ules that keep the summaries up to date by contacting data-bases only when needed, and then we evaluate the quality ofour schedules experimentally over real web databases.

1 Introduction

A substantial amount of information on the web is storedin databases and is not indexed by search engines such asGoogle. One way to provide one-stop access to the informa-tion in text databases is throughmetasearchers, which canbe used to query multiple databases simultaneously. Theda-tabase selectionstep of the metasearching process, in whichthe best databases to search for a given query are identified, iscritical for efficiency, since a metasearcher typically providesaccess to a large number of databases. The state-of-the-artdatabase selection algorithms rely on aggregate statistics thatcharacterize the database contents. These statistics, whichare known ascontent summaries[15] (or, alternatively, asre-source descriptions[3]), usually include thefrequencyof thewords that appear in the database, plus perhaps other sim-

∗Columbia University†University of California, Los Angeles

ple statistics such as the number of documents in the data-base. These summaries, which provide sufficient informa-tion to decide which databases are the most promising forevaluating a given query, are the focus of this paper.

So far, database selection research has largely assumedthat databases are static. However, real-life databases are notalways static and the statistical summaries that describe theircontents need to be updated periodically to reflect databasecontent changes. Defining schedules for updating the data-base content summaries is a challenging task, because therate of change of the database contents might vary drasticallyfrom database to database. Furthermore, finding appropriatesuch schedules is important so that content summaries arekept up to date but without overloading databases unneces-sarily to regenerate summaries that are already (at least closeto) up to date.

In this paper, we start by presenting an extensive studyon how the content of 152 real web databases evolved overa period of 52 weeks. Given that small changes in the da-tabases might not necessarily be reflected in the (relativelycoarse) content summaries, we examined how these sum-maries change over time. Our study shows that summariesindeed change and that old summaries eventually become ob-solete, which then calls for a content summary update strat-egy. To model content changes, we resort to the field ofstatistics named “survival analysis.” Using the Cox propor-tional hazards regression model [10], we show that databasecharacteristics can be used to predict the pattern of change ofthe summaries. Finally, we exploit our change model to de-velop summary update strategies that work well even undera resource-constrained environment. Our strategies attemptto contact the databases only when needed, thus minimiz-ing the communication with the databases. To conclude thediscussion, we report the results of an extensive experimen-tal evaluation over our 152 real web databases, showing theeffectiveness of our update strategies.

In brief, the contributions of this paper are as follows:

• In Section3, we report the results of our extensive ex-perimental study on how the content summaries of 152real web databases evolved over a period of 52 weeks.

• In Section4, we use survival analysis techniques to dis-cover database properties that help predict the rate ofchange of database content summaries.

1

D1, with |D1|=51,500w f(w, D1)algorithm 7,210cassini 5saturn 2

D2, with |D2|=5,730w f(w, D2)algorithm 2cassini 3,260saturn 3,730

Table 1.A fragment of the content summaries of two data-bases.

• In Section5, we show how to update content summariesby exploiting our change model. The resulting strate-gies attempt to contact the databases only when strictlyneeded, thus avoiding wasting resources unnecessarily.

Finally, Section6 discusses related work, while Section7provides further discussion and concludes the paper.

2 Background

This section introduces the notation and necessary back-ground for this paper. We first define the notion of a “contentsummary” for a text database and briefly summarize how da-tabase selection algorithms exploit these summaries (see [18]for an expanded version of this discussion). Then, we reviewhow to obtain database content summaries via querying.

Definition 1: Thecontent summaryC(D) of a databaseDconsists of:

• The actual number of documents inD, |D|, and

• For each wordw, the number ofD documentsf(w, D)that includew.

For efficiency, a metasearcher should evaluate a queryonly on a relatively small number of databases that are rel-evant to the query. The database selection component of ametasearcher typically makes the selection decisions usingthe information in the content summaries, as the followingexample illustrates:

Example 1: Consider the query[cassini saturn]and two da-tabasesD1 andD2. Based on the content summaries of thesedatabases (Table1), a database selection algorithm may in-fer thatD2 is a promising database for the query, since eachquery word appears in manyD2 documents. In contrast,D1

will probably be deemed not as relevant, since it containsonly up to a handful of documents with each query word.

Database selection algorithms work best when the con-tent summaries are accurate and up to date. The most desir-able scenario is when each database either (1) is crawlable,so that we can (periodically) download its contents and gen-erate content summaries, or (2) exports these content sum-maries directly and reliably (e.g., using a protocol such asSTARTS [14]). Unfortunately, the so-calledhidden-webda-tabases [16], which abound on the web, are not crawlable and

only provide access to their documents via querying; further-more, no protocol is widely adopted for web-accessible da-tabases to export metadata about their contents. Hence, othersolutions have been proposed to automate the constructionof content summaries from hidden-web databases that do notexport such information.

Callan and Connell [4] presented an algorithm for build-ing (approximate) content summaries of hidden-web text da-tabases via document sampling. This algorithm first extractsa small document sample (of about 300 documents) from agiven databaseD via single-word queries. The documentsample is then treated as a small database whose contentsummary is used to approximate that ofD’s. (Alternativequery-based techniques [17] use different querying strate-gies.) In this paper, we use the document sampling and con-tent summary approximation strategy from [4], and we usethe “hat” notation to refer to an approximate content sum-mary:

Definition 2: Theapproximate, sample-based content sum-maryC(D) of a databaseD consists of:

• An estimateˆ|D| of the number of documents inD, and

• For each wordw, an estimatef(w, D) of f(w, D).

TheC(D) estimates are computed from a sample of the doc-uments inD as described in [4].

Next, we present the results of our study that examinedhow content summaries of 152 text databases changed overa period of 52 weeks.

3 Studying Content Changes of Real Text Da-tabases

One of the goals of this paper is to study how text databasechanges are reflected over time in the database content sum-maries. First, we discuss our dataset in detail (Section3.1).Then, we report our study of the effect of database changeson the content summaries (Section3.2). The conclusions ofthis study will be critical later in the paper, when we discusshow to model content summary change patterns.

3.1 Data for our Study

Our study and experiments involved 152 searchable data-bases, whose contents were downloaded weekly from Oc-tober 2002 through October 2003. These databases havepreviously been used in a study of the evolution of webpages [23]. The databases were –roughly– the five top-ranked web sites in a subset of the topical categories of theGoogle Directory,3 which, in turn, reuses the hierarchicalclassification of web sites from the Open Directory Project.4

3http://directory.google.com4http://dmoz.org

Domain com edu gov misc org% 47.3% 13.1% 17.1% 6.8% 15.7%

Table 2.Distribution of domains in our dataset.

(Please refer to [23] for more details on the rationale be-hind the choice of these web sites.) From these web sites,we picked only those sites that provided a search interfaceover their contents, which are needed to generate sample-based content summaries (see Section2). Also, since wewanted to study content changes, we only selected databaseswith crawlable content, so that every week we can retrievethe databases’ full contents using a crawler. A completelist of the sites included in our experiments is available athttp://webarchive.cs.ucla.edu/ . Table2 shows thebreakdown of web sites in the set by high-level DNS do-main, where themisccategory represents a variety of rela-tively small domains (e.g.,mil, uk, dk, andjp).

We downloaded the contents of the 152 web sites everyweek over one year, up to a maximum of 200,000 pagesper web site at a time. (Only four web sites were af-fected by this efficiency-motivated page-download limita-tion: hti.umich.edu , eonline.com , pbs.org , andintelihealth.com .) Each weekly snapshot consistedof three to five million pages, or around 65 GB before com-pression, for a total over one year of almost 3.3 TB of historydata.

We treat each web site as a database, and created –eachweek– the complete content summaryC(D) of each data-baseD by downloading and processing all of its documents.This data allowed us to study how the complete content sum-maries of the databases evolved over time. In addition, wealso studied the evolution over time ofapproximatecontentsummaries. For this, we used query-based sampling (seeSection2) to create every week an approximate content sum-maryC(D) of each databaseD.5

3.2 Measuring Content Summary Change

We now turn to measuring how the database content sum-maries –both the complete and approximate versions– evolveover time. For this, we resort to a number of metrics of con-tent summary similarity and quality from the literature. Wediscuss these metrics and the results for the 152 web data-bases next.

For our discussion, we refer to the “current” and completecontent summary of a databaseD asC(D), whileO(D, t) isthe complete summary ofD as oft weeks into the past. TheO(D, t) summary can be considered as an (old) approxima-tion of the (current)C(D) summary, simulating the realis-tic scenario where we extract a summary for a databaseD

5To reduce the effect of sampling randomness in our experiments, wecreate five approximate content summaries of each database each week, inturn derived from five document samples, and report the various metrics inour study as averages over these five summaries.

5 10 15 20 25 30 35 4t

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Weighted RecallUnweighted Recall

± .95% Confidence Interval

45 500

Figure 1.The recall of content summaryO(D, t) with re-spect to the “current” content summaryC(D), as a functionof time t and averaged over each databaseD in the dataset.

and keep it unchanged fort weeks. In the following defini-tions, Wo is the set of words that appear inO(D, t), whileWc is the set of words that appear inC(D). Valuesfo(w, D)andfc(w,D) denote the document frequency of wordw inO(D, t) andC(D), respectively.Recall: An important property of the content summary ofa database is its coverage of the current database vocabu-lary. An up-to-date and complete content summary alwayshas perfect recall, but an old summary might not, since itmight not include, for example, words that appear only innew database documents. Theunweighted recall (ur)ofO(D, t) with respect toC(D) is the fraction of words in thecurrent summary that are also present in the old summary:ur = |Wo∩Wc|

|Wc| . This metric gives equal weight to all wordsand takes values from 0 to 1, with a value of 1 meaning thatthe old content summary contains all the words that appearin the current content summary, and a value of 0 denoting nooverlap between the summaries. An alternative recall met-ric, which gives higher weight to more frequent terms, isthe weighted recall (wr)of O(D, t) with respect toC(D):wr =

Pw∈Wo∩Wc

fc(w,D)Pw∈Wc

fc(w,D) . We will use analogous definitions

of unweighted and weighted recall for a sample-based con-tent summaryO(D, t) of databaseD obtainedt weeks intothe past with respect to the current content summaryC(D)for the same database.

Figure1 focuses on complete content summaries. Specif-ically, this figure shows the weighted and unweighted re-call of t-week-old summaries with respect to the “current”summary, as a function oft and averaged over every pos-sible choice of “current” summary. In Figure1 (as well asin all subsequent figures), we report our results with a 95%confidence interval. Predictably, both the weighted and un-weighted recall values decrease ast increases. For example,on average, 1-week-old summaries have unweighted recall of91%, while older, 25-week-old summaries have unweightedrecall of about 80%. The weighted recall figures are higher,as expected, but still significantly less than 1: this indicates

0 5 10 15 20 25 30 35 40 45 50 55

T

0.685

0.690

0.695

0.700

0.705

0.710

Weig

hte

d R

ecall

K, tau

K, size, tau

Size, tau

Naive

± 95% Confidence Interval

Figure 2.The weighted recall of “old” sample-based contentsummaries with respect to the “current” ones, as a function ofthe timeT between updates and averaged over each databaseD in the dataset, for different scheduling policies (τ = 0.5).

0 5 10 15 20 25 30 35 40 45 50

T

0.320

0.325

0.330

0.335

0.340

0.345

K, tau

K, size, tau

Size, tau

Naive

Un

we

igh

ted

Re

ca

ll

± 95% Confidence Interval

Figure 3.The unweighted recall of “old” sample-based con-tent summaries with respect to the “current” ones, as a func-tion of the timeT between updates and averaged over eachdatabaseD in the dataset, for different scheduling policies(τ = 0.5).

that the newly introduced words have low frequencies, butconstitute a substantial fraction of the database vocabularyas well.

The curves labeled “Naive” in Figures2 and3 show thecorresponding results for approximate, sample-based contentsummaries. (Please ignore the other curves for now; we willexplain their meaning in Section5.) As discussed in Sec-tion 2, hidden-web text databases, which are not crawlable,require that content summaries be approximated via sam-pling, so studying sample-based summaries is important. Asexpected, the recall values for the sample-based summariesare substantially smaller than the ones for the complete sum-maries. Also, the recall values of the sample-based sum-maries do not change much over time, because the sample-based summaries are not too accurate to start with, and do notsuffer a significant drop in recall over time. This shows thatthe inherent incompleteness of the sample-based summaries“prevails” over the incompleteness introduced by time.

Another interesting observation is that recall figures ini-tially decrease (slightly) for approximately 20 weeks, then

5 10 15 20 25 30 35 40t

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Weighted Precision

Unweighted Precision

± 95% Confidence Interval

45 50

Figure 4.The precision of content summaryO(D, t) with re-spect to the “current” content summaryC(D), as a functionof time t and averaged over each databaseD in the dataset.

remain stable, and then, surprisingly, increase, so that a 50-week old content summary has higher recall than a 20-weekold one, for example. This unexpected result is due to an in-teresting periodicity: some events (e.g., “Christmas,” “Hal-loween”) appear at the same time every year, allowing sum-maries that are close to being one year old to have higherrecall than their younger counterparts. This effect is only vis-ible in the sample-based summaries that cover only a smallfraction of the database vocabulary, and is not observed inthe complete summaries, perhaps because they are larger andare not substantially affected by a relatively small number ofwords.Precision: Another important property of the content sum-mary of a database is the precision of the summary vocabu-lary. Up-to-date content summaries contain only words thatappear in the database, while older summaries might includeobsolete words that appeared only in deleted documents. Theunweighted precision (up)of O(D, t) with respect toC(D)is the fraction of words in the old content summary thatstill appear in the current summaryC(D): up = |Wo∩Wc|

|Wo| .This metric, likeunweighted recall, gives equal weight toall words and takes values from 0 to 1, with a value of 1meaning that the old content summary only contains wordsthat are still in the current content summary, and a value of0 denoting no overlap between the summaries. The alterna-tive precision metric, which –just as in theweighted recallmetric– gives higher weight to more frequent terms, is theweighted precision (wp)of O(D, t) with respect toC(D):wp =

Pw∈Wo∩Wc

fo(w,D)Pw∈Wo

fo(w,D) . We use analogous definitions of

unweighted and weighted precision for a sample-based con-tent summaryO(D, t) of a databaseD with respect to thecorrect content summaryC(D).

Figure4 focuses on complete content summaries. Specif-ically, this figure shows the weighted and unweighted preci-sion of t-week-old summaries with respect to the “current”summary, as a function oft and averaged over every pos-sible choice of “current” summary. Predictably, both the

0 5 10 15 20 25 30 35 40 45 50

T

0.83

0.84

0.85

0.86

0.87

0.88

0.89

0.90

0.91

0.92

Wei

ghte

d Pr

ecis

ion

K, tau

K, size, tau

Size, tau

Naive ± 95% Confidence Interval

Figure 5.The weighted precision of “old” sample-based con-tent summaries with respect to the “current” ones, as a func-tion of the timeT between updates and averaged over eachdatabaseD in the dataset, for different scheduling policies(τ = 0.5).

0 5 10 15 20 25 30 35 40 45 50

T

0.72

0.74

0.76

0.78

0.80

0.82

0.84

0.86

Unw

eigh

ted

Prec

isio

n

K, tau

K, size, tau

Size, tau

Naive

± 95% Confidence Interval

Figure 6.The unweighted precision of “old” sample-basedcontent summaries with respect to the “current” ones, as afunction of the timeT between updates and averaged overeach databaseD in the dataset, for different scheduling poli-cies (τ = 0.5).

weighted and unweighted precision values decrease ast in-creases. For example, on average, a 48-week-old summaryhas unweighted precision of 70%, showing that 30% of thewords in the old content summary do not appear in the data-base anymore.

The curves labeled “Naive” in Figures5 and6 show thecorresponding results for approximate, sample-based contentsummaries. (Again, please ignore the other curves for now;we will explain their meaning in Section5.) As expected, theprecision values decrease over time, and do so much fasterthan their corresponding recall values (Figures2 and3). Forexample, almost 20% of the words in a 15-week-old sample-based content summary are absent from the database. For theprecision results, the periodicity that appeared in the recallfigures is not visible: the sample-based content summaries

5 10 15 20 25 30 35 40

t

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

KL

± 95% Confidence Interval

45 50

Figure 7.The KL divergence of content summaryO(D, t)with respect to the “current” content summaryC(D), as afunction of timet and averaged over each databaseD in thedataset.

contain many more “obsolete” words that do not appear inthe database anymore. Hence, a small number of words thatappear periodically cannot improve the results.Kullback-Leibler Divergence: Precision and recall mea-sure the accuracy and completeness of the content sum-maries, basedonly on the presence of words in the sum-maries. However, these metrics do not capture the accu-racy of the frequency of each word as reported in the con-tent summary. For this, theKullback-Leibler divergence[19]of O(D, t) with respect toC(D) (KL for short) calculatesthe “similarity” of the word frequencies in the old contentsummaryO(D, t) against the “current” word frequencies inC(D): KL =

∑w∈Wo∩Wc

pc(w|D) · log pc(w|D)po(w|D) , where

pc(w|D) = fc(w,D)Pw′∈Wo∩Wc

fc(w′,D) is the probability of ob-

servingw in C(D), andpo(w|D) = fo(w,D)Pw′∈Wo∩Wc

fo(w′,D)

is the probability of observingw in O(D, t). The KL di-vergence metric takes values from 0 to infinity, with 0 indi-cating that the two content summaries being compared areequal. Intuitively, KL divergence measures how many bitsare necessary to encode the difference between the two dis-tributions.

Figure 7 focuses on complete content summaries andshows that the KL divergence of old content summariesO(D, t) increases ast increases. This confirms the previ-ously observed results and shows that the word frequencydistribution changes substantially over time. The curve la-beled “Naive” in Figure8 shows the KL divergence forsample-based content summaries of increasing age. (Again,please ignore the other curves for now; we will explain theirmeaning in Section5.) The KL divergence of the old sum-maries increases with time, indicating that approximate con-tent summaries become obsolete just as their complete coun-terparts do.Conclusion: We studied how content summaries of text da-tabases evolve over time. We observed that the quality ofcontent summaries (both complete and sample-based) deteri-

0 5 10 15 20 25 30 35 40 45 50

T

0.56

0.58

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

KL

K, tau

K, size, tau

Naive Size, tau

± 95% Confidence Interval

Figure 8.The KL divergence of “old” sample-based contentsummaries with respect to the “current” ones, as a function ofthe timeT between updates and averaged over each databaseD in the dataset, for different scheduling policies (τ = 0.5).

orates as they become increasingly older. Therefore, it is im-perative to have a policy for periodically updating the sum-maries to reflect the current contents of the databases. Weturn now to this important issue and show how we can use“survival analysis” for this purpose.

4 Predicting Content Summary Change Fre-quency

In the previous section, we established the need for up-dating database content summaries as the underlying textdatabases change. Unfortunately, updating a content sum-mary involves a non-trivial overhead: as discussed, the con-tent summaries of hidden-web databases are constructed byquerying the databases, while the summaries of crawlabledatabases are constructed by downloading and processing allthe database documents. Therefore, in order to avoid over-loading the databases unnecessarily, it is important to sched-ule updates carefully. In this section, we present our “sur-vival analysis” modeling approach for decidingwhento up-date content summaries. First, Sections4.1 and 4.2 reviewthe necessary background on survival analysis and the Coxregression model from the literature [21]. Then, Section4.3shows how we can use this material for our own scenario, tomodel content summary changes.

4.1 Survival Analysis

Survival analysis is a collection of statistical techniquesthat help predict the time until an event occurs [21]. Thesemethods were initially used to predict the time of survivalfor patients under different treatments, hence the name “sur-vival analysis.” For the same reason the “time until an eventoccurs” is also calledsurvival time. For our purposes, thesurvival time is the number of weekst such that an old da-tabase content summaryO(D, t) is “sufficiently different”

from the current summaryC(D). (We formally define thesurvival time of a database in Section4.3.)

Survival times can be modeled through asurvival functionS(t) that captures the probability that the survival time of anobject is greater than or equal tot. In the survival analysisliterature, the distribution ofS(t) is also described in termsof a hazard functionh(t), which is the “rate of failure” at

time t, conditional on survival until timet: h(t) = −dS(t)

dt

S(t) .A common modeling choice forS(t) is theexponential dis-tribution, whereS(t) = e−λt, and so the hazard functionis constant over time (h(t) = λ). A generalization of theexponential distribution is theWeibull distribution, whereS(t) = e−λtγ

, and so the hazard function varies over time(h(t) = λγtγ−1).6

We could use the exponential distribution to model thesurvival time of a database. This choice is reinforced by re-cent findings that indicate that the exponential function is agood model to describe changes in webdocuments[1, 6].However, we will see in Section4.3that the exponential dis-tribution does not accurately describe changes for summariesof webdatabases, so we will use the Weibull distribution in-stead.

As described so far, the survival functionS(t) and thehazard functionh(t) are used to describe a single database,and are not “instantiated” since we do not know the values ofthe configuring parameters. Of course, it is important to esti-mate the parameters of the survival functionS(t) for each da-tabase, to have a concrete, database-specific change model.Even more imperative is to discoverpredictor variablesthatcan influence the survival times. For example, when ana-lyzing the survival times of patients with heart disease, theweight of a patient is a predictor variable and can influencethe survival time of the patient. Analogously, we want to pre-dict survival times individually for each database, accordingto its characteristics. Next, we describe the Cox proportionalhazards regression model that we use for this purpose.

4.2 Cox Proportional Hazards Regression Model

TheCox proportional hazards regression model[10] is atechnique widely used in statistics for discovering importantvariables that influence survival times. It is a non-parametricmodel, because it makes no assumptions about the nature orshape of the hazard function. The only assumption is that thelogarithm of the underlying hazard rate is a linear7 functionof the predictor variables.

Let x be a predictor variable, andxA andxB be the val-ues of that variable for two databasesA andB, respectively.Under the Cox model, the hazard functionshA(t) andhB(t)

6The exponential distribution corresponds to the case whereγ = 1.7The “linearity” or “proportionality” requirement is essentially a “mono-

tonicity” requirement (e.g., the higher the weight of a patient, the higher therisk of heart attack). If a variable monotonically affects the hazard rate, thenan appropriate transformation (e.g.,log(·)) can make its effect linear.

can be expressed for databasesA andB as:

hA(t) = eβxAh0(t) ⇒ ln hA(t) = ln h0(t) + βxA (1a)

hB(t) = eβxBh0(t) ⇒ ln hB(t) = ln h0(t) + βxB (1b)

whereh0(t) is a baseline hazard function, common for allthe members of the population. The Cox model can be gen-eralized forn predictor variables:log h(t) = log h0(t) +∑n

i=1 βixi, where thexi’s are the predictor variables, andtheβi’s are the model coefficients. The algorithm presentedby Cox [10] shows how to compute theβi values.

The Cox model, as presented so far, seems to solve thesame problem addressed by multiple regression. However,the dependent variable (survival time) in our case is not nor-mally distributed, but usually follows the exponential or theWeibull distribution – a serious violation for ordinary multi-ple regression. Another important distinction is the fact thatthe Cox model effectively exploits incomplete or “censored”data, from cases that “survived” the whole study period. Ex-cluding these cases from the study would seriously affect theresult, introducing a strong bias in the resulting model. Thoseobservations are calledcensoredobservations and containonly partial information, indicating thatthere was no failureduring the time of observation. The Cox model effectivelyuses the information provided from censored cases. (Formore information, see [10].)

The Cox proportional hazards model is one of the mostgeneral models for working with survival data, since it doesnot assume any specific baseline hazard function. This modelallows the extraction of a “normalized” hazard functionh0(t)that is not influenced by predictor variables. This allows foreasier generalization of the results, sinceh0(t) is not de-pendent on the distribution of the predictor variables in thedataset used to extracth0(t). The only requirement for theapplicability of Cox’s model is that the predictor variablesfollow the “proportional hazard” (PH, or linearity) assump-tion, which means that for two individual groupsA andB

the hazard ratiohA(t)hB(t) is constant over time.

An interesting variation of the Cox model that overcomesthe PH assumption is thestratified Cox model[26], whichis used to account for variables that do not satisfy the pro-portionality assumption. In this case, the variables that donot satisfy the proportionality assumption are used to splitthe dataset into different “strata.” Theβi Cox coefficientsremain the same across the different strata, but each stratumnow has different baseline functionsh0(t).

Next, we describe how we use the Cox regression modelto represent changes in text database content summaries.

4.3 Using Cox Regression to Model Content Sum-mary Changes

Before using any survival analysis technique for our prob-lem, we need to define “change.” A straightforward defini-tion is that two content summariesC(D) andO(D, t) are

“different” when they are not identical. However, even asmall change in a single document in a database will proba-bly result in a change in its content summary, but such changeis unlikely to be of importance for database selection. There-fore, we relax this definition and say that two content sum-maries are different whenKL > τ (see Section3.2 for thedefinition of KL divergence), whereτ is a “change sensitiv-ity” threshold.8 Higher values ofτ result in longer survivaltimes and the exact value ofτ should be selected based on thecharacteristics of the database selection algorithm of choice.We will see how we can effectively use the Cox model toincorporateτ in our change model. Later, in Section5, weshow that we can define update schedules that adapt to thechosen value ofτ .

Definition 3: Given a value of the change sensitivity thresh-old τ > 0, the survival time of a databaseD at a point intime –with associated “current” content summaryC(D)– isthe smallest timet for which the KL divergence ofO(D, t)with respect toC(D) is greater thanτ .

Computing Survival Times: Using the study of Section3as well as Definition3, we computed the survival time ofeach content summary for different values of thresholdτ .For some databases, we did not detect a change within theperiod of the study. As explained in Section4.2, these “cen-sored” cases are still useful since they provide evidence thatthe content summary of a database with the given character-isticsdid not changewithin the allotted time period and forthe thresholdτ of choice. The result of our study is a setof survival times, some marked as censored, that we use asinput to the Cox regression model.

Feature Selection: After extracting the survival times, weselect the database features that we pass as parameters to theCox model. We use two sets of features: a set of “current”features and a set of “evolution” features. Thecurrent fea-tures are characteristics of the database at a given point intime. For example, the topic of the database and its DNS do-main arecurrent features of a database. On the other hand,we extract theevolutionfeatures by observing how the data-base changes over a (training) time period. For the remainderof the discussion –and because of space constraints– we fo-cus on the features for the important case of approximate,sample-based content summaries. Analogous features canbe defined for crawlable databases, for which we can extractcomplete summaries.

The initial set ofcurrent features that we used was:

• The thresholdτ .8We use KL divergence for our change definition (as opposed to pre-

cision or recall) because KL depends on the whole word-frequency distri-bution. As our later experiments show, an update policy derived from theKL-based change definition improves not only the KL divergence but alsoprecision and recall.

• The logarithm of the estimated size of the database,where we estimate the size of the database using the“sample-resample” method from [25].

• The number of words in the current sampleC(D).

• The topic of each database, defined as the top level cat-egory under which the database is classified in the OpenDirectory. This is a categorical variable with 16 distinctvalues (e.g., “Arts,” “Sports,” and so on). We encodedthis variable as a set of dummy binary variables: eachvariable has the value 1 if the database is classified un-der the corresponding category, and 0 otherwise.

• The domain of the database, which is a categorical vari-able with five distinct values (com, org, edu, gov, misc).We encoded this variable as a set of 5 binary variables.

To extract the set ofevolution features, we retrievedsample-based content summaries from each database everyweek over a period of 10 weeks. Then, for each database wecompared every pair ofapproximatesummaries that were ex-tracted exactlyk weeks apart (i.e., on weekst andt + k) us-ing the precision, recall, and KL divergence metrics. Specif-ically, the features that we computed were:

• The average KL divergenceκ1, . . . , κ9 between sum-maries extracted with time difference of1, . . . , 9 weeks.

• The average weighted and unweighted precision ofsummaries extracted with time difference of1, . . . , 9weeks.

• The average weighted and unweighted recall of sum-maries extracted with time difference of1, . . . , 9 weeks.

After selecting the initial set of features, we trained theCox model using the variables indicated above. We validatedthe results using leave-one-out cross validation.9 The resultsof the initial run indicated that from thecurrent features, thenumber of words and the topic of the database are not goodpredictor variables, while from theevolutionfeatures, pre-cision and recall are not good predictor variables; the KLfeatures are good predictors, and strongly and positively cor-related with each other.Given these results, we decided to drop the number of wordsand the topic variables from thecurrent set, keeping onlythe thresholdτ , the database size, and the domain. Also, wedropped the recall and precision features from theevolutionset, keeping only theκ1 feature: given its presence, featuresκ2 throughκ9 were largely redundant. Furthermore, we re-duced the training time from 10 to three weeks. To examinewhether any of the selected features –other than thresholdτ ,which we always keep– are redundant, we trained Cox using(a) size andτ ; (b) κ1 and τ ; and (c)κ1, size, andτ . Wedescribe our findings next.

9Since each database generates multiple survival times, we leave out onedatabaseat a time for the cross-validation.

Features βs βκ βτ

size,τ 0.179 - -1.313κ1, τ - 8.3 -1.308

κ1, size,τ 0.094 6.762 -1.305

Table 3.The coefficients of the Cox model, when trained forvarious sets of features.

Training the Cox Model: After the initial feature selec-tion, we trained the Cox model again. The results indicatedthat all the features that we had selected are good predic-tor variables10 and strongly influence the survival time of theextracted summaries. However, the domain variable did notsatisfy the proportionality assumption, which is required bythe Cox model (see Section4.2): the hazard ratio betweentwo domains was not constant over time. Hence, we resortedto thestratified Cox model, stratifying on domain.11

The result of the training was a set of coefficientsβs, βκ,andβτ for features size,κ1, andτ , respectively. We showthe Cox coefficients that we obtained in Table3. The pos-itive values ofβs andβκ indicate that larger databases aremore likely to change than smaller ones and that databasesthat changed during training are more likely to change in thefuture than those that did not change. In contrast, the nega-tive value forβτ shows that –not surprisingly– higher valuesof τ result in longer survival times for content summaries.

Given the results of the analysis, for two databasesD1

andD2 from the same domain, we have:

ln S1(t) = exp(βs ln(|D1|) + βκκ11 + βτ τ1) · ln S0(t)ln S2(t) = exp(βs ln(|D2|) + βκκ12 + βτ τ2) · ln S0(t)

whereS0(t) is the baseline survival function for the respec-tive domain. The baseline survival function corresponds to a“baseline” databaseD with size|D| = 1 (i.e., ln(|D|) = 0),κ1 = 0, andτ = 0.

Under the Cox model, the returned baseline survival func-tions remain unspecified and are defined only by a set of val-uesS0(t1), S0(t2), . . . , S0(tn). In our experiments, we hadfive baseline survival functions, one for each domain (i.e.,com, edu, org, gov, misc). To fit the baseline survival func-tions, we assumed that they follow the Weibull distribution(see Section4.1), which has the general formS(t) = e−λtγ

.We applied curve fitting using a least-squares method (inparticular the Levenberg-Marquardt method [22]) to esti-mate the parameters of the Weibull distribution for each do-main. For all estimates, the statistical significance was at the0.001% level. Table4 summarizes the results.

An interesting result is that the survival functions donot follow the exponential distribution (γ = 1). Previousstudies [6] indicated that individual webdocumentshave

10For all models, the statistical significance is at the 0.001% level accord-ing to the Wald statistic [21].

11This meant that we had to compute separate baseline hazard functionsfor each domain.

Features Domain λdom γdom

com 0.0211 0.844edu 0.0392 0.578

size,τ gov 0.0193 0.701misc 0.0163 1.072org 0.0239 0.723com 0.0320 0.886edu 0.0774 0.576

κ1, τ gov 0.0245 0.795misc 0.0500 1.014org 0.0542 0.715com 0.0180 0.901edu 0.0205 0.585

κ1, size,τ gov 0.0393 0.780misc 0.0236 1.050org 0.0274 0.724

Table 4.The parameters for the baseline survival functionsfor the five domains. The baseline survival functions describethe survival time of a databaseD in each domain with size|D| = 1 (ln(|D|) = 0), with average distance between thesample summariesKL = 0 and for thresholdτ = 0.

lifetimes that follow the exponential distribution. Our re-sults, though, indicate that content summaries, with aggre-gate statistics aboutsets of documents, change more slowly.

Modeling Conclusions: We have presented a statisticalanalysis of the survival times of database content summaries.We used Cox regression analysis to examine the effect of dif-ferent variables in the survival time of database content sum-maries and showed that the survival times of content sum-maries follow the Weibull distribution, in most cases withγ < 1 (i.e., they tend to remain unchanged for longer timeperiods as their age increases). We summarize our results inthe following definition:

Definition 4: The functionSi(t) that gives the survival func-tion for a databaseDi is:

Si(t) = exp (−λitγdom) , with (2a)

λi = λdom

(|Di|βs · exp (βκκ1i) · exp (βτ τi))

(2b)

where|Di| is the size of the database,κ1i is the KL diver-gence of the samples obtained during the training period,βs,βκ, andβτ are the Cox coefficients from Table3, λdom andγdom are the domain-specific constants from Table4, andτi

is the value of the change threshold forDi (Definition3).

Definition4 provides a concrete change model for a data-baseD that is specific to the database characteristics and tothe change sensitivity, as controlled by the thresholdτ . Aninteresting result is that summaries of large databases changemore often than those of small databases, as indicated by thepositive value ofβs, which corresponds to the database size.Figure9 shows the shape ofS(t) for different domains, fora hypothetical databaseD with |D| = 1000 andκ1 = 0.1,and forτ = 0.5. This figure shows that content summaries

S(t)

1

0.8

0.6

0.4

0.2

t50403020100

com

edu

gov

misc

org

Figure 9.The survival functionS(t) for different domains(|D| = 1, 000, τ = 0.5, κ1 = 0.1).

tend to vary substantially across domains (e.g., compare the“misc” curve against the “gov” curve).

5 Scheduling Updates

So far, we have described how to compute the survivalfunctionS(t) for a text database. In this section, we describehow we can exploitS(t) to schedule database content sum-mary updates and contact each database only when neces-sary. Specifically, we first describe the theory behind ourscheduling policy (Section5.1). Then, we present the exper-imental evaluation of our policy (Section5.2), which showsthat sophisticated update scheduling can improve the qualityof the extracted content summaries in a resource-restrictedenvironment.

5.1 Deriving an Update Policy

A metasearcher may provide access to hundreds or thou-sands of databases and operate under limited network andcomputational resources. To optimize the overall quality ofthe content summaries, the metasearcher has to carefully de-cide when to update each of the summaries, so that they areacceptably up to date during query processing.

To model the constraint on the workload that a meta-searcher might handle, we defineF as the average number ofcontent summary updates that the metasearcher can performin a week. Then, under aNaivestrategy that allocates up-dates to databases uniformly,T = n

F represents the averagenumber of weeks between two updates of a database, wheren is the total number of databases. For example,T = 2weeks means that the metasearcher can update the contentsummary of each database every two weeks, on average.

As we have seen in Section4.3, the rate of change ofthe database contents may vary drastically from database todatabase, so theNaive strategy above is bound to allocateupdates to databases suboptimally. Thus, the goal of ourupdate scheduling is to determine the update frequencyfi

Di λi T = 40 T = 10tomshardware.com 0.088 46 weeks 5 weeksusps.com 0.023 34 weeks 12 weeks

Table 5.Optimal content-summary update frequencies fortwo databases

for each databaseDi individually, in such a way that thefunction

∑ni=1 Si(t) is maximized, while at the same time

not exceeding the number of updates allowed. In this case,we maximize the average probability that the content sum-maries are up to date. One complication is that the sur-vival function Si(t) changes its value over time, so differ-ent update scheduling policies may be considered “optimal”depending on whenSi(t) is measured. To address this is-sue, we assume that the metasearcher wants to maximizethe time-averagedvalue of the survival function, given as:S = limt→∞ 1

t

∫ t

0

∑ni=1 Si(t)dt. This formulation of the

scheduling problem is similar to that in [7] for the problemof keeping the index of a search engine up to date. In short,we formulate our goal as the following optimization prob-lem.

Problem 1: Find the optimal update frequencyfi for eachdatabaseDi such thatS is maximized under the constraint∑n

i=1 fi = nT .

Given the analytical forms of theSi(t) functions in the pre-vious sections, we can solve this optimization problem us-ing theLagrange-multiplier method(as shown for examplein [7, 24]). Cho et al. [7] investigated a special case of thisoptimization problem whenγ = 1 (i.e., when the rate ofchange is constant over time), and observed the following:

1. Whenλi (which can be interpreted as denoting “howoften the content summary changes”) is small relativeto the constraintF , the optimal revisit frequencyfi be-comes larger asλi grows larger.

2. Whenλi is large compared to the resource constraintF ,the optimal revisit frequencyfi becomes smaller asλi

grows larger.

In our solution to the above generalized optimizationproblem, we also observed similar trends even whenγ 6= 1(i.e., when the rate of change varies over time). As an exam-ple, in Table5 we show the optimal update frequencies forthe content summaries of two databases,tomshardware.com and usps.com . We can see that, whenT is small(T = 10), we updatetomshardware.com more oftenthanusps.com , sinceλi is larger fortomshardware.com. However, whenT is large (T = 40) the optimal updatefrequencies are reversed. The scheduling algorithm decidesthattomshardware.com changes “too frequently” and isnot beneficial to allocate more resources to try to keep it upto date. Therefore, the algorithm decides to update the con-tent summary fromtomshardware.com less frequently,

and instead focus on databases likeusps.com that can bekept up to date. This trend holds across domains and acrossvalues ofγ.

5.2 Experimental Results

In Section4.3, we showed how to compute the form andparameters of the survival functionSi(t), which measuresthe probability that the summary of a databaseDi is up todatet weeks after it was computed. Based on Cox’s model,we derived a variety of models that computeSi(t) based onthree different sets of features (see Tables3 and4). Now, weuse these models to devise three update policies, using theapproach from Section5.1and the following feature sets:

• κ1, size,τ : We use all the available features.• size andτ : We do not use the history of the database,

i.e., we ignore the evolution featureκ1 and we use onlythe database size and the change sensitivity thresholdτ .

• κ1 andτ : We use only the history of the database andthe thresholdτ . We consider this policy to examinewhether we can work with databases without estimat-ing their size.12

We also consider theNaivepolicy, discussed above, wherewe uniformly update all summaries everyT weeks.13

Quality of Content Summaries under Different Policies:We examine the performance of each updating policy, bymeasuring the average (weighted and unweighted) precisionand recall, and the average KL divergence of the generatedapproximatesummaries. We consider different values ofT ,whereT is the average number of weeks between updates.

Figures 2 and 3 show the average weighted and un-weighted precision of the approximate summaries, obtainedunder the scheduling policies that we consider. The resultsindicate that, by using any of our policies, we can keep therecall metrics almost stable, independently of the resourceconstraints. Figures5 and6 show the average weighted andunweighted precision of the approximate summaries, respec-tively. Again, our three scheduling policies demonstrate sim-ilar performance, and they are all significantly better than theNaivepolicy. The difference with theNaivepolicy is statis-tically significant, even when the summaries are updated rel-atively frequently (i.e., even for small values ofT ). Finally,Figure8 shows that our updating policies keep the averageKL divergence of the approximate summaries almost con-stant even for a large number of weeksT between updates.

An interesting observation is that the three policies that wepropose demonstrate minimal differences in performance,

12The size estimation method that we use [25] relies on the database re-turning the number of matches for each query. This method becomes prob-lematic for databases that do not report such numbers with the query results.

13Due to space constraints, the results presented in this paper focus onsample-based content summaries. We also ran analogous experiments forthe complete content summaries, and the results were similar.

0 5 10 15 20 25 30 35 40 45 50

T

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

K, tau Size, tau K, size, tau

Upd

ate

Pre

cisi

on

± 95% Confidence Interval

Figure 10.The precision of the updates performed by thedifferent scheduling algorithms, as a function of the averagetime between updatesT and forτ = 0.5.

and these differences are not statistically significant. Addi-tionally, all techniques are significantly better than theNaivepolicy. This indicates that it is possible to work with asmaller set of features, without decreasing performance. Forexample, we may ignore the evolution featureκ1 and avoidcomputing the history of a database, which involves frequentsampling of the database for a (small) period of time.

Precision of Update Operations: To measure how “pre-cise” the updates scheduled by our policies are, we define anupdate as “precise” if it contacts a database when the newsummary of the database is different from the existing sum-mary according to the definition of change in Section4.3. Wemeasured the precision of the update operations as the ratioof the precise updates over the total number of updates per-formed. Figure10 shows the precision results as a functionof T and forτ = 0.5. For this value ofτ and for the data-bases in our dataset, very low values ofT (i.e.,T < 10) areunnecessary, since then the databases are contacted too oftenand before they have changed sufficiently. A decrease in thevalue ofτ cause the curves to “move” towards the left: thesummaries change more frequently and then the updates be-come more precise. For example, forτ = 0.25 andT = 10,precision is approximately 40%, while forT = 25 it is ap-proximately 80%.

Interestingly, the update precision can be predicted ana-lytically, using the target functionS described in Section5.1.The average probability of survival (our target function) cor-responds in principle to the percentage of non-precise up-dates. This result is intuitive, since our target function essen-tially encodes the probability that the summary of the data-base has changed. Therefore, during scheduling, it is possi-ble to select a value ofT that achieves (approximately) thedesired update precision.

Conclusion: As a general conclusion, we have observedthat our scheduling policies allow for better quality of the

extracted content summaries, even under strict constraintson the allowable update frequency. Also, our modeling ap-proach allows us to predict the precision of our update oper-ations, in turn allowing the metasearcher to tune the updatefrequency and efficiently keep the content summaries up todate.

6 Related Work

We are not aware of prior work to experimentally measuredatabase content summary evolution over time or to scheduleupdates to the content summaries to maintain their freshness.However, several previous studies have focused on variousaspects of the evolution of the web and of the related prob-lem of web crawling. Ntoulas et al. [23] studied the changesof individual web pages, using the same dataset as we did inthis paper. Ntoulas et al. concluded that 5% of new content(measured in “shingles”) is introduced in an average week inall pages as a whole. Additionally, [23] observed a strongcorrelation between the past and the future degrees of thechanges of a web page and showed that this correlation mightbe used to predict the future changes of a page. For exam-ple, by measuring how much a page changed in the past oneweek, we might predict how much the page would change inthe next one week quite accurately. In this paper (Section3),we investigated this high-level idea more formally throughsurvival analysis and modeled the change behavior of webdatabases using the Cox proportional hazard model. Thismodel was then used for designing the optimal schedulingalgorithm for summary updates. Lim et al. [20] and Fetterlyet al. [13] presented pioneer measurements of the degree ofchange of web pages over time, where change was measuredusing the edit distance [20] or the number of changed “shin-gles” [13] over successive versions of the web pages. Otherstudies of web evolution include [1, 5, 27, 11, 2], and focuson issues that are largely orthogonal to our work, such aspage modification rates and times, estimation of the changefrequencies for the web pages, and so on.

Web crawling has attracted a substantial amount of workover the last few years. In particular, references [7, 9, 12, 8]study how a crawler should download pages to maintainits local copy of the web up to date. Assuming that thecrawler knows the exact change frequencies of pages, ref-erences [7, 9] present an optimal page downloading algo-rithm, while [12] proposes an algorithm based on linear pro-gramming. Cho and Ntoulas [8] employ sampling to detectchanged pages. All this work on web crawling mainly fo-cuses on maintaining a local copy of the web as up-to-date aspossible, which requires maximizing the fraction of remotepages whose local copy is up to date. Our goal is different:we want to maximize the freshness of the content summariesthat describe the various web sites, so that we produce moreaccurate database selection decisions.

Olston et al. [24] proposed a new algorithm for cache syn-

chronization in which data sources notify caches of impor-tant changes. Cho et al. [6, 7] proposed optimal algorithmsfor web-page cache synchronization. The definition of “di-vergence” or “change” in [24] is quite general and can beapplied to our context. Their high-level optimization goalis also similar to ours. However, the proposed push modelmight not be applicable when data sources are “uncooper-ative” and do not inform others of their changes as is thecase on the web. The algorithms proposed in [6, 7] areproven to be optimal when web-page changes follow a Pois-son process. Unfortunately, the changes of database contentsummaries do not follow a Poisson process, and our updatescheduling algorithm was derived based on a more generalassumption.

7 Conclusions

We presented a study –over 152 real web databases– of theeffect of time on the database content summaries on whichmetasearchers rely to select appropriate databases where toevaluate keyword queries. Predictably, the quality of thecontent summaries deteriorates over time as the underlyingdatabases change, which highlights the importance of up-date strategies for refreshing the content summaries. Wedescribed how to use survival analysis techniques, in par-ticular how to exploit the Cox proportional hazards regres-sion model, for this update problem. We showed that thechange history of a database can be used to predict the rateof change of its content summary in the future, and that sum-maries of larger databases tend to change faster than sum-maries of smaller databases. Finally, based on the results ofour analysis, we suggested update strategies that work wellin a resource-constrained environment. Our techniques adaptto the change sensitivity desired for each database, and con-tact databases selectively –as needed– to keep the summariesup to date while not exceeding the resource constraints.

References

[1] B. E. Brewington and G. Cybenko. How dynamic is the web?In Proceedings of the Ninth International World Wide WebConference (WWW9), pages 257–276, 2000.

[2] B. E. Brewington and G. Cybenko. Keeping up with thechanging web.IEEE Computer, 33(5):52–58, May 2000.

[3] J. P. Callan. Distributed information retrieval. InAdvancesin Information Retrieval, pages 127–150. Kluwer AcademicPublishers, 2000.

[4] J. P. Callan and M. Connell. Query-based sampling oftext databases.ACM Transactions on Information Systems,19(2):97–130, 2001.

[5] J. Cho and H. Garcıa-Molina. The evolution of the web andimplications for an incremental crawler. InProceedings ofthe 26th International Conference on Very Large Databases(VLDB 2000), pages 200–209, 2000.

[6] J. Cho and H. Garcıa-Molina. Estimating frequency ofchange.ACM Transactions on Internet Technology, 3(3):256–290, Aug. 2003.

[7] J. Cho, H. Garcıa-Molina, and L. Page. Synchronizing a da-tabase to improve freshness. InProceedings of the 2000 ACMSIGMOD International Conference on Management of Data(SIGMOD 2000), pages 117–128, 2000.

[8] J. Cho and A. Ntoulas. Effective change detection using sam-pling. In Proceedings of the 28th International Conference onVery Large Databases (VLDB 2002), pages 514–525, 2002.

[9] E. G. Coffman, Jr., Z. Liu, and R. R. Weber. Optimal robotscheduling for web search engines.Journal of Scheduling,1(1):15–29, June 1998.

[10] D. R. Cox. Regression models and life-tables (with discus-sion).Journal of the Royal Statistical Society, B(34):187–220,1972.

[11] F. Douglis, A. Feldmann, B. Krishnamurthy, and J. C. Mogul.Rate of change and other metrics: A live study of the worldwide web. In1st USENIX Symposium on Internet Technolo-gies and Systems (USITS 1997), pages 16–31, 1997.

[12] J. Edwards, K. S. McCurley, and J. A. Tomlin. An adap-tive model for optimizing performance of an incremental webcrawler. InProceedings of the 10th International World WideWeb Conference (WWW10), pages 106–113, 2001.

[13] D. Fetterly, M. Manasse, M. Najork, and J. Wiener. A large-scale study of the evolution of web pages. InProceed-ings of the 12th International World Wide Web Conference(WWW12), pages 669–678, 2003.

[14] L. Gravano, K. C.-C. Chang, H. Garcıa-Molina, andA. Paepcke. STARTS: Stanford proposal for Internet meta-searching. InProceedings of the 1997 ACM SIGMOD Inter-national Conference on Management of Data (SIGMOD’97),pages 207–218, 1997.

[15] L. Gravano, H. Garcıa-Molina, and A. Tomasic.GlOSS: Text-source discovery over the Internet.ACM Transactions on Da-tabase Systems, 24(2):229–264, June 1999.

[16] L. Gravano, P. G. Ipeirotis, and M. Sahami. QProber: Asystem for automatic classification of hidden-web databases.ACM Transactions on Information Systems, 21(1):1–41, Jan.2003.

[17] P. G. Ipeirotis and L. Gravano. Distributed search over thehidden web: Hierarchical database sampling and selection.In Proceedings of the 28th International Conference on VeryLarge Databases (VLDB 2002), pages 394–405, 2002.

[18] P. G. Ipeirotis and L. Gravano. When one sample is notenough: Improving text database selection using shrinkage.In Proceedings of the 2004 ACM SIGMOD International Con-ference on Management of Data (SIGMOD 2004), 2004.

[19] F. Jelinek. Statistical Methods for Speech Recognition. TheMIT Press, 1999.

[20] L. Lim, M. Wang, S. Padmanabhan, J. S. Vitter, and R. C.Agarwal. Characterizing web document change. InThe Sec-ond International Conference on Web-Age Information Man-agement (WAIM 2001), pages 133–144, 2001.

[21] J. P. Marques De Sa.Applied Statistics. Springer Verlag, 2003.[22] J. J. More. The Levenberg-Marquardt algorithm: Implemen-

tation and theory. InNumerical Analysis, Lecture Notes inMathematics 630, Springer Verlag, pages 105–116, 1977.

[23] A. Ntoulas, J. Cho, and C. Olston. What’s new on the web?The evolution of the web from a search engine perspective. InProceedings of the 13th International World Wide Web Con-ference (WWW13), pages 1–12, 2004.

[24] C. Olston and J. Widom. Best-effort cache synchronizationwith source cooperation. InProceedings of the 2002 ACMSIGMOD International Conference on Management of Data(SIGMOD 2002), pages 73–84, 2002.

[25] L. Si and J. P. Callan. Relevant document distribution estima-tion method for resource selection. InProceedings of the 26thAnnual International ACM SIGIR Conference on Researchand Development in Information Retrieval, SIGIR 2003, pages298–305, 2003.

[26] D. M. Stablein, W. H. Carter, Jr., and J. W. Novak. Analysis ofsurvival data with nonproportional hazard functions.ControlClinical Trials, 2(2):149–159, June 1981.

[27] C. E. Wills and M. Mikhailov. Towards a better understandingof web resources and server responses for improved caching.In Proceedings of the Eighth International World Wide WebConference (WWW8), pages 1231–1243, 1999.