The Rise and Fall and Rise of Dependency Theory

Part II: The Rise from the Ashes

Ronald Fagin IBM Almaden Research Center

2

Dependencies were Considered Harmful

Dependencies were undesirable Except for keys and referential integrity constraints Database normalization eliminated dependencies

BCNF: each FD is a logical consequence of keys 4NF: each MVD is a logical consequence of keys 5NF: each JD is a logical consequence of keys

3

But then:

Dependencies took on a new, very positive role!

4

Data Integration and Data Exchange

Data integration:

Describe data in a global schema in terms of data in local schemas

Data exchange:

Describe data in a target schema in terms of data in a source schema, and actually produce the target database

5

Data Integration and Data Exchange

These are old, but recurrent, database problems

Phil Bernstein – 2003 “Data exchange is the oldest database problem”

EXPRESS: IBM San Jose Research Lab – 1977 for transforming data between hierarchical databases

The universal relation model is an early case of data integration

We will focus mainly on data exchange

6

Schema Mappings & Data Exchange

Source S Target T

Schema Mapping M = (S, T, Σ) Source schema S, Target schema T High-level, declarative assertions Σ that specify the

relationship between S and T

Data Exchange via the schema mapping M = (S, T, Σ):

Transform a given source instance I to a target instance J, so that <I, J> satisfy the specifications Σ of M

IJ

Σ

7

Schema Mapping Specification Language

The relationship between source and target is typically given by source-to-target tgds

(x) y (x, y) where

(x) is a conjunction of atoms over the source (x, y) is a conjunction of atoms over the target

(Student(s) Enrolls(s,c)) t g (Teaches(t,c) Grade(s,c,g))

There may also be target tgds and egds:

Grade(s,c,g)) Grade(s,c,g’)) (g = g’)

8

New Role of Dependencies

In data exchange, dependencies play a crucial role in describing how to transform data from one format to another

9

Solutions in Schema Mappings

Definition: Schema Mapping M = (S, T, Σ) If I is a source instance, then a solution for I is a

target instance J such that <I, J> satisfy Σ

Fact: In general, for a given source instance I, there may be no solutions at all or there may be multiple solutions; in fact there may be

infinitely many solutions

10

Universal Solutions in Data Exchange

[Fagin, Kolaitis, Miller, Popa – ICDT 2003] introduced universal solutions as the “best” solutions in data exchange

By definition, a solution is universal if it has homomorphisms to all other solutions Thus, it is a “most general” solution

Constants: entries in source instances

Variables (labeled nulls): entries besides constants in target instances

Homomorphism h: J1 → J2 between target instances: h(c) = c, if c is a constant If P(a1,…,am) is in J1,, then P(h(a1),…,h(am)) is in J2

11

How to Obtain a Universal Solution?

Answer: Use our old friend the chase!

Theorem [Fagin, Kolaitis, Miller, Popa – ICDT 2003]:If there is a solution, then the chase produces a universal solution

12

Standard schema mappings

[Fagin, Kolaitis, Miller, Popa – ICDT 2003] define a weakly acyclic set of tgds

[Deutsch, Tannen - ICDT 2003] have a slightly more restrictive notion

Let a standard schema mapping be one specified by s-t tgds, target egds, and a weakly acyclic set of target tgds.

Theorem [Fagin, Kolaitis, Miller, Popa – ICDT 2003]:

For standard schema mappings, the chase runs in polynomial time (data complexity)

13

Query Answering in Data Exchange

Schema S Schema T

IJ

Σ q

Question: What is the semantics of target query answering?

Definition: The certain answers of a query q over T on I

certain(q,I) = ∩ { q(J): J is a solution for I }

Note: It is the standard semantics in data integration

14

Computing the Certain Answers

Theorem [Fagin, Kolaitis, Miller, Popa – ICDT 2003]: Assume a standard schema mapping. Let q be a union of

conjunctive queries over the target.

If I is a source instance and J is a universal solution for I:

certain(q,I) = the set of all “null-free” tuples in q(J).

Hence, certain(q,I) is computable in polynomial time

1. Compute a universal solution J, using the chase, in polynomial time

2. Evaluate q(J) and remove tuples with nulls

15

Composing Schema Mappings

Given M12 = (S1, S2, 12) and M23 = (S2, S3, 23), derive a

schema mapping M13 = (S1, S3, 13) that is “equivalent” to

the sequence M12 and M23

Schema S1 Schema S2 Schema S3

M12 M23

M13

What does it mean for M13 to be “equivalent” to the composition of M12 and M23?

16

Semantics of Composition

13 has to have the property that:

<I1,I3> ⊨ 13 if and only if there exists I2 such that <I1,I2> ⊨ 12 and <I2,I3> ⊨ 23

17

Result of the composition

Question: If M12 and M23 are each specified by s-t tgds, what language is needed for specifying the composition of M12 and M23?

Answer: [Fagin, Kolaitis, Popa, Tan – PODS 2004]:

second-order tgds

18

Second-Order Tgds

Definition: Let S be a source schema and T a target schema.

A second-order tuple-generating dependency (SO-tgd) is a formula of the form:

f1 … fm( (x1(1 1)) … (xn(n n)) ),

where

fi is a function symbol

i is a conjunction of atoms over S and equalities of terms

i is a conjunction of atoms from T

Example: f (e( Emp(e) Mgr(e,f(e) ) e( Emp(e) (e=f(e)) SelfMgr(e) ) )

19

Composition and SO-Tgds

Theorem [Fagin, Kolaitis, Popa, Tan – PODS 2004]:

The composition of any finite sequence of schema mappings specified by s-t tgds can be specified by an SO-tgd

Conversely, every SO-tgd specifies the composition of a finite sequence of mappings that are each specified by s-t tgds.

Recently [Arenas, Fagin, Nash – ICDT 2010] showed that the sequence need only be of size 2

20

Composition with Target Constraints [Arenas, Fagin, Nash – ICDT 2010] defined s-t SO

dependencies, which generalize SO tgds by allowing not only target atoms but also equalities in the conclusion

Theorem [Arenas, Fagin, Nash – ICDT 2010] : • The composition of any finite sequence of standard schema

mappings can be specified by an s-t SO dependency (along with target egds and target tgds)

• Conversely, every s-t SO dependency specifies the composition of a finite sequence of standard schema mappings

– In fact, again, the sequence need only be of size 2 The chase procedure can be extended to schema mappings

specified by s-t SO dependencies, so that it produces universal solutions in polynomial time (data complexity)

21

Conclusions

Dependencies now play a crucial role in data integration and data exchange

We even have second-order dependencies, which have in fact been implemented in IBM Infosphere Data Architect.

Dependency theory is alive and well!

22

Extra slides

23

The Smallest Universal Solution Fact: Universal solutions need not be unique Question: Is there a “best” universal solution? Answer: [Fagin, Kolaitis, Popa – PODS 2003] took a

“small is beautiful” approach:

There is a smallest universal solution (if solutions exist); hence,

the most compact one to materialize Definition: The core of an instance J is the smallest

subinstance J’ that is homomorphically equivalent to J Fact:

Every finite relational structure has a core The core is unique up to isomorphism

24

Core: The smallest universal solution

Theorem [Fagin, Kolaitis, Popa – PODS 2003] :

All universal solutions have the same core

The core of the universal solutions is the smallest universal solution

If the target constraints are egds, then the core is polynomial-time computable (data complexity)

Theorem [Gottlob and Nash – PODS 2006]:

If the target constraints are egds and a weakly acyclic set of tgds, then the core is polynomial-time computable

25

Old Conclusions

Dependencies now play a crucial role in data integration and data exchange

We even have second-order dependencies, which have in fact been implemented in practice!

Lately, even probabilistic dependencies have been studied [Dong, Halevy, Yu – VLDB 2007] [Das Sarma, Dong, Halevy – SIGMOD 2008] [Fagin, Kimelfeld, Kolaitis – ICDT 2010]

Probabilistic dependencies on probabilistic databases

Dependency theory is alive and well!