Information Distance

More Applications

1. Information Distance from a Question to an

Answer

Question & Answer

Practical concerns: Partial matching does not satisfy triangle

inequality. When x is very popular, and y is not, x contains

a lot of irrelevant information w.r.t. y, then C(x|y) << C(y|x), and d(x,y) prefers y.

dmax does not satisfy universality. Neighborhood density -- there are answers that

are much more popular than others. Nothing to compress: a question and an

answer.

Partial matching

Triangle inequality does not hold:

d(man,horse) ≥ d(man, centaur) + d(centaur, horse)

Separate Irrelevant Information In max theory, we wanted smallest p, converting

x,y:

Now let’s remove redundant information from p:

We now wish to minimize q+s+t.

st

x p y

x q y

The Min Theory (Li, Int’l J. TCS, 2007, Zhang et al, KDD’2007)

Emin (x,y) = smallest program p needed to convert between x and y, but keeping irrelevant information out from p.

Formally: Emin (x,y) =min{|p| : U(x,p,r)=y, U(y,p,q)=x, |p|+|q|+|r| ≤

E(x,y) }

All other development similar to E(x,y). Define: min {C(x|y), C(y|x) }

dmin (x,y) = ----------------------- min {C(x),C(y)}

Fundamental Theorem II: Emin (x,y) = min { C(x|y), C(y|x) }

Other propertiesTheorem 1. dmin(x,y) ≤ dmax(x,y)

Theorem 2. dmin(x,y) is universal, does not satisfy triangle inequality is symmetric has required density properties: good guys

have more neighbors.

How to approximate dmax(x,y), dmin(x,y)

Each term C(x|y) may be approximated by one of the following:

1. Compression.2. Shannon-Fano code (Cilibrasi, Vitanyi): an

object with probability p may be encoded by –logp + 1 bits.

3. Mixed usage of (1) and (2) – in question and answer application. This is especially useful for Q&A systems.

Shannon-Fano Code Consider n symbols 1,2, …, N, with decreasing

probabilities: p1 ≥ p2 ≥, … ≥ pn. Let Pr=∑i=1..rpi. The binary code E(r) for r is obtained by truncating the binary expansion of Pr at length |E(r)| such that

- log pr ≤ |E(r)| < -log pr +1 Highly probably symbols are mapped to shorter

codes, and 2-|E(r)| ≤ pr < 2-|E(r)|+1

Near optimal: Let H = -∑rprlogpr --- the average number of bits needed to encode 1…N. Then we have

- ∑rprlogpr ≤ H < ∑r (-log pr +1)pr = 1 - ∑rprlogpr

Query-Answer System X. Zhang, Y. Hao, X. Zhu, M. Li, KDD’2007

Adding conditions to normalized information distance, we built a Query-Answer system.

The information distance naturally measures Good pattern matches – via compression Frequently occurring items – via Shannon-Fano

code Mixed usage of the above two.

Some comparisons

0% 10% 20% 30% 40% 50% 60% 70% 80%

QUANTA

AskED

NSIR

DFKI

Start

ASU

BrainBoost

Ask

LCC

Google

Yahoo

MSN Mean RAR

Top 1 Hit Rate

1. Benchmark is based on a common factoid QA test set of 109 questions that are provided by Text Retrieval Conference (TREC) sponsored by National Institute of Standard and Technology (NIST).

2. MRAR = (∑C(i)/i)/N = (C1 + C2*0.5 + C3*0.33 + C4*0.25 + C5*0.2)/109

2. Parameter-Free Data Mining (Keogh, Lonadi, Ratanamahatana, KDD’04)

Most data mining algorithms require setting many input parameters. Parameter-laden methods have 2 dangers: Wrong parameter setting leads to errors. Over fitting causes more problems.

Data mining algorithms should have ideally no parameters – thus, no prejudices, expectations, or presumptions.

Compared (a variant of) information distance with every time serious distance (51 of them) appeared in SIGKDD, SIGMOD, ICDM, ICDE, VLDB, ICML, SSDB, PKDD, PADDD during the previous decade.

Experiment on 18 pairs of time series(length 1000 each):

Q = correct /total

• Inf. Dist. Q=1• ¾ of measures, Q=0• Best of them, HMM, Q=0.33• Data cover: finance, science, medicine, industry• Data: all in SAX format

Anomaly detection (KLR, KDD’04) – algorithm: use divide and conquer to find a region that have large inf. dist. from other parts.

* All other methods produced wrong results – different from cardiologists.

Anomaly Detection (KLR KDD’04)

3. Identifying Multiword Expressions (Fan Bu, Tsinghua University)

Multiword expressions (MWEs) appear frequently and ungrammatically in English.

An MWE is a sequence of neighboring words whose meaning cannot be derived from the meaning of its components. Example: Kolmogorov complexity

Automatically identifying MWEs is a major challenge in computational linguistics.

Distance from an n-gram to its semantics Given an n-gram, let me define the

“semantics” of the n-gram to be the set of all web pages containing all the words in the n-gram.

The (plain) information distance simplifies to:

Dmax(n-gram g, its semantics) =

log (#pages(g)/#pages(g’s semantics))

Experiment on1529 idioms, 1798 compositional phrases

Complex name entity extraction:

4. Texture classification (Campana & Keogh, 2010)

Why are we interested in this

Their method Used, as information distance:

d(x,y) = [K(x|y) + K(y|x) ] / K(xy)

MPEG-1 format for images x, y. To do K(x|y), they created a movie of a pair of frames x and y. They used MPEG video compression (lossy).

Note that the algorithm has no access to color or shape information, this clustering is based only on texture.

Dictionnaire D'Histoire Naturelle by Charles Orbigny. 1849

 Ornaments from the Hand-Press period. The particularity of this period is the use of block of wood to print the ornaments on the books. The specialist historians want to record the individual instances of ornament occurrence and to identify the individual blocks. This identification work is very useful to date the books and to authenticate outputs from some printing-houses and authors. Indeed, numerous editions published in the past centuries do not reveal on the title page their true origin. The blocks could be re-used to print several books, be exchanged between the printing-houses or duplicated in the case of damage. Mathieu Delalandre

The algorithm can handle very subtle differences.

Clovis  Egyptian 

5. Gene Expression Dynamics (Nykter et al, PNAS, 2008)

Macrophage (white blood cells) 94 microarrays, 9941 differentially expressed

genes. Computed Normalized information distance

between two time-point measurements. Observed the underlying dynamical network

of the macrophage exhibits criticality. Somebody figure out what this is about – and

present in class.

6. Tree from metabolic network (Nykter et al, Phy. Rev. Lett. 2008)

Metabolic networks of 107 organisms from KEGG.

Normalized information distance is computed between each pair and a tree was generated.

Red – BacteriaBlue – archaeaGreen -- eukaryotes

Phylogenetic Compression (Ane-Sanderson, Syst. Biol. 2005)

Several interesting questions from bioinformatics: Is the parsimony tree the best tree? DNA sequence compression by sequence only

can be optimal? The authors propose a scheme to encode

the tree and data simultaneously to minimize the descriptive complexity of tree(s) plus data. Better compression More economical than the parsimony tree.