1

Disentangling from Babylonian Confusion –

Unsupervised Language Identification

Chris Biemann, Sven TeresniakUniversity of Leipzig, Germany

Cicling-05, Mexico CityFebruary 18, 2005

2

Outline1. Review: Supervized Language Identification

2. Co-occurrence graphs• Co-occurrences• Visualizing co-occurrences

3. Chinese Whispers Algorithm• Finding words of the same language

4. Sorting text by language• Evaluation and limitations

3

Review: Supervized Language Identification

• needs training • Operates on letter n-grams or common words as features• Works almost error-free for texts from 500 letters on

Drawbacks:• Does not work for previously unknown languages• Danger of misclassifying instead of reporting „unknown“

Example: http://odur.let.rug.nl/~vannoord/TextCat/Demo • “xx xxx x xxx …” classified as Nepali• “öö ö öö ööö …” classified as Persian

4

Co-occurrence Statistics• Co-occurrence: occurrence of two or more words within

a well-defined unit of information (sentence, nearest neighbors, window...)

• Significant Co-occurrences reflect relations between words.

• Significance Measure (log-likelihood):

• In the following, sentence-based co-occurrence statistics are used.

( , ) log log !with number of sentences,

.

sig A B x k x kn

abxn

5

Co-occurrence Graphs • The entirety of all

significant co-occurrences is a co-occurrence graph G(V,E) withV: Vertices = WordsE: Edges (v1, v2, s) with v1, v2 words, s significance value.

• Co-occurrence graph is- weighted- undirected

• Small-world-property

6

Chinese Whispers - Motivation• (small-world) graphs consist of regions with a high

clustering coefficient and hubs that connect those regions

• The nodes in cluster regions should be assigned the same label per region

• Every node gets a label and whispers it to its neighbouring nodes. A node changes to a label if most of its neighbours whisper this label – or it invents a new one

• Under assumption of semantic closeness when being strongly connected there should emerge motivated clusters

7

Chinese Whispers AlgorithmAssign different labels to every node in the graph;

For iteration i from 1 to total_iterations {mutation_rate= 1/(i^2);For each word w in the graph {new_label of w = highest ranked label in neighbourhood of w;with probability mutation_rate: new_label of w = new class label;}

labels = new_labels;}

• graph clustering algorithm• linear time in the number of nodes • random mutation can be omitted but showed better results for

small graphs

8

Assigning New Labels

• Node A changes label from L1 to L3: Sum(L3)=9; Sum(L4)=8; Sum(L2)=5

• Other strategies result in different kinds of partitioning- threshold for share- weighting by node degrees

AL1->L3

DL2

EL3

BL4

CL3

58

6 3

9

Chinese Whispers on 7 Languages

10

Chinese Whispers on 7 languages

11

Assigning languages to sentences• Use word-based language identification tool• Largest clusters form word lists for different languages• A sentence is assigned a cluster label if

- it contains at least 2 words from the cluster and - not more words from another cluster

Questions for Evaluation:• up to what number of languages is that possible ?• How much can the corpus be biased ?

12

Evaluation: Mix of 7 languages

• Languages used: Dutch, Estonian, English, French, German, Icelandic and Italian

• At least 100 sentences per language are necessary for consistent clusters

Precision, Recall and F-value for 7-lingual corpora

0,96

0,97

0,98

0,99

1

100 1000 10000 100000

# of sentences per language

P/R

/F

Precision Recall F-value

13

Common mistakes• Unclassified:

- mostly enumerations of sport teams - very short sentences, e.g. headlines- legal act ciphers in estonian case, e.g. 10.12.96 jõust.01.01.97 - RT I 1996 , 89 , 1590

• Misclassified: mixed-language-sentences, likeFrench: Frönsku orðin "cinéma vérité" þýða "kvikmyndasannleikur“

English: Die Beatles mit "All you need is love".

14

Evaluation: Bilingual biased

• Language pairs used: English-Estonian, French-Italian, Dutch-German

• 1st language varied between 100-10‘000 sentences, 2nd language 100‘000 sentences

• Factor up to 200 does not result in deterioration• Above factor 200, the 1st language cluster is not

distinguishable in size from 2nd-language ‚noise‘

English noise in a 100K sentence Estonian Corpus

0,995

0,9975

1

100 1000 10000

# English sentences

Prec

isio

n/R

ecal

l

P Estonian P English R English

French noise in a 100K sentence Italian Corpus

0,925

0,95

0,975

1

100 1000

# French sentences

P/R

P Italian

R Italian

P French

R French

P total

R total

15

Conclusion• Unsupervized Language Identification is possible

• It fails to name the languages, but rather sorts them

• It works for previously undescribed languages, even for dialects

• Accurracy on sentences (here about 120 characters) is compareable to supervized approaches

• When classifying documents, there should be virtually no errors

• Time-linear graph-clustering algorithm

16

Questions?

THANK YOU!

17

Small Cooccurring Worlds Angenommene Struktur von Kookkurrenzgraphen: skalenfreie

Small Worlds• kurze Weglänge zwischen den Knoten• hoher Clustering Coeffizient• Power-Law-Verteilung von Knotengraden• Power-Law-Verteilung von Komponentengrößen

Knotengrad: Anzahl (ausgehender) KantenKomponente: Zusammenhängende Menge von Knoten

Power-Law-Verteilungen lassen sich einfach aufzeichnen.

18

Strategien zur FarbübernahmeEin Knoten ändert seine Farbe auf eine neue Farbe aus der

Umgebung, wenn diese (1) mit stärkster Signifikanzsumme auftritt. (top)(2) mit stärkster Signifikanzsumme gewichtet nach

Knotengrad auftritt (a - linear, b - logarithmisch) (dist)(3) mit stärkster Signifikanzsumme auftritt und anteilig über

einer gewissen Schwelle liegt (vote <x>)

AL1

DL2

EL3

BL4

CL3

58

6 3

deg=1deg=2

deg=3deg=5

deg=4

Beispiel: Einfärben von A(1): Sum(L3)=9; Sum(L4)=8; Sum(L2)=5(2a): wSum(L2)=5; wSum(L4)=4; wSum(L3)=2.2(2b): wSum(L4)=7,28; wSum(L3)=5,51; wSum(L2)=3,46(3): nSum(L3)=0,409; nSum(L4)=0,363; nSum(L2)=0,227

19

7 Clusters – 7 languages68701:(3792): [...] a-t-elle, a-t-il, a-t-on, aanval, abandonné, abattu, abattus, aborder, abords, abouti, absolu, absolue, acceptent,

accepter, accepté, accessibles, accession, accord, accords, accordé, accordée, accusation, accusations, accuse, accusé, accusée, acheter, achevé, achevée, acte, actes, actifs, action, actionnaires, actions, actions-suicides, activement, activiste, activistes, activités, actuelle, actuellement, adeptes, adjoint, admettre, administratif, admis, [...]

80266:(3616) [...] a, abandoned, able, ablösen, aboard, abortion, abortions, about, above, abroad, absence, absolute, absolutely, accepted, accessible, accident, accidents, acclaim, according, accounting, accused, accusing, acid, acidic, acknowledged, acquire, acquistare, acre, acres, across, act, acting, active, activist, activists, acts, actually, added, addicts, adding, additional, address, administration, administrator, admitted, adopt, adopted, adoption, adults, advance,

68952:(3312) [...] abbandonato, abbastanza, abbia, abbiamo, abbiano, abile, abitante, abitanti, abitazioni, abruzzesi, accade, accaduto, accenno, accertare, accertato, accesso, accoglienza, accolto, accordi, accordo, accorta, accorti, accusa, acquisito, ad, addetti, addirittura, addosso, adesso, adottata, aereo, affari, affermato, affetto, affidare, affidato, affiliati, affiliato, affonda, affrontare, affronteranno, agenti, agenzie, agevolare, aggiunge, aggiungere, aggiunto, agli [...]

75760:(3249) [...] af, afar, afgreiðslutíma, afl, afla, aflaheimilda, aflaheimildum, aflann, aflaverðmætið, afli, aflinn, aflýst, afnema, afnotagjalda, afnotagjaldið, afnotagjöld, afnotagjöldin, afnotagjöldum, afrek, afráðið, afstýra, aftur, afurðaverðs, afurðum, aka, al-Qaeda, al-Zawahri, ala, aldar, aldrei, aldri, aldur, allan, allar, allir, allra, allri, alls, allt, alltaf, alltof, allur, almannafjölmiðla, almannamiðla, almenna, almennt, altari, alveg, andvirði, annan, annar, annarra [...]

81089:(2894) [...] an, aandacht, aandachtsgebied, aangehouden, aangekeken, aangenomen, aangepakt, aangesloten, aangevuld, aangewezen, aangezien, aanleiding, aanmerking, aanpak, aanslag, aansluiting, aantal, aantrekkelijk, aanvankelijk, aanvragen, aanwezige, aanwezigheid, aanwijsbaar, aanwijzingen, aanzien, aarde, aardige, abortuspil, abortuswetgeving, acceptabel, achtduizend, achter, achtergrond, achterhalen, achterover, actie, actieve, actuele [..]

68872:(2791) [...] ab, abend, aber, abermals, abgebaut, abgelaufenen, abgeschlossen, abgeschlossenen, abschneiden, abzuwarten, acht, achtzehn, achtziger, afghanischen, akzeptieren, allein, allem, allen, allenfalls, aller, allerdings, allgemein, allgemeinen, alt, alte, alten, alter, am, amerikanische, amerikanischen, amerikanischer, anderen, anderer, anerkennen, angedroht, angegeben, angehende, angekündigt, angenommen, angesichts, angestellt, angetastet [...]

72602:(2247) [...] aadress, aadressi, aadressil, aadressina, aasta, aastaaruande, aastabilansi, aastabilanss, aastaks, aastal, aastas, aastat, aastate, aeg, aegumistähtaeg, aga, agressiooni, ainuaktsionäri, ainult, ainuõigus, ajaks, ajal, ajast, ajutiselt, akt, aktid, aktide, aktsia, aktsiad, aktsiaid, aktsiakapital, aktsiakapitali, aktsiakapitalist, aktsiaraamatusse, aktsiaselts, aktsiaseltsi, aktsiaseltsiga, aktsiaseltsil, aktsiaseltsile, aktsiat, aktsiate, aktsiatega [...]

60154:(195) [...] afferma, créée, dimostrano, dom, domicilio, dovranno, escl, esclusi, escluso, feriale, festivi, festivo, fóru, gleðilegs, gratuita, gravidanza, incarico, intero, inv, io, jr, jäetud, jõust, liðna, lõige, lõiked, nere, næstir, pagamento, punktides, sab, saper, scenario, servono, sindacale, sjálfsm, socc, soccorso, spiegato, spilurum, techniques, ventanni, warnte, zuständige [...]

[...]84023:(3) Inter, Mailand, Ronaldo