Language Model in Turkish IR

Melih KandemirF. Melih Özbekoğlu

Can ŞardanÖmer S. Uğurlu

Outline

• Indexing problem and proposed solution

• Previous Work

• System Architecture

• Language Modeling Concept

• Evaluation of the System

• Conclusion

Indexing Problem

• “A Language Modeling Approach to Information Retrieval” Jay M. Ponte and. W. Bruce Croft, 1998

• Indexing model is important at probabilistic retrieval model

• Current models do not lead to improved retrieval results

Indexing Problem

• Failure because of unwarranted assumptions:

• 2-Poisson model– “elite” documents

• N-Poisson model– Mixture of more than 2 Poission distributions

Proposed Solution

• Retrieval based on probabilistic language modeling

• Language model refers to probabilistic distribution that captures statistical regularities of the generation of language

• A language model is inferred for each document

Proposed Solution

• Estimate probability of generating the query

• Documents are ranked according to these probabilities

• Users have a reasonable idea of terms

• tf, idf are integral parts of language model

Previous Work

• Robertson–Sparck Jones model and Croft–Harper model– They focus on relevance

• Fuhr integrated indexing and retrieval models.– Used statistics as heuristics

• Wong and Yao used utility theory and information theory

Previous Work

• Kalt’s approach is the most similar– Maximum likelihood estimator is used– Collection statistics are integral parts of the

model– Documents are members of language classes

System OverviewApplication Server

Index DB (PostgreSQL)

LM-Search

JDBC

UI

Document Repository

Query EvaluatorIndexer

USER

Different Resultsets

System Architecture

Document Repository

Stemming & Term Selection

No Stemming

First 5

Lemmatiser Inverted Index

Generation

tf.idf

Language Model

Index DB

tf.idf vs. Language model

Different Resultsets GUI for seeing differences

between results

tf.idf LM LM

tf.idf

Vocabulary Extraction• No stemmer

– Turkish is aggluntinative– Expectation: low precision

• First 5 characters– As effective as more complex solutions

• Lemmatiser: – Expectation: high precision. – Zemberek2 (MPL license)

• Open Source Software• Java Interface, easy to use• Find stems of the words• First valid stem will be used,• Word sense disambiguation (using wordnet or

POS) may be added in the future

Stemming & Term Selection

No Stemming

First 5

Lemmatiser

Database

Index DB

Language Modeling : Inverted Index Implementation

An example inverted index for m terms :t1 cft d1 P(t1|Md1) tf1d1 dn P(t1|Mdn) tf1dn

t2 cft d3 P(t2|Md3) tf2d2 dp P(t2|Mdp) tf2dp

tm cft d4 P(t4|Md4) tfmd4 dk P(tm|Mdk) tfmdk

…

…

…

…

If a document does not contain term then probability can be calculated using cft

ft = mean term frequency= mean probability of t in documents containing it

cft =frequency of t in all documents

The Baseline Approach : tf.idf

We will use the traditional tf.idf term weighting approach as the baseline modelRobertson’s tf score

Standard idf score

Language Modeling : Definition• An alternative approach to indexing and retrieval• Definition of Language Model: A probability distribution that captures the statistical regularities of the generation of language• Intuition Behind : Users have a reasonable idea of terms that are likely to occur in documents of interest and will choose query terms that distinguish these documents from others in the collection

Language Modeling : The Approach

The following assumptions are not made :• Term distributions in the documents are parametric• Documents are members of pre-defined classes

•“Query generation probability” rather than “Probability of relevance”

Language Modeling : The Approach

P(t | Md) : Probability that the term t is generated by the language model of document Md

Language Modeling : Theory

Maximum likelihood estimate of the probability of term t under the term distribution for document d:

tf(t,d) : raw term frequency in document d

dld : total number of terms in the document

Language Modeling : Theory

An additional more robust estimate from a larger amount of data :

pavg : Mean probability of term t in documents containing it

dft : Number of documents that contain term t

Language Modeling : Theory

The risk function :

: Mean term frequency of term t in documents which contains it.

Language Modeling : The Ranking Formula

Let the probability of term t being produced by document d given the document model Md :

The probability of producing Q for a given document model Md is :

Language Modeling : Inverted Index Implementation

An example inverted index for m terms :t1 cft d1 P(t1|Md1) dn P(t1|Mdn)

t2 cft d3 P(t2|Md3) dp P(t2|Mdp)

tm cft d4 P(t4|Md4) dk P(tm|Mdk)

…

…

…

…

Evaluation

• Perform recall/precision experiments– Recall/precision results – Non-interpolated average precision – Precision figures for the top N documents

• For several values of N• R-Precision

Other Metrics

• Compare the baseline (tf.idf) results to our language model.– Percent Change between two result sets– I / D

• I : count of queries performance improved• D : count of queries performance changed

Document Repository

• Milliyet (2001-2005)• XML file ( 1.1 GB )• 408304 news • Ready for indexing

• XML Schema ......(FIXME)

Document Source

Summary

• Indexing and stemming– Zemberek2 lemmatiser– Java environment

• Data– News archive from 2001 to 2005, from Milliyet

• Evaluation– Methods will be compared according to

performance over recall/precision values

Conclusion

• First language modelling approach to Turkish IR

• The LM approach– Non-parametric– Less assumptions– Relaxed

• Expected a better performance than baseline tf.idf method

Thanks for listening…

Any Questions?