Information Retrieval and

Information Extraction

Akhil Deshmukh (06D5007)Narendra Kumar (06D05008)Kumar Lav (06D05012)

image: google

IR and IEIR is the process of selecting

documents from the collection.IE is retrieval of specific

information from the document.IE transform the unstructured

data into the structured.They together provide powerful

tool for text processing.

contd ....IR and IE can be seen in our day to

day life.The email-inboxThe searches we make on

internet.This seminar itself is a result of IR

and IE.

Information retrieval TasksInformation retrieval Tasks

•Search Engines (eg. Google and Yahoo).

•Feedback.

•Meta search.

•Data clustering and Visualization search results.

Clustering of search results

IR measuresIR measuresDivision of documents.

PrecisionRecallF-score.

Models for IRModels for IRLanguage Models In LM, the idea is to compute the conditional

probability P(q | d), i.e. the probability of generating a query q given the observation of a document d.

Vector Space Model (VSM) VSM is a algebraic model, for representing

objects as vectors of identifiers.

Model 1Given a query q, we want to compute the relevance of

document s. We find the probability -

p(r=1| q, d) , where r is the binary label for ‘relevant’.

Applying Bayes rule - p(r = 1| q, d) = p(d, q | r=1) p(r=1)

p(q , d)

We make two assumptions –

1. p(q | d, r =1) is computed from unigram model created from d,

2. p(q | d, r = 0) = p(q | r = 0)

Here we consider the ratio of log values –

log p(r = 1 | q, d) = log p(q | d, r=1) + log p( r=1 | d)

p(r = 0 | q, d) p(q | d, r=0) p(r=0 | d)

• Disadvantages

Language Models for IRLanguage Models for IR

Vector Space Model for IRVector Space Model for IREach document d is represented as a

scalar tf.idf, known as tf-idf scheme.

Where “ tf ” is normalize term frequency, and “ idf “ is the inverse document frequency.

tfw = c(w, d)/ maxv c(v, d) , c =count

idfw = log(N/nw) N = no. Of documents in

collection n = no. Of documents

containing w

so we have tf . idfw = tfw × idfw.

Fig: http://www.miislita.com/term-vector/term-vector-3.html

sim(d, q) = cos( )

= dT q = ∑j Wd,j x Wq,j

|d| |q| |d| |q|

here |d|= ∑j Wd,j 2 , |q| = ∑j Wq,j 2

and Wd,j = weight of term j in document d

i.e. Wd,j = tfd,j x idfj

Wq,j = weight of term j in query q

•Normalization ??

•Documents are ranked by their cosine similarity to q

Trigger Language Model

Here we calculate the probability p(q| Md). where Md is the language model for

document d. a) Estimation p(qi| Md) = tf(qi, d) , tf(qi, d)=

frequency of qi in d dld dld = total no. of

terms in d.

b) Zero counts p(qi | Md) = p(qi | Md) , if tf(qi, d) >0

= cfqi , otherwise

cs cfqi = raw count of qi in all

documents cs = total no. of terms in all

documentsc) Smoothing: calculate pavg(qi) .

d)Final term probability, p#(qi | Md) = p(qi | Md)(1-R) x pavg(qi)R , if tf(qi, d)>0

= cfqi ,

otherwise cs

e) Finally:

p(q | Md) = πqi∈q p#(qi | Md) x πqi∈q [1- p#(qi | Md)]

To minimize the smoothing risk, R = [1/(1+f)] x [f/(1+f)]tf

Where f is the mean term frequency of term, f = pavg(qi) x dld

Fig by : ZHANG Jun-lin, SUN Le QU Wei-min, SUN Yu-fang,2002

Information Extraction (IE) Task of locating specific

information from a natural language document.

Convert information from different text documents into database entries (structured format).

It runs on the top of IR systems.

Is IE and IR same ? IE systems do not understand the

text in the input documents.IR retrieves relevant documents,

IE extracts relevant informationIE sees text as structure, for IR

they are bag of words.

Both, together provide powerful tools for text processing.

Extraction Free text:

◦ Natural language. Articles in newspapers etc.

◦Uses NLP techniques. like POS tagging.◦Rules based on syntactic and semantic

information.Structured Text:

◦Text as in database or tables. ◦Extraction is easy and direct.◦Rules are based on the structure of the

text.

Extractioncontd...

Semi-structured:◦are between free and structured text.◦comprises of phrases.◦regular expressions are used.

Information may be :single-slot : single information.

eg. Micrsoft CEO Bill Gates

multi-slot : set of information. eg. Powai: 1 BHK Rs.7,000 p.m., 2 BHK Rs.

20,000 p.m.

IE System DesignTwo main approaches :Knowledge engineering :

◦grammars expressing rules are constructed by hand.

◦specific to application.◦rely on the skill of knowledge engineer.

Automatic training :◦based on training-testing paradigm.◦highly depend on the corpus for

accuracy.

Learning pattern by LP (LP)2

It learns rules by generalising over a set of examples marked via SGML tags in a training corpus

It induces two types of symbolic rules: tagging rules :

◦ inserts single SGML tag.◦ selects a tag in the training corpus and extracts

from the text a window of w words to the left and w words to the right.

◦ Windows are tagged by SGML tags.◦ Uses Context rules.◦ eg. Seminar at 9 pm. After tagging Seminar at

<time> 9 </time> pm

(LP)2 contd ..

Correction rules:◦shifts wrongly positioned tags to the

correct positions.◦Identical to the tagging rules except:

patterns match also the tags inserted by the tagging rules

actions shift misplaced tags rather than adding new ones.

(LP)2 contd ..

Generalisation:It generalises the initially induced

rules. Each generalisation is tested on the training corpus and the best k are retained. adding the wildcard characters like in 4

pm * pm Using POS tagger NLP knowledge

(LP)2 contd ..

Information extraction:Information is extracted in following

ways: best pool rules are used to tag. correction rules are applied to remove

errors. tags are validated i.e. uncoupled tags are

removed.

(LP)2 contd ..

NLP based.

[7]

WHISKSystem name

Text style

Multi-slot Need syntax

Need trim

WI Struct Yes No No

SRV Semi No No No

RAPIER Semi No No No

AutoSlog Free No Yes Yes

CRYSTAL Free Yes Yes Yes

CRYSTAL Semi Yes Yes Yes

WHISK Struct Yes No No

WHISK Semi Yes No No

WHISK Free Yes Yes No

IE Systems

[6]

WHISK:

Input Data.WHISK rule representation.WHISK Algorithm.

Input Data:Structured data Input:

eg: Profiles on web host, consider facebook, orkut etc.

Semi-Structured:eg: Ads in newspaper.

Free text:eg: Any random article.

Single-slot / Multi-slot

WHISK Rules:For Structured and semi-

structured text:eg: Consider an add for renting house. The corresponding rule will be of the form:

• ID:: 1

Pattern:: * ( Digit ) ‘R’ * ‚’$’ ( Number )

OutPut:: Rental {Bedrooms $1} {Price $2}

• Disjunction allowed:Bdrm = (brs | br | bds | bdrm | bd | bedrooms |

bedroom | bed)

• Thus the newly formed rule will be:ID:: 2

Pattern:: * ( Nghbr ) * ( Digit ) ‘ ‘ Bdrm * ‘$‘ ( Number )

Output:: Rental {Neighborhood $1} {Bedrooms $2} {Price $}

We have used Nghbr, what does it stand for?

Contd:

Extension of the rules for grammatical text:

• Pre-processing required, Syntactic analysis.@S[ {SUBJ @PN[ C. Vincent Protho ]PN , @PS[ chairman and chief excutive officer ]

of this maker of semiconductots, }

{VB @Passive was named @nam }

{PP to the additional post of @PS[ president ]PS , }

{REL_V succeeding @succeed @PN[ John W. Smith ]PN ,

who resigned @resign to pursue @pursu other interests. }

]@S 8910130051-1

• The corresponding rule pattern will be:ID:: 3

Pattern:: *(Person)* ‘@Passive‘ *F ‘named‘ * {PP *F (Position) * ‘@succeed ‚ (Person)

Output:: Succession {PersonIn $1} {Post $2} {PersonOut $3}

WHISK Algorithm:Features:

◦ A supervised learning algorithm◦ Tagging process is interleaved with the learning.◦ Presents the user with a batch of instances to tag.◦ Induces a set of rules from the expanded training

set.

Progress:◦ Begins with a reservoir of untagged instances.◦ At each iteration a set of untagged instances are

selected from the reservoir and presented to the user to annotate.

◦ The user adds a tag for each case frame to be extracted from the instance, ‘Training Instances’

Algorithm:@S[ Capitol Hill – 1 br twnhne. Fplc D/W W/D. Undrgrnd pkg incl

$675. 3 BR,upper flr of turn of ctry HOME. Incl gar, grt N. Hill loc $995. (206)

999-9999 <br><i> <font size=-2> (This ad last ran on 08/03/97.) </font> </i>

<hr>]@S 5@@TAGS Rental {Neighborhood Capitol Hill} {Bedrooms 1} {Price

675}@@TAGS Rental {Neighborhood Capitol Hill} {Bedrooms 3} {Price

995}

WHISK(Reservoir)RuleSet = NULLTraining = NULLRepeat at user‘s request

Select a batch of NewInst from Reservoir(User tags the NewInst)Add NewInst to TrainingDiscard rules with errors on NewInstFor each Inst in Training

For each Tag of InstIf Tag is not covered by RuleSetRule = GROW_RULE(Inst, Tag, Training)

Prune RuleSet

Anchoring the extraction Slot:

Empty Rule: “ * ( * ) * ( * ) * ( * ) * “Anchoring Slot 1:

Base_1: * ( Nghbr )Base_2: ‘@start‘ ( * ) ‘ -‘

Anchoring Slot 2:Base_1: * ( Nghbr ) * ( Digit )Base_2: * ( Nghbr ) * ‘- ‘ ( * ) ‘ br‘

Anchoring Slot 3:Base_1: * ( Nghbr ) * ( Digit ) * ( Number )Base_2: * ( Nghbr ) * ( Digit ) * ‘$‘ ( * ) ‘.‘

Contd:

GROW_RULE(Inst, Tag, Training)Rule = empty rule (terms replaced by wildcards)For i = 1 to number of slots in Tag

ANCHOR(Rule, Inst, Tag, Training, i)Do until Rule makes no errors on Training or no improvement in Laplacian

EXTEND_RULE(Rule, Inst, Tag, Training)

ANCHOR(Rule, Inst, Tag, Training, i)Base_1 = Rule + terms just within extraction iTest first i slots of Base_1 on TrainingWhile Base_1 does not cover Tag

EXTEND_RULE(Base_1, Inst, Tag, Training)Base_2 = Rule + terms just outside extraction iTest first i slots of Base_2 on TrainingWhile Base_2 does not cover Tag

EXTEND_RULE(Base_2, Inst, Tag, Training)Rule = Base_1If Base_2 covers more of Training than Base_1

Rule = Base_2

Laplacian = (e +1) / (n +1), where e is the number of errors and n is the number of

extractions made on the training set

Contd:

EXTEND_RULE(Rule, Inst, Tag, Training)Best_Rule = NULLBest_L = 1.0If Laplacian of Rule within error tolerance

Best_Rule = RuleBest_L = Laplacian of Rule

For each Term in InstProposed = Rule +TermTest Proposed on TrainingIf Laplacian of Proposed < Best_L

Best_Rule = ProposedBest_L = Laplacian of Proposed

Rule = Best_Rule

WHISK represents a form of hill climbing, and cannot guarantee that the rules it grows on are optimal.

Contd:

Interactive preparation of trainingSelecting informative instances: In each iteration of WHISK, a batch of instances is

selected from the reservoir of untagged instances, presented to the user for tagging, and then added to the training set.

Classes of untagged instances:◦ Instances covered by an existing rule◦ Instances that are near misses of a rule◦ Instances not covered by any rule

When to stop tagging?

ConclusionTill now the IR is done with the

assumption of term independence, work is going on to overcome this assumption.

Ability to use dynamic knowledge in IR.

Visual IR and IE will improve our ability to use the WWW.

ReferencesReferences1. Xiaojin Zhu, “Advanced NLP: Information Retrieval”,

CS838-1, WISC,2007.

2. Dr. Ian Soboroff, ”Information Retrieval”, CMSC 491I/691I UMBC, 2002

3. Nazli Goharian, “Language Models”, CS429, Illinois Institute of Technology, 2007

4. Fabio Ciravegna, “(LP)2, an Adaptive Algorithm for Information Extraction from Web-related Texts”,IJCAI-2001 Workshop on Adaptive Text Extraction and Mining , 2001.

5. Stephen Soderland, “Learning Information Extraction Rules for Semi-Structured and Free Text”,Machine Learning 34, 233–272 ,1999

6. Line Eikvil, “Information Extraction from world wide web”, A Survey. No. 945, ISBN 82-539-0429-0, pp. 39, 1999