1

Generating Definitional Generating Definitional DescriptionsDescriptions

Kathleen McKeown

Sasha Blair-Goldensohn

2

What is a definitional What is a definitional description?description? Long answer response

More descriptive than dictionary definitions

Shorter than an encyclopedia entry

3

What is the Hajj?What is the Hajj?

The Hajj, or pilgrimage to Makkah [Mecca], is the central duty of Islam. More than two million Muslims are expected to take the Hajj this year. Muslims must perform the hajj at least once in their lifetime if physically and financially able. The Hajj is a milestone event in a Muslim's life. The annual hajj begins in the twelfth month of the Islamic year (which is lunar, not solar, so that hajj and Ramadan fall sometimes in summer, sometimes in winter). The Hajj is a week-long pilgrimage that begins in the 12th month of the Islamic lunar calendar. Another ceremony, which was not connected with the rites of the Ka'ba before the rise of Islam, is the Hajj, the annual pilgrimage to 'Arafat, about two miles east of Mecca, toward Mina. The hajj is one of five pillars that make up the foundation of Islam.

4

ApproachApproach

Integration of goal-driven and data-driven strategies

Goal-driven: Search for sentences conveying typical

definitional information using predicates Genus: ``The Hajj is a pilgrimage.’’

Data-driven Drawn from summarization Redundancies determine content

5

DefScriberDefScriber

Implemented prototype 3 predicates

Genus, species, non-specific definitional (NSD)

New method for identifying redundancies Heuristics from summarization for fluency

Robust, coherent definitional descriptions

Preliminary, subjective evaluation

6

9 Genus-Species Sentences

The Hajj, or pilgrimage to Makkah (Mecca), is the central duty of Islam.The Hajj is a milestone event in a Muslim 's life.The hajj is one of five pillars that make up the foundation of Islam. …

11 Web documents, 1127 total sentences

Input “What isthe Hajj?”

N = 20L = 6

383 Non-specific Definitional sentences

Sentence clusters,

importance ordering

DocumentRetrieval

Data-DrivenAnalysis

DefinitionCreation

PredicateIdentification

7

PredicatesPredicatesPredicate Example

Explicit Synonym

The hajj, or Pilgrimage to Mecca, is the central duty of Islam..

Etymology In Arabic, the word Hajj means a resolve of magnificent duty.

Genus, Species The Hajj is a milestone event in a Muslim 's life.

History Mohammed, the founder of Islam, started the tradition of the Hajj in 632 C.E.

Cause Doing a Hajj can cause all past sins of a muslim to be forgiven.

Target Partition(two instances)

Qiran, Tamattu’, and Ifrad are three different types of Hajj.

Others: Target Original, Target Anaphor, Target Synonym, Definitional

8

Implementation of PredicatesImplementation of Predicates

Training data 81 documents marked for predicate instances Drawn from retrieved documents for 14 terms in broad

categories

Feature-based classification Learned features to classify sentences

Term frequency, sentence position, words Definitional (81%), history (65%)

Pattern recognition Based on sentence structure Semantic typing Genus, species

9

ExampleSentence To Pattern

S

NP

VPThe Hindu Kush

PPNP

NP

between two major plates: the Indian

and Eurasian.

represents

the boundary

The original Genus-Species sentence A matching sentence

S

NP VP

The Hajj,

PPNP

NP

is

the central duty

NPNP

or

Pilgrimage to Makkah (Mecca),

of Islam.

MatchesInput

SentenceS

NP VP

NP

DT? TERM

contains

VP

FormativeVb

PPNP

NP

Genus PREP Species

The extracted partial syntax-tree pattern

contains

10

Data-drivenData-drivenAn integration of four techniquesAn integration of four techniques

Centroid based (Top N) Find information central to definitions NSD sentences sorted in order of distance from centroid

Clustering Clusters are now ordered by distance from centroid Take one sentence from each cluster

Local IDF weighting Terms “hajj”, “Mecca”, “Koran” problematic Weight distance metrics with local IDF from input sentences

Sentence ordering Pick sentences 2-N based on cohesion with previous

sentence

11

What is Sarin?What is Sarin?Genus/species sentences:

Sarin is one of the organophosphorus nerve gases developed during the Second World War.

Sarin is an extremely toxic substance that disrupts the nervous system, overstimulating muscles and vital organs.

Two of 51 clusters:

1. Although sarin can be made with publicly available chemicals, a sophisticated lab is needed to make sarin that is pure and long lasting. (1) Nerve agents such as Sarin have been the chemical weapons of choice since World War II. (2) Once produced, sarin presents both a storage and delivery problem.

2. The latest use of Sarin was in the nerve gas attack of a Tokyo subway by the Aum Shinrikyo cult in 1995. (5) Subway riders injured in Aum Shinrikyo sarin gas attack, Tokyo, March 20, 1995. (1)

12

What is Sarin? (full response)What is Sarin? (full response)

Sarin is one of the organophosphorus nerve gases developed during the Second World War. (10) The latest use of Sarin was in the nerve gas attack of a Tokyo subway by the Aum Shinrikyo cult in 1995. (5) A year earlier, the cult killed seven people in a sarin gas attack in the central Japanese city of Matsumoto. (1) If purer sarin had been released, particularly as an aerosol, the attack might have been much worse. (1) Under wet and humid weather conditions, Sarin degrades swiftly, but as the temperature rises up to a certain point, Sarins lethal duration increases, despite the humidity. (4) Just prior to the attack, Aum hurriedly produced a low-lethality batch of sarin. (1) Iraq began producing sarin in 1984 and admitted to possessing 790 tons of it in 1995. (8) Sarin was first produced by German scientists in 1938 ; although it was not used during the war, many have speculated that it could have given the Germans a decisive advantage. (2)

13

EvaluationEvaluation

Participated in AQUAINT-wide F02 TopN 52% f-measure w/recall 5 times precision

Survey based evaluation S03 Baselines: TopN, Simple cluster, Local IDF Defscriber: an integration of all four Questions

Rate structure/organization How relevant? Breadth of coverage? How redundant? Rate overall understanding

14

Evaluation HighlightsEvaluation Highlights

Using all features gives best results on structure/organization term understanding relevance

All methods significantly less on redundancy than TopN

Local IDF top on coverage

15

Mean and Standard Dev of Feature Ratings

0

0.5

1

1.5

2

2.5

Relevance Redundancy Structure Coverage Understanding

Mean

Stdev

Mean Difference Per Feature / Definition Type

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Rel

evan

ce

Red

unda

ncy

Stru

ctur

e

Cov

erag

e

Und

erst

andi

ng

TopN

SimpleCluster

PrincipledOrder

Local IDF

GenusSpecies

d

16

ConclusionsConclusions

Uses goal-driven approach to screen definitional from large volume of non-definitional information

Uses data-driven methods for exploiting redundancy on the web through summarization

Implemented prototype that scores well in evaluation

17

What’s Next?What’s Next?

More predicates

Summarization ``smoothing’’ techniques

In-depth evaluation