CHAPTER 7CHAPTER 7

Natural Language Natural Language ProcessingProcessing

Natural Language Natural Language ProcessingProcessing

Natural language processing is a branch of AI Natural language processing is a branch of AI whose goal is to facilitate communication whose goal is to facilitate communication between humans and computers using written between humans and computers using written (monologue) or oral (dialogue) of the human (monologue) or oral (dialogue) of the human languagelanguage

Natural language processing consist of two Natural language processing consist of two main areas :main areas : Natural language understanding : Natural language understanding :

to make computers understand instruction given in natural to make computers understand instruction given in natural languagelanguage

Natural language generation : Natural language generation : to make computers generate natural languageto make computers generate natural language

Study of LanguageStudy of Language Language: Language:

Written: Long-term record of knowledge from one generation Written: Long-term record of knowledge from one generation to anotherto another

Spoken: primary mean of coordinating day-to-day behavior Spoken: primary mean of coordinating day-to-day behavior with otherswith others

Natural (eg. Malay, English) vs. Artificial (Java, Prolog, Coding)Natural (eg. Malay, English) vs. Artificial (Java, Prolog, Coding) CommunicationCommunication

Use sign / natural language/ body languageUse sign / natural language/ body language Sender and ReceiverSender and Receiver

Studied in several disciplines:Studied in several disciplines: Linguist: structure of languageLinguist: structure of language PsychoLinguists: the process of human language production PsychoLinguists: the process of human language production

and comprehensionand comprehension Philosopher: how words can mean anything & how they Philosopher: how words can mean anything & how they

identify object in the world, what it means to have belief, identify object in the world, what it means to have belief, goals and intention, cognitive capabilities relate to languagegoals and intention, cognitive capabilities relate to language

CL: to develop a computational theory of Language (using CL: to develop a computational theory of Language (using the notions of algorithm & data structure from CS)the notions of algorithm & data structure from CS)

Application of NLUApplication of NLU

It represents It represents the meaning of sentences in the meaning of sentences in some representation languagesome representation language that can be that can be used later for further processing : used later for further processing : applicationsapplications

Text-based applicationsText-based applications Written text processing (books,newpaper, Written text processing (books,newpaper,

reports, manual, email, sms) = reading-based reports, manual, email, sms) = reading-based taskstasks

Searching/finding from database of textSearching/finding from database of text Extracting information from textExtracting information from text Translating documents (MT)Translating documents (MT) Summarizing texts for certain purposeSummarizing texts for certain purpose Story understandingStory understanding

Application of NLUApplication of NLU

Dialogue-based applicationsDialogue-based applications – involve human- – involve human-machine communication (spoken / machine communication (spoken / keyboard/mouse/ recognizer)keyboard/mouse/ recognizer) Q&A systems, eg. Query databaseQ&A systems, eg. Query database Automated customer service (phone)Automated customer service (phone) Tutoring systems (interaction with students)Tutoring systems (interaction with students) Spoken language control of machineSpoken language control of machine General cooperative problem-solving systemGeneral cooperative problem-solving system

Speech recognition <> Language understanding Speech recognition <> Language understanding system (only identify the word spoken from a system (only identify the word spoken from a given speech signal, not – how words are used to given speech signal, not – how words are used to communicate)communicate)

Discuss ELIZA systemDiscuss ELIZA system

ELIZA systemELIZA system Mid-1960s, MIT, a Therapist (system) & patient Mid-1960s, MIT, a Therapist (system) & patient

(user), Weizenbaum, 1966(user), Weizenbaum, 1966 Algorithm:Algorithm:

Has a Dbase of particular words (keywords)Has a Dbase of particular words (keywords) For each keyword -> store an integer, a pattern to match For each keyword -> store an integer, a pattern to match

against the input and a specification of the outputagainst the input and a specification of the output Given Sentence(S), find a keyword in S whose pattern Given Sentence(S), find a keyword in S whose pattern

matches Smatches S If > 1 keyword, pick the one with highest integer valueIf > 1 keyword, pick the one with highest integer value Use the output specification that is associated with this Use the output specification that is associated with this

keyword to generate next sentencekeyword to generate next sentence If there are No keywords, generate an innocuous If there are No keywords, generate an innocuous

continuation statement, eg: Tell me more, Go on. (figure continuation statement, eg: Tell me more, Go on. (figure 1.2, 1.3 Allen)1.2, 1.3 Allen)

Flow of Language Flow of Language AnalysisAnalysis

Natural language understanding follows the following stagesNatural language understanding follows the following stages Parsing Parsing

Involves the analysis of the syntactic structure of Involves the analysis of the syntactic structure of sentences. Parsing determines that a sentence follows the sentences. Parsing determines that a sentence follows the syntactic rules of the language. The output of the parsing syntactic rules of the language. The output of the parsing stage is a parse treestage is a parse tree

Semantic interpretation Semantic interpretation Involves the production of a representation (propositions, Involves the production of a representation (propositions,

conceptual graphs, frames) of the meaning of a sentenceconceptual graphs, frames) of the meaning of a sentence Incorporation of world knowledgeIncorporation of world knowledge

Involves the generation of an expanded representation of Involves the generation of an expanded representation of the sentence’s meaning for the complete understanding of the sentence’s meaning for the complete understanding of the sentencethe sentence

The output produced could then be used by application systems The output produced could then be used by application systems such as the database query handler, expert system interface, such as the database query handler, expert system interface, translator and HCI systems.translator and HCI systems.

Flow of Language Flow of Language AnalysisAnalysis

1.1. ParsingParsing

Sentence : Ahmad kicked the ballSentence : Ahmad kicked the ball

sentence

noun phrase verb phrase

noun

Ahmad

verb noun phrase

kicked determiner noun

the ball

Stages of Language Stages of Language AnalysisAnalysis

2. Semantic Interpretation2. Semantic Interpretation

Eg. Eg. Sentence : Ahmad kicked the ballSentence : Ahmad kicked the ball

ahmad ballkicked

Flow of Language Flow of Language AnalysisAnalysis

3. Incorporation of World Knowledge3. Incorporation of World Knowledge

Sentence : Ahmad kicked the ballSentence : Ahmad kicked the ball

ahmad ballkicked

football fieldat

human

is a

leg

haskicked

two

number

The Different Levels of The Different Levels of Language AnalysisLanguage Analysis

Phonetics/phonology Knowledge (K)-Phonetics/phonology Knowledge (K)- how words are related to the sounds that how words are related to the sounds that realize them realize them

Morphology KMorphology K– how words are constructed – how words are constructed from more basic meaning units called from more basic meaning units called morphememorpheme, the primitive unit of meaning in a , the primitive unit of meaning in a languagelanguage

Syntactic KSyntactic K– how words can be put together – how words can be put together to form correct sentences and determines to form correct sentences and determines what structural role each word plays in the what structural role each word plays in the sentence and what phrases are subparts (eg. sentence and what phrases are subparts (eg. POS) of what other phrasesPOS) of what other phrases

Levels of Language Analysis Levels of Language Analysis cont.cont.

Semantic KSemantic K– what words mean (lexical semantics) – what words mean (lexical semantics) and how these meanings combine in sentences to and how these meanings combine in sentences to form larger meaning, eg. sentence meanings form larger meaning, eg. sentence meanings (compositional semantic). Study of (compositional semantic). Study of context-context-independent meaningindependent meaning

Pragmatic KPragmatic K – concern how sentences are used in – concern how sentences are used in different situations and how use affects the different situations and how use affects the interpretation of the sentence (kind of polite and interpretation of the sentence (kind of polite and indirect language); indirect language); Context-dependent meaningContext-dependent meaning..

Discourse K-Discourse K- how the immediately preceding how the immediately preceding sentences affect the interpretation of the next sentences affect the interpretation of the next sentence. (pronoun and temporal aspects of sentence. (pronoun and temporal aspects of information conveyed)information conveyed)

World KWorld K – includes the general knowledge about the – includes the general knowledge about the structure of the world that language users must have structure of the world that language users must have in order to eg. Maintain a conversation. Includes what in order to eg. Maintain a conversation. Includes what each language user must know about the other user’s each language user must know about the other user’s beliefs and goals (discourse model)beliefs and goals (discourse model)

Morphological AnalysisMorphological Analysis The construction of words from more basic The construction of words from more basic

componentscomponents Large vocabulary system has a problem in Large vocabulary system has a problem in

representing lexiconrepresenting lexicon Reasons:Reasons:

A large number of words. Word can be formed A large number of words. Word can be formed in 2 ways:in 2 ways:

Inflectional form : go+es/ne = goes/gone (v -> v)Inflectional form : go+es/ne = goes/gone (v -> v) Derivational form : friend + ly = friendly (n -> adj)Derivational form : friend + ly = friendly (n -> adj)

Open Class words (noun, verb, adj, adv) & Open Class words (noun, verb, adj, adv) & Closed class words (articles, pronouns, Closed class words (articles, pronouns, prepositions)prepositions)

One SolutionOne Solution

Preprocess the input sentence into a Preprocess the input sentence into a sequence of sequence of morphemesmorphemes

A word may consist of a single A word may consist of a single morpheme, but often a word consists morpheme, but often a word consists of of a root form plus an affixa root form plus an affix

ExampleExample The word: The word: goesgoes

Root word : Root word : gogo Suffix : Suffix : eses (plural, present tense) (plural, present tense) Without pre-processing, a lexicon needs to list Without pre-processing, a lexicon needs to list

all the form of go, including: all the form of go, including: went, going, gonewent, going, gone With preprocessing, there would be ONE With preprocessing, there would be ONE

morpheme go that may combine with suffixes morpheme go that may combine with suffixes such as such as –ing, -es, –ing, -es, and and –en;–en; and ONE entry for and ONE entry for the irregular form: the irregular form: wentwent. Thus, the lexicon . Thus, the lexicon would only need to store TWO entries (would only need to store TWO entries (gogo and and wentwent) rather than FOUR.) rather than FOUR.

Other examples: Other examples: eaten, happiesteaten, happiest Some word cannot be decomposed into a Some word cannot be decomposed into a

root form and a suffix. Example is the root form and a suffix. Example is the word word seedseed

Finite State Transducer Finite State Transducer (FST)(FST)

A lexicon would have to encode what A lexicon would have to encode what forms are allowed with each rootforms are allowed with each root

One famous model is based on FSTsOne famous model is based on FSTs This model is like the This model is like the Finite State Finite State

MachinesMachines except that they except that they produce produce anan output given an inputoutput given an input

FST cont.FST cont.

An arc is labeled with a pair of symbolsAn arc is labeled with a pair of symbols For eg:For eg:

An arc labeled An arc labeled i:yi:y ; could only be followed if the ; could only be followed if the current input is the letter current input is the letter ii and the output is the and the output is the letter letter yy

FST can be used to concisely represents the FST can be used to concisely represents the lexicon and to transform the surface form lexicon and to transform the surface form of words into a sequence of morphemes.of words into a sequence of morphemes.

Show examples in Allen, pg 71-72Show examples in Allen, pg 71-72

FST cont.FST cont.

Arcs labeled by a single letter have that Arcs labeled by a single letter have that letter as both input and outputletter as both input and output

FST accepts the appropriate forms and FST accepts the appropriate forms and outputs the desired sequence of morphemesoutputs the desired sequence of morphemes

The entire lexicon can be encoded as an FST The entire lexicon can be encoded as an FST that encodes all the legal words and that encodes all the legal words and transforms them into morphemic sequencestransforms them into morphemic sequences

The different suffixes need only be defined The different suffixes need only be defined once, and all root forms that allow that suffix once, and all root forms that allow that suffix can point to the same nodecan point to the same node

Syntactic AnalysisSyntactic Analysis

Syntactic analysis involves analyzing the Syntactic analysis involves analyzing the structure of a sentence. This would require structure of a sentence. This would require checking whether the sentence is formed checking whether the sentence is formed according to a set of syntactic rules – grammaraccording to a set of syntactic rules – grammar

Parsing is an activity that takes a sentence as Parsing is an activity that takes a sentence as a set of linguistic token (words) and checks the a set of linguistic token (words) and checks the ordering of the tokens against a grammar. If ordering of the tokens against a grammar. If the sentence is derived from the grammar the sentence is derived from the grammar then parsing yields a parse tree of the then parsing yields a parse tree of the sentencesentence

Parsing using context free Parsing using context free grammarsgrammars

A context free grammar comprises rules that are A context free grammar comprises rules that are made up of two types of symbols – terminals and made up of two types of symbols – terminals and non terminalsnon terminals

Non – terminalsNon – terminals Terms that describe higher-level linguistic concepts such Terms that describe higher-level linguistic concepts such

as sentence, noun phrase verb phase. Non terminals as sentence, noun phrase verb phase. Non terminals need to be further expanded as they may contain other need to be further expanded as they may contain other non terminals and terminalsnon terminals and terminals

Terminals Terminals Terms that are usually individual words. Terminals Terms that are usually individual words. Terminals

cannot be further expanded. They never appear on the cannot be further expanded. They never appear on the right of a ruleright of a rule

Parsing using context free Parsing using context free grammarsgrammars

Parsing of a sentence begins with the non-terminals Parsing of a sentence begins with the non-terminals symbol symbol sentencesentence at the top of the parse tree at the top of the parse tree

Parsing progresses by way of substitutions according Parsing progresses by way of substitutions according to the rules of the grammar. to the rules of the grammar.

A legal substitution replaces the left-side of a rule A legal substitution replaces the left-side of a rule with the non-terminal (and terminal) symbols of the with the non-terminal (and terminal) symbols of the right side of the rule. In this case, higher level non-right side of the rule. In this case, higher level non-terminal symbols are replaces by lower level non-terminal symbols are replaces by lower level non-terminal symbols or terminals.terminal symbols or terminals.

Parsing is terminated when all the lower nodes of the Parsing is terminated when all the lower nodes of the parse tree comprise terminals, i.e. individual words.parse tree comprise terminals, i.e. individual words.

If the order of the terminals in the parse tree is the If the order of the terminals in the parse tree is the same as that of the original sentence when it is said same as that of the original sentence when it is said the sentence follows the rules of the language, i.e. is the sentence follows the rules of the language, i.e. is a legal sentencea legal sentence

Parsing a Natural Language Parsing a Natural Language SentenceSentence

Consider the grammar :Consider the grammar :1.1. sentence -> noun_phrase verb_phrasesentence -> noun_phrase verb_phrase2.2. noun_phrase -> nounnoun_phrase -> noun3.3. noun_phrase -> article nounnoun_phrase -> article noun4.4. verb_phrase -> verb verb_phrase -> verb 5.5. verb_phrase -> verb noun_phraseverb_phrase -> verb noun_phrase6.6. article -> thearticle -> the7.7. article -> aarticle -> a8.8. noun -> mannoun -> man9.9. noun ->carnoun ->car10.10. verb -> droveverb -> drove

Step 1Step 1

The man drove a The man drove a carcar sentence

noun_phrase verb_phrase

The man drove a car

Step 2Step 2

The man drove a The man drove a carcar sentence

noun_phrase verb_phrase

The man drove a car

article noun verb noun_phrase

Step 3Step 3

The man drove a The man drove a carcar

article

sentence

noun_phrase verb_phrase

The man drove a car

noun verb noun_phrase

article noun

Step 4Step 4

The man drove a The man drove a carcar

article

sentence

noun_phrase verb_phrase

noun verb noun_phrase

article noun

the man adrove car

Parsing a Natural Language Parsing a Natural Language SentenceSentence Derivation of the sentence “the man drove a car” Derivation of the sentence “the man drove a car”

according to the given grammaraccording to the given grammar

STRINGSTRING APPLY RULE #APPLY RULE #

1. 1. sentencesentence 11

2. 2. noun_phrasenoun_phrase verb_phrase verb_phrase 33

3. article noun 3. article noun verb_phraseverb_phrase 55

4. article noun verb 4. article noun verb noun_phrasenoun_phrase

33

5. 5. articlearticle noun verb article noun verb article nounnoun

66

6. 6. thethe nounnoun verb article noun verb article noun 88

7. 7. thethe manman verbverb article noun article noun 1010

8. 8. thethe manman drovedrove articlearticle noun noun 77

9. 9. thethe manman drovedrove thethe nounnoun 99

9. 9. thethe manman drovedrove thethe carcar Sentence parsedSentence parsed

Representation & Representation & UnderstandingUnderstanding

A crucial component of A crucial component of understanding involves computing a understanding involves computing a representation of the meaning of representation of the meaning of sentences and texts. (Reason: sentences and texts. (Reason: Senses & ambiguity)Senses & ambiguity)

Representations and Representations and UnderstandingUnderstanding

Computing a representation of the meaning of Computing a representation of the meaning of sentences and texts (Notion of representation)sentences and texts (Notion of representation)

Why can’t use the sentence itself as a representation Why can’t use the sentence itself as a representation of its meaning? Most words have multiple meanings of its meaning? Most words have multiple meanings ((SensesSenses). eg. Cook, bank, still (verb or noun), ). eg. Cook, bank, still (verb or noun), I made her duck.I made her duck. I saw a man in the park with a telescopeI saw a man in the park with a telescope

Thus, Thus, ambiguityambiguity inhibit system from making the inhibit system from making the appropriate inferences needed to model appropriate inferences needed to model understanding (need to resolve or disambiguate: eg. understanding (need to resolve or disambiguate: eg. Use Lexical disambiguation: POS, word-sense Use Lexical disambiguation: POS, word-sense disambiguation, ontology)disambiguation, ontology)

A program must explicitly consider each senses of a A program must explicitly consider each senses of a word to understand a sentenceword to understand a sentence

Represent meaning: must have a more Represent meaning: must have a more precise languageprecise language

Mathematics & Logic and the use of Mathematics & Logic and the use of formally specified representation formally specified representation languages (formal language) – notion of an languages (formal language) – notion of an atomic symbolatomic symbol

Useful representation languages have 2 Useful representation languages have 2 properties:properties: Precise and unambiguousPrecise and unambiguous Capture the intuitive structure of the natural Capture the intuitive structure of the natural

language sentences that it representslanguage sentences that it represents

RepresentationRepresentation SyntaxSyntax – indicates the way that words in the – indicates the way that words in the

sentence are related to each othersentence are related to each other The structure illustrates how the words are The structure illustrates how the words are

grouped together into phrases, what words grouped together into phrases, what words modify what other words and what words are of modify what other words and what words are of central importance in the sentencecentral importance in the sentence

It may identify the types relationships that exist It may identify the types relationships that exist between phrases and can store information about between phrases and can store information about the particular sentence structure that may be the particular sentence structure that may be needed for later processingneeded for later processing

Eg: 1. John sold the book to MaryEg: 1. John sold the book to Mary 2. The book was sold to Mary by John2. The book was sold to Mary by John

Representation cont.Representation cont.

Sentence Structure does not reflect Sentence Structure does not reflect its meaning (although have the same its meaning (although have the same syntactic structure, eg. the catch)syntactic structure, eg. the catch)

The intended meaning of a sentence The intended meaning of a sentence depends on depends on the situationthe situation in which in which the sentence is produced.the sentence is produced.

Context independent (the logical Context independent (the logical form,LF) vs. Context dependentform,LF) vs. Context dependent

Semantic Analysis:Semantic Analysis:The Logical Form, LFThe Logical Form, LF

LF = encodes possible word senses and identifies LF = encodes possible word senses and identifies the semantic relationships between the words the semantic relationships between the words and phrasesand phrases

Many of the relationships are captured using an Many of the relationships are captured using an abstract set of semantic relationships between abstract set of semantic relationships between the verb and its NPthe verb and its NP

Context Independent Context Independent Eg: Selling event, John is the seller, the book is Eg: Selling event, John is the seller, the book is

the object being sold and Mary is the buyer.the object being sold and Mary is the buyer. These roles are instances of the abstract These roles are instances of the abstract

semantic roles: AGENT, THEME and TO-POSS semantic roles: AGENT, THEME and TO-POSS (final possessor), respectively.(final possessor), respectively.

Show another example: invite - the ballShow another example: invite - the ball

The Final Meaning The Final Meaning RepresentationRepresentation

The final representation: a The final representation: a general Knowledge general Knowledge Representations languageRepresentations language, which is the system , which is the system uses to represent and reason about its application uses to represent and reason about its application domaindomain

The goal of contextual interpretation is to take a The goal of contextual interpretation is to take a representation of the structure of a sentence and representation of the structure of a sentence and its logical form, and to map this into some its logical form, and to map this into some expression in the KR that allow the system to expression in the KR that allow the system to perform the appropriate task in the domain.perform the appropriate task in the domain.

This is the language in which all the specific This is the language in which all the specific knowledge based on the application is representedknowledge based on the application is represented

Use FOPC, Semantic NetworkUse FOPC, Semantic Network Eg: Q-A application – a Q might map to a DB; Story Eg: Q-A application – a Q might map to a DB; Story

Understanding application – a sentence might map Understanding application – a sentence might map into a set of expressions that represent the into a set of expressions that represent the situation that the sentence describes.situation that the sentence describes.

Discourse & Pragmatic Discourse & Pragmatic AnalysisAnalysis

Context DependentContext Dependent Discourse Structure TheoryDiscourse Structure Theory

Discourse RelationsDiscourse Relations Discourse ModelDiscourse Model Discourse StructureDiscourse Structure

World KnowledgeWorld Knowledge Domain SpecificDomain Specific CorpusCorpus

DiscussionDiscussion

Use the following sentences to Use the following sentences to understand (to describes) the understand (to describes) the distinction between syntax, distinction between syntax, semantics and pragmatics:semantics and pragmatics: Language is one of the fundamental Language is one of the fundamental

aspects of human behavior and is a aspects of human behavior and is a crucial component of our lives.crucial component of our lives.

Green frogs have large noses.Green frogs have large noses. Green ideas have large noses.Green ideas have large noses. Large have green ideas noses.Large have green ideas noses.

Discuss the following Discuss the following sentences (ambiguity)sentences (ambiguity)

1. I made her duck. (5 meanings)1. I made her duck. (5 meanings)

2. I saw a man in the park with a 2. I saw a man in the park with a telescope. (2 meanings)telescope. (2 meanings)

Make your own ambiguous sentencesMake your own ambiguous sentences

BibliographyBibliography

ACL (Association for CL) / EACLACL (Association for CL) / EACL COLING (int conference of CL)COLING (int conference of CL) Applied NLPApplied NLP Workshop on Human Language TechnologyWorkshop on Human Language Technology Journal: CL & NLEJournal: CL & NLE IEEE ICASSP: Acoustic, Speech and Signal IEEE ICASSP: Acoustic, Speech and Signal

ProcessingProcessing IEEE Transactions on Pattern Analysis and IEEE Transactions on Pattern Analysis and

Machine IntelligenceMachine Intelligence IJCAI: Int Joint Conference on AIIJCAI: Int Joint Conference on AI Journal: AI, Computational Intelligence, Cognitive Journal: AI, Computational Intelligence, Cognitive

ScienceScience