natural language processing csci 4152/6509 | lecture 19
TRANSCRIPT
Natural Language ProcessingCSCI 4152/6509 — Lecture 19Natural Language Syntax
Instructor: Vlado KeseljTime and date: 14:35 – 15:55, 18-Nov-2021Location: Dunn 135 and On-line
CSCI 4152/6509, Vlado Keselj Lecture 19 1 / 27
Previous Lecture
Neural networks and deep learning (continued)I biological neuron, perceptron, feed-forward
networkI activation functions, softmax functionI neural language model, RNN, stacked and
bidirectional RNNI LSTM, self-attention, transformers
ParsingProlog introduction
I unification and backtracking
CSCI 4152/6509, Vlado Keselj Lecture 19 2 / 27
Variables in Prolog
Variable names start with uppercase letter orunderscore (‘ ’)
is a special, anonymous variable
Examples: ?- faster(rabbit,franklin).
Yes ;
...
?- faster(rabbit,X).
X = franklin ;
...
?- hare(X).
X = rabbit ;
CSCI 4152/6509, Vlado Keselj Lecture 19 3 / 27
Lists (Arrays), Structures.
Lists are implemented as linked lists. Structures (records)are expressed as terms. Examples:In program: person(john,public,’123-456’).
Interactively: ?- person(john,X,Y).
[] is an empty list.A list is created as a nested term, usually a specialfunction ‘.’ (dot):?- is_list(.(a, .(b, .(c, [])))).
CSCI 4152/6509, Vlado Keselj Lecture 19 4 / 27
List Notation
(.(a, .(b, .(c, []))) is the same as [a,b,c]
This is also equivalent to:[ a | [ b | [ c | [] ]]]
or[ a, b | [ c ] ]
A frequent Prolog expression is: [H|T]
where H is head of the list, and T is the tail, which isanother list.
CSCI 4152/6509, Vlado Keselj Lecture 19 5 / 27
Example: Calculating Factorial
factorial(0,1).
factorial(N,F) :- N>0, M is N-1, factorial(M,FM),
F is FM*N.
After saving in factorial.prolog and loading to Prolog:?- [’factorial.prolog’].
% factorial.prolog compiled 0.00 sec, 1,000 bytes
Yes
?- factorial(6,X).
X = 720 ;
CSCI 4152/6509, Vlado Keselj Lecture 19 6 / 27
Example: List Membership
Example (testing membership of a list):
member(X, [X|_]).
member(X, [_|L]) :- member(X,L).
CSCI 4152/6509, Vlado Keselj Lecture 19 7 / 27
Natural Language Syntax
Reading: Chapter 12 “Formal Grammars of English”
Syntax = sentence structure; i.e., study of the phrasestructure
syntaxis (Greek) — “setting out together,arrangement”
words are not randomly ordered — word order isimportant and non-trivial
There are “free-order” languages (e.g., Latin,Russian), but they are not completely order free.Two ways of organizing sentence structure:
I phrase structureI dependency structure
CSCI 4152/6509, Vlado Keselj Lecture 19 8 / 27
Phrase Structure
Phrase Structure Grammars or Context-FreeGrammarsA hierarchical view of sentence structure:
I words form phrasesI phrases form clausesI clauses form sentences
The main NLP problem in syntax is parsing; i.e.,given a sentence, find a correct structure of thesentence (typically a parse tree).
CSCI 4152/6509, Vlado Keselj Lecture 19 9 / 27
Example Sentence
the man took the book
CSCI 4152/6509, Vlado Keselj Lecture 19 10 / 27
Phrase Structure Parse Tree Examples
Phrase Structure parse trees are also calledContext-Free parse trees
This example is from the seminal Noam Chomsky’spaper in 1956:
Sentence
NP
the man
VP
Verb
took
NP
the book
CSCI 4152/6509, Vlado Keselj Lecture 19 11 / 27
Parse Tree Examples (2)
• Using Penn treebank tagset:S
NP
DT
the
NN
man
VP
VBD
took
NP
DT
the
NN
book
CSCI 4152/6509, Vlado Keselj Lecture 19 12 / 27
Parse Tree Examples (3)
• Sometimes we simplify a parse tree by ignoring a partof the structure, as in:
S
NP
DT
the
NN
man
VP
VBD
took
NP
the book
CSCI 4152/6509, Vlado Keselj Lecture 19 13 / 27
Parse Tree Examples (4)
• Another example:S
NP
DT
That
NN
man
VP
VBD
caught
NP
DT
the
NN
butterfly
CSCI 4152/6509, Vlado Keselj Lecture 19 14 / 27
Example Sentence
That man caught the butterfly with a net
CSCI 4152/6509, Vlado Keselj Lecture 19 15 / 27
Parse Tree Examples (5)
• Extending the previous example:S
NP
DT
That
NN
man
VP
VBD
caught
NP
DT
the
NN
butterfly
PP
IN
with
NP
DT
a
NN
net
CSCI 4152/6509, Vlado Keselj Lecture 19 16 / 27
Parse Tree Example (6)
• Representing parse trees in the bottom-up direction:
DT NN VBD DT NN IN DT
That man caught the butterfly with a
NN
net.
NP NP NP
PP
VP
S
CSCI 4152/6509, Vlado Keselj Lecture 19 17 / 27
Some Basic Notions in Context-Free Trees
Context-free trees, also called phrase structure trees,parse trees, syntactic trees
Node relations: root, leaf, parent (mother), child(daughter), sibling, ancestor, descendant, dominate
Context-free grammar
Consider for example the context-free grammarinduced by the last parse tree shown
CSCI 4152/6509, Vlado Keselj Lecture 19 18 / 27
Context-Free Grammars (CFG) Review
CFG is a tuple (V, T, P, S), where
V is a finite set of variables or non-terminals;e.g., V = {S ,NP ,DT ,NN ,VP ,VBD,PP , IN}T is a finite set of terminals, words, or lexemes;e.g., T = {That, man, caught, the, butterfly, with, a,net}P is a set of rules or productions in the formX → α, where X ∈ V and α ∈ (V ∪ T )∗; e.g.,P = {S → NP VP , NP → DT NN , DT →That, NP → ε}S is the start symbol S ∈ V
CSCI 4152/6509, Vlado Keselj Lecture 19 19 / 27
Some Notions about CFGs
CFG, also known as Phrase-Structure Grammar(PSG)
Equivalent to BNF (Backus-Naur form)
Idea from Wundt (1900), formally defined byChomsky (1956) and Backus (1959)
Typical notation (V, T, P, S); also (N,Σ, R, S)
Direct derivation, derivation
Example of a direct derivation: S ⇒ NP VP
Example of a derivation (beginning of):S ⇒ NP VP ⇒ DT NN VP ⇒ That NN VP ⇒. . .
Left-most and right-most derivation
CSCI 4152/6509, Vlado Keselj Lecture 19 20 / 27
Parse Tree Example (revisited)
DT NN VBD DT NN IN DT
That man caught the butterfly with a
NN
net.
NP NP NP
PP
VP
S
CSCI 4152/6509, Vlado Keselj Lecture 19 21 / 27
Leftmost Derivation Example
S ⇒ NP VP ⇒ DT NN VP ⇒ ThatNN VP ⇒ That manVP
⇒ That manVBD NP PP
⇒ That man caughtNP PP
⇒ That man caughtDT NN PP
⇒ That man caught theNN PP
⇒ That man caught the butterflyPP
⇒ That man caught the butterfly IN NP
⇒ That man caught the butterfly withNP
⇒ That man caught the butterfly withDT NN
⇒ That man caught the butterfly with aNN
⇒ That man caught the butterfly with a net
CSCI 4152/6509, Vlado Keselj Lecture 19 22 / 27
Some Notions about CFGs (continued)
Language generated by a CFG
Context-Free languages
Parsing task
Ambiguous sentences
Ambiguous grammars
Inherently ambiguous languages
CSCI 4152/6509, Vlado Keselj Lecture 19 23 / 27
Bracket Representation of a Parse Tree
(S (NP (DT That)
(NN man))
(VP (VBD caught)
(NP (DT the)
(NN butterfly))
(PP (IN with)
(NP (DT a)
(NN net)
) ) ) )
CSCI 4152/6509, Vlado Keselj Lecture 19 24 / 27
Some Notes on CFGs
Left-hand side (lhs) and right-had side (rhs) of a production
S︸︷︷︸lhs
→ NP VP︸ ︷︷ ︸rhs
Empty rule (epsilon rule, epsilon production): V → ε
Unit production: A→ B , where A and B are non-terminals
Notational variations:
I use of ‘|’: P → N | A P , instead of P → N , P → A PI BNF notation: P ::= N |API use of word ‘opt’: NP ::= DT NN PPopt
I or Kleene star: NP ::= DT NN PP∗
CSCI 4152/6509, Vlado Keselj Lecture 19 25 / 27
Using Prolog to Parse NL
Example: Consider a simple CFG to parse the followingtwo sentences: “the dog runs” and “the dogs run”The grammar is:S -> NP VP N -> dog
NP -> D N N -> dogs
D -> the VP -> run
VP -> runs
How to parse: the dog runs
CSCI 4152/6509, Vlado Keselj Lecture 19 26 / 27
Using Difference Lists: Idea
Consider rule: S -> NP VP
CSCI 4152/6509, Vlado Keselj Lecture 19 27 / 27