LING 581: Advanced Computational Linguistics

Lecture NotesFebruary 2nd

From last time…

Homework ExerciseUse the bracketing guides and choose three “interesting” constructions Find all occurrences in the WSJ PTB

Homework Exercise

• 581 Homework rules– Due next lecture– Present your findings in class (slides)

Today’s Lecture

• More on Bikel Collins parser• Evaluation: EVALB• Homework

Bikel Collins

• Paper– Daniel M. Bikel. 2004. Intricacies of Collins’

Parsing Model. (PS) (PDF) in Computational Linguistics, 30(4), pp. 479-511.

– http://www.cis.upenn.edu/~dbikel/papers/collins-intricacies.pdf

Bikel Collins

Observations from Training Data

• (mod ((with IN) (milk NN) PP (+START+) ((+START+ +START+)) NP-A NPB () false right) 1.0)– modHeadWord (with IN)– headWord (milk NN)– modifier PP– previousMods (+START+) – previousWords ((+START+ +START+))– parent NP-A– head NPB– subcat ()– verbIntervening false– side right

• (mod ((+STOP+ +STOP+) (milk NN) +STOP+ (PP) ((with IN)) NP-A NPB () false right) 1.0)– modHeadWord (+STOP+ +STOP+) – headWord (milk NN)– modifier +STOP+– previousMods (PP)– previousWords ((with IN))– parent NP-A– head NPB– subcat ()– verbIntervening false– side right

Frequency 1 observed data for:(NP (NP (DT a)(NN milk))(PP (IN with)(NP (ADJP (CD 4)(NN %))(NN butterfat))))

Observations from Training Data

76.8% singular events 94.2% 5 or fewer occurrences

Example of Brittleness

• “Milk” example …

EVALB• How to evaluate parsing accuracy?

– count bracketing matches– (LR) Bracketing recall =

(number of correct constituents) ---------------------------------------- (number of constituents in the goldfile)

– (LP) Bracketing precision = (number of correct constituents) ----------------------------------------

(number of constituents in the parsed file)

• Program is called evalb– http://nlp.cs.nyu.edu/evalb/– written in C– get it to compile on your system (Makefile)

EVALB

file evalbevalb: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.0, not

stripped

source file

Use command make to build executable on your machine

EVALB• Example (on MacOSX)douglass-dhcp8:EVALB sandiway$ makegcc -Wall -g -o evalb evalb.cevalb.c:25:20: error: malloc.h: No such file or directoryevalb.c: In function ‘main’:evalb.c:379: warning: pointer targets in passing argument 1 of ‘fgets’ differ in signedness

EVALB

On MacOSXdelete this line

EVALB• douglass-dhcp8:EVALB sandiway$ make• gcc -Wall -g -o evalb evalb.c• evalb.c: In function ‘main’:• evalb.c:378: warning: pointer targets in passing argument 1 of ‘fgets’ differ in signedness• evalb.c:385: warning: pointer targets in passing argument 1 of ‘__builtin___strcpy_chk’ differ in signedness• evalb.c:385: warning: pointer targets in passing argument 2 of ‘__builtin___strcpy_chk’ differ in signedness• evalb.c:385: warning: pointer targets in passing argument 1 of ‘__inline_strcpy_chk’ differ in signedness• evalb.c:385: warning: pointer targets in passing argument 2 of ‘__inline_strcpy_chk’ differ in signedness• evalb.c:388: warning: pointer targets in passing argument 1 of ‘fgets’ differ in signedness• evalb.c:403: warning: pointer targets in passing argument 1 of ‘fgets’ differ in signedness• evalb.c: In function ‘calc_result’:• evalb.c:878: warning: pointer targets in passing argument 2 of ‘__builtin___strncpy_chk’ differ in signedness• evalb.c:878: warning: pointer targets in passing argument 2 of ‘__inline_strncpy_chk’ differ in signedness• evalb.c:892: warning: pointer targets in passing argument 2 of ‘__builtin___strncpy_chk’ differ in signedness• evalb.c:892: warning: pointer targets in passing argument 2 of ‘__inline_strncpy_chk’ differ in signedness• evalb.c:904: warning: pointer targets in passing argument 2 of ‘__builtin___strncpy_chk’ differ in signedness• evalb.c:904: warning: pointer targets in passing argument 2 of ‘__inline_strncpy_chk’ differ in signedness• evalb.c:932: warning: pointer targets in passing argument 2 of ‘__builtin___strncpy_chk’ differ in signedness• evalb.c:932: warning: pointer targets in passing argument 2 of ‘__inline_strncpy_chk’ differ in signedness• douglass-dhcp8:EVALB sandiway$ file evalb• evalb: Mach-O 64-bit executable x86_64

Can ignore c compiler warnings

EVALB[6] THE PARAMETER (.prm) FILE

The .prm file sets options regarding the scoring method. COLLINS.prm gives the same scoring behaviour as the scorer used in (Collins 97). The options chosen were:

1) LABELED 1to give labelled precision/recall figures, i.e. a constituent must have the same span *and* label as a constituent in the goldfile.

2) DELETE_LABEL TOP Don't count the "TOP" label (which is always given in the output of tgrep) when scoring.

3) DELETE_LABEL -NONE- Remove traces (and all constituents which dominate nothing but traces) when scoring. For example.... (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1)))) (. .)))would be processed to give.... (VP (VBD reported)) (. .)))

4)DELETE_LABEL , -- for the purposes of scoring remove punctuationDELETE_LABEL :DELETE_LABEL ``DELETE_LABEL ''DELETE_LABEL .

5) DELETE_LABEL_FOR_LENGTH -NONE- -- don't include traces when calculating the length of a sentence (important when classifying a sentence as <=40 words or >40 words)

6) EQ_LABEL ADVP PRT

Count ADVP and PRT as being the same label when scoring.

EVALB• To run the scorer:• > evalb -p Parameter_file Gold_file Test_file • For example to use the sample files:• > evalb -p sample.prm sample.gld sample.tst

EVALBGold standard:• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A-SBJ-1 (P this)) (B-WHATEVER (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A

(P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))))

• (S (A (P this)) (B (Q is) (A (R a) (T test))) (-NONE- *))• (S (A (P this)) (B (Q is) (A (R a) (T test))) (: *))

Test:• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (C (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (U test))))• (S (C (P this)) (B (Q is) (A (R a) (U test))))• (S (A (P this)) (B (Q is) (R a) (A (T test))))• (S (A (P this) (Q is)) (A (R a) (T test)))• (S (P this) (Q is) (R a) (T test))• (P this) (Q is) (R a) (T test)• (S (A (P this)) (B (Q is) (A (A (R a) (T test)))))• (S (A (P this)) (B (Q is) (A (A (A (A (A (R a) (T test))))))))• (S (A (P this)) (B (Q was) (A (A (R a) (T test)))))• (S (A (P this)) (B (Q is) (U not) (A (A (R a) (T test)))))• (TOP (S (A (P this)) (B (Q is) (A (R a) (T test)))))• (S (A (P this)) (NONE *) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (S (NONE abc) (A (NONE *))) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (TT test))))• (S (A (P This)) (B (Q is) (A (R a) (T test))))• (S (A (P That)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))))• (S (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T

test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))) (A (P this)) (B (Q is) (A (R a) (T test))))

• (S (A (P this)) (B (Q is) (A (R a) (T test))) (-NONE- *))• (S (A (P this)) (B (Q is) (A (R a) (T test))) (: *))

EVALBResults: Sent. Matched Bracket Cross Correct Tag

ID Len. Stat. Recal Prec. Bracket gold test Bracket Words Tags Accracy============================================================================ 1 4 0 100.00 100.00 4 4 4 0 4 4 100.00 2 4 0 75.00 75.00 3 4 4 0 4 4 100.00 3 4 0 100.00 100.00 4 4 4 0 4 3 75.00 4 4 0 75.00 75.00 3 4 4 0 4 3 75.00 5 4 0 75.00 75.00 3 4 4 0 4 4 100.00 6 4 0 50.00 66.67 2 4 3 1 4 4 100.00 7 4 0 25.00 100.00 1 4 1 0 4 4 100.00 8 4 0 0.00 0.00 0 4 0 0 4 4 100.00 9 4 0 100.00 80.00 4 4 5 0 4 4 100.00 10 4 0 100.00 50.00 4 4 8 0 4 4 100.00 11 4 2 0.00 0.00 0 0 0 0 4 0 0.00 12 4 1 0.00 0.00 0 0 0 0 4 0 0.00 13 4 1 0.00 0.00 0 0 0 0 4 0 0.00 14 4 2 0.00 0.00 0 0 0 0 4 0 0.00 15 4 0 100.00 100.00 4 4 4 0 4 4 100.00 16 4 1 0.00 0.00 0 0 0 0 4 0 0.00 17 4 1 0.00 0.00 0 0 0 0 4 0 0.00 18 4 0 100.00 100.00 4 4 4 0 4 4 100.00 19 4 0 100.00 100.00 4 4 4 0 4 4 100.00 20 4 1 0.00 0.00 0 0 0 0 4 0 0.00 21 4 0 100.00 100.00 4 4 4 0 4 4 100.00 22 44 0 100.00 100.00 34 34 34 0 44 44 100.00 23 4 0 100.00 100.00 4 4 4 0 4 4 100.00 24 5 0 100.00 100.00 4 4 4 0 4 4 100.00============================================================================ 87.76 90.53 86 98 95 16 108 106 98.15

=== Summary ===

-- All --Number of sentence = 24Number of Error sentence = 5Number of Skip sentence = 2Number of Valid sentence = 17Bracketing Recall = 87.76Bracketing Precision = 90.53Complete match = 52.94Average crossing = 0.06No crossing = 94.122 or less crossing = 100.00Tagging accuracy = 98.15

-- len<=40 --Number of sentence = 23Number of Error sentence = 5Number of Skip sentence = 2Number of Valid sentence = 16Bracketing Recall = 81.25Bracketing Precision = 85.25Complete match = 50.00Average crossing = 0.06No crossing = 93.752 or less crossing = 100.00Tagging accuracy = 96.88

EVALB

http://www.aclweb.org/anthology-new/H/H91/H91-1060.pdf

Paper on evalb

EVALB[5] HOW TO CREATE A GOLDFILE FROM THE PENN TREEBANK

The gold and parsed files are in a format similar to this:

(TOP (S (INTJ (RB No)) (, ,) (NP (PRP it)) (VP (VBD was) (RB n't) (NP (NNP Black) (NNP Monday))) (. .)))

To create a gold file from the treebank:

tgrep -wn '/.*/' | tgrep_proc.prl

will produce a goldfile in the required format. ("tgrep -wn '/.*/'" printsparse trees, "tgrep_process.prl" just skips blank lines).

For example, to produce a goldfile for section 23 of the treebank:

tgrep -wn '/.*/' | tail +90895 | tgrep_process.prl | sed 2416q > sec23.gold

You don’t have the ancient program tgrep…

EVALB

• However you can use tsurgeon from the Stanford tregex you downloaded to accomplish the same thing

• Example:– file: wsj_0927.mrg

EVALB./tsurgeon.sh -treeFile wsj_0927.mrg -s( (S (NP-SBJ-1 (NNP H.) (NNP Marshall) (NNP Schwarz)) (VP (VBD was) (VP (VBN named) (S (NP-SBJ (-NONE- *-1)) (NP-PRD (NP (NP (NN chairman)) (CC and) (NP (NN chief) (JJ executive) (NN officer))) (PP (IN of) (NP (NP (NNP U.S.) (NNP Trust) (NNP Corp.)) (, ,) (NP (NP (DT a) (JJ private-banking) (NN firm)) (PP (IN with) (NP (NP (NNS assets)) (PP (IN under) (NP (NN management))) (PP (IN of) (NP (QP (IN about) ($ $) (CD 17) (CD billion)) (-NONE- *U*)))))))))))) (. .)))( (S (NP-SBJ (NP (NNP Mr.) (NNP Schwarz)) (, ,) (ADJP (NP (CD 52) (NNS years)) (JJ old)) (, ,)) (VP (MD will) (VP (VB succeed) (NP (NNP Daniel) (NNP P.) (NNP Davison)) (NP-TMP (NNP Feb.) (CD 1)) (, ,) (SBAR-TMP (RB soon) (IN after) (S (NP-SBJ (NNP Mr.) (NNP Davison)) (VP (VBZ reaches) (NP (NP (NP (DT the) (NN company) (POS 's)) (JJ mandatory) (NN retirement) (NN age)) (PP (IN of) (NP (CD 65))))))))) (. .)))( (S (NP-SBJ-1 (NP (NNP Mr.) (NNP Schwarz)) (, ,) (SBAR (WHNP-2 (WP who)) (S (NP-SBJ (-NONE- *T*-2)) (VP (VBZ is) (NP-PRD (NP (NN president)) (PP (IN of) (NP (NNP U.S.) (NNP Trust))))))) (, ,)) (VP (MD will) (VP (VB be) (VP (VBN succeeded) (NP (-NONE- *-1)) (PP-LOC (IN in) (NP (DT that) (NN post))) (PP (IN by) (NP-LGS (NP (NNP Jeffrey) (NNP S.) (NNP Maurer)) (, ,) (NP (CD 42)) (, ,) (SBAR (WHNP-3 (WP who)) (S (NP-SBJ (-NONE- *T*-3)) (VP (VBZ is) (NP-PRD (NP (JJ executive) (NN vice) (NN president)) (PP (IN in) (NP (NP (NN charge)) (PP (IN of) (NP (NP (DT the) (NN company) (POS 's)) (NN asset-management) (NN group)))))))))))))) (. .)))( (S (NP-SBJ (NP (NNP U.S.) (NNP Trust)) (, ,) (NP (NP (DT a) (JJ 136-year-old) (NN institution)) (SBAR (WHNP-2 (WDT that)) (S (NP-SBJ (-NONE- *T*-2)) (VP (VBZ is) (NP-PRD (NP (CD one)) (PP (IN of) (NP (NP (DT the) (JJS earliest) (NN high-net) (JJ worth) (NNS banks)) (PP-LOC (IN in) (NP (DT the) (NNP U.S.)))))))))) (, ,)) (VP (VBZ has) (VP (VBN faced) (NP (NP (VBG intensifying) (NN competition)) (PP (IN from) (NP (NP (JJ other) (NNS firms)) (SBAR (WHNP-3 (WDT that)) (S (NP-SBJ (-NONE- *T*-3)) (VP (VBP have) (VP (VP (VBN established) (NP (-NONE- *RNR*-1))) (, ,) (CC and) (VP (ADVP-MNR (RB heavily)) (VBN promoted) (NP (-NONE- *RNR*-1))) (, ,) (NP-1 (NP (JJ private-banking) (NNS businesses)) (PP (IN of) (NP (PRP$ their) (JJ own))))))))))))) (. .)))

• You can then redirect standard output to a file …

EVALB

Example• Put chapter 23 in one file

(but not one tree per line)cat ~/research/TREEBANK_3/parsed/mrg/wsj/23/*.mrg > wsj_23.mrg

• Run tsurgeon./tsurgeon.sh -treeFile wsj_23.mrg -s > wsj_23.gold

• File wsj_23.gold contains one tree per line

Homework

– WSJ corpus: sections 00 through 24– Evaluation: on section 23– Training: normally 02-21 (20 sections)– How does the Bikel Collins vary in precision and recall?– if you randomly pick 1, 2, 3 up to 20 sections to do the

training with… – plot graph with evalb…

– Present your results next time

Bikel Collins Training

• Relevant WSJ PTB files

Bikel Collins Parsing• Parsing

– Command

– Input file format (sentences)

wsj_XX.obj.gz file

Bikel Collins Parsing

• You can extract the sentences in section 23 for parsing yourself or you can download– wsj-23.txt

• from the course webpage