“eve eat dust mop”mernst/advice/poster...“eve eat dust mop” measuring syntac6c development...

“Eve Eat Dust Mop” Measuring Syntac6c Development in Child Language with Natural Language Processing and Machine Learning Shannon N. Lube-ch (Pomona College ’15) Ins-tute for Crea-ve Technologies, University of Southern California Automa6c IPSyn Goal: Create a program that, given a transcript, automa-cally calculates IPSyn score. Results: Automa6c IPSyn Acknowledgements I would like to thank Professor Kenji Sagae for introducing me to this project, and for providing his guidance, advice, support, and shared knowledge along the way. I thank Evan Suma for organizing my REU, and NSF for generously providing the funding for it. I would also thank to thank everyone at ICT for trea-ng me as part of the family. Fully Data-Driven Measurement Goal: Eliminate explicit IPSyn Items by training a machine on expert level annota-ons and scores so that it can produce an IPSyn score based solely on features of a transcript. SUBJ n:prop n:prop v n:prop SUBJ v Eve eat dust mop. n:prop v n n SUBJ OBJ MOD Data-Driven Approach to Age Predic6on Background: Index of Produc6ve Syntax (IPSyn) The higher the score, the more gramma6cally complex the language Are we going to the party? N1: Noun N4: 2- word Noun Phrase N5: Article before a noun N6: 2- word Noun Phrase after prep N2: Pronoun V1: Verb V2: Preposition V3: Prepositional phrase V6: Auxiliary V7: Progressive suffix (-ing) Q8: Y/N question with inverted aux S1: 2-word combination IPSyn measures gramma-cal complexity of language by evalua-ng 100 successive uWerances and awarding points based on defined “IPSyn Items.” There are 60 items that can earn up to 2 points each, resul-ng in a total possible score of 120 points. This score can be broken down into subsec-ons of Noun Phrases, Verb Phrases, Ques-ons and Nega-ons, and Sentence Structures. Pattern: N4 Description: 2-word NP GR: DET MOD QUANT Parent requirements: Index: forward-1 POS: n this unicorn fluffy unicorn one unicorn DET MOD QUANT Defining IPSyn Items Preprocessing Scarborough 1990: 75 transcripts Hours of manual scoring Automa-c IPSyn (this work): 593 transcripts Minutes to compute When measuring child language development, researchers oeen face choices between easily computable metrics focused on superficial aspects of language and more expressive metrics that rely on gramma-cal structure but require substan-al labor. To advance research in child language development, we present an automa-c scoring system facilita-ng easy analysis of large numbers of transcripts. Addi-onally, we explore a machine learning approach to produce scores of gramma-cal complexity based on extrac-ng morphological and syntac-c features of child uWerances. Both techniques achieve accuracy similar to that of language researchers and reveal trends in syntac-c development, offering promising results for future research and applica-on. We can further apply our data-driven approach to predic-ng age, which does not require a previously-defined, language-specific inventory of gramma-cal structures. Average absolute difference Manual: ~5 points Automa-c: 3.6 points Child (corpus) Mean Abs Err Pearson (r) Adam (Brown) 2.5 0.93 Ross (MacWhinney) 3.7 0.84 Naomi (Sachs) 3.1 0.91 Child (corpus) MLU r IPSyn r Regression r Adam (Brown) 0.37 † 0.53 † 0.85 † Ross (MacWhinney) 0.19 0.34* 0.79 † Naomi (Sachs) 0.27 0.52 0.82 † † p < 0.0001 ∗ p < 0.05 For children at least 3 yrs 4 months old Future Work ¡ Expand to different languages (Spanish, Japanese, Hebrew) ¡ Improve patterns ¡ Train classifier on more data MOR (MacWhinney 2000) Morphological analyzer Part of speech POST (Parisse & Le Normand 2000) Part of speech tagger Disambiguate MEGRASP (Sagae et al 2005/2007) Dependency parser Gramma-cal rela-ons Eve eat dust mop. n:prop v n n SUBJ OBJ MOD Feature Extrac-on

Upload: others

Post on 10-Oct-2020

1 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

“EveEatDustMop”MeasuringSyntac6cDevelopmentinChildLanguage

withNaturalLanguageProcessingandMachineLearningShannonN.Lube-ch(PomonaCollege’15)

Ins-tuteforCrea-veTechnologies,UniversityofSouthernCalifornia

Automa6cIPSynGoal:Createaprogramthat,givenatranscript,automa-callycalculatesIPSynscore.

Results:Automa6cIPSyn

AcknowledgementsIwouldliketothankProfessorKenjiSagaeforintroducingmetothisproject,andforprovidinghisguidance,advice,support,andsharedknowledgealongtheway.IthankEvanSumafororganizingmyREU,andNSFforgenerouslyprovidingthefundingforit.IwouldalsothanktothankeveryoneatICTfortrea-ngmeaspartofthefamily.

FullyData-DrivenMeasurementGoal:EliminateexplicitIPSynItemsbytrainingamachineonexpertlevelannota-onsandscoressothatitcanproduceanIPSynscorebasedsolelyonfeaturesofatranscript.

SUBJ

n:prop

n:propv

n:propSUBJv

Eveeatdustmop.n:propvnn

SUBJ OBJ

MOD

Data-DrivenApproachtoAgePredic6on

Background:IndexofProduc6veSyntax(IPSyn)Thehigherthescore,themoregramma6callycomplexthelanguage

Arewegoingtotheparty?

N1:Noun

N4:2-wordNounPhrase

N5:Articlebeforeanoun

N6:2-wordNounPhraseafterprep

N2:Pronoun

V1:Verb

V2:Preposition V3:

Prepositionalphrase

V6:AuxiliaryV7:Progressivesuffix(-ing)

Q8:Y/Nquestionwithinvertedaux

S1:2-wordcombination

IPSynmeasuresgramma-calcomplexityoflanguagebyevalua-ng100successiveuWerancesandawardingpointsbasedondefined“IPSynItems.”Thereare60itemsthatcanearnupto2pointseach,resul-nginatotalpossiblescoreof120points.Thisscorecanbebrokendownintosubsec-onsofNounPhrases,VerbPhrases,Ques-onsandNega-ons,andSentenceStructures.

Pattern:N4Description:2-wordNPGR:DETMODQUANTParentrequirements:Index:forward-1POS:n

thisunicorn

fluffyunicorn

oneunicorn

DET

MOD

QUANT

DefiningIPSynItems Preprocessing

Scarborough1990:75transcriptsHoursofmanualscoring

Automa-cIPSyn(thiswork):593transcriptsMinutestocompute

Whenmeasuringchildlanguagedevelopment,researchersoeenfacechoicesbetweeneasilycomputablemetricsfocusedonsuperficialaspectsoflanguageandmoreexpressivemetricsthatrelyongramma-calstructurebutrequiresubstan-allabor.Toadvanceresearchinchildlanguagedevelopment,wepresentanautoma-cscoringsystemfacilita-ngeasyanalysisoflargenumbersoftranscripts.Addi-onally,weexploreamachinelearningapproachtoproducescoresofgramma-calcomplexitybasedonextrac-ngmorphologicalandsyntac-cfeaturesofchilduWerances.Bothtechniquesachieveaccuracysimilartothatoflanguageresearchersandrevealtrendsinsyntac-cdevelopment,offeringpromisingresultsforfutureresearchandapplica-on.Wecanfurtherapplyourdata-drivenapproachtopredic-ngage,whichdoesnotrequireapreviously-defined,language-specificinventoryofgramma-calstructures.

AverageabsolutedifferenceManual:~5points Automa-c:3.6points

Child(corpus) MeanAbsErr Pearson(r)

Adam(Brown) 2.5 0.93

Ross(MacWhinney) 3.7 0.84

Naomi(Sachs) 3.1 0.91

Child(corpus) MLUr IPSynr Regressionr

Adam(Brown) 0.37† 0.53† 0.85†

Ross(MacWhinney) 0.19 0.34* 0.79†

Naomi(Sachs) 0.27 0.52 0.82††p<0.0001

∗p<0.05

Forchildrenatleast3yrs4monthsold

FutureWork¡  Expandtodifferentlanguages(Spanish,Japanese,Hebrew)