Slide 1

SPOKEN LANGUAGE CORPUS PROJECT SPOKEN CORPORA FOR THE 9 OFFICIAL SOUTH AFRICAN AFRICAN LANGUAGES

Slide 2

Workshop Overview The Asmara Declaration Rusandre Whats the point of spoken language corpora? Jens Overview of the project and its phases Rusandre The recording phase Jens/Mmem The transcription phase Jens The checking phase Jens The tagging phase Leif/Rusandre Research output - Jens

Slide 3

THE ASMARA DECLARATION - 2000 Dialogue among African languages is essential: African languages must use the instrument of translation to advance communication among all people, including the disabled. All African children have the inalienable right to attend school and learn in their mother tongues. All effort should be made to develop African languages at all levels of education.

Slide 4

ASMARA DECLARATION - CNTD Promoting research on African languages is vital for their development, while the advancement of African research and documentation will be best served by the use of African languages. The effective and rapid development of science and technology in Africa depends on the use of African languages and modern technology must be used for the development of African languages.

Slide 5

Whats the point of spoken language corpora? Jens Allwood Corpus linguistics / Armchair linguistics

Slide 6

PROJECT MANAGEMENT Goteborg/Unisa Nguni Rhodes Fort Hare UPE/Vista Natal Unizul Sotho N-SothoTswana Univ of NorthNorthwest Univ Venda Univ.Univ. Botswana Venda Venda Univ Tsonga

Slide 7

OBJECTIVES To develop a platform of computer supported basic linguistic resources for the previously disadvantaged languages of SA The resources will be in the form of archived audio-visual recordings of activity-based natural language use; machine-readable transcriptions of recordings for corpus-driven searches; morphologically tagged corpora for corpus-based searches.

Slide 8

PROJECT PHASES 2002 - 2004 1.Ongoing Audio-video recordings of activity- based spoken language use (min. 200hrs p/l). 2.Transcriptions (enriched with comment lines) of recordings in machine-readable text format. 3.Checking and editing of transcriptions. 4.Manual morphological tagging of corpora. 5.Automated tagging of corpora. 6.Research outputs.

Slide 9

The recording phase What to record Activity types What to think about when recording natural language dialogues Keep it natural The video camera, microphone, etc Keep the camera fixed!

Slide 10

Recording and transcription Practical exercise! 1.A short recording 2.Transcribe together

Slide 11

Transcription Structure Header (background information about transcription and recorded activity) Body (the actual transcription consisting of two kinds of elements) Contributions (transcribed utterances of participants in the recorded activity) Information lines - marks various peculiar aspects in the contributions and recorded activity

Slide 12

Example of a header @ Recorded activity ID: V010501 @ Activity type: Informal conversation @ Recorded activity title: Getting to know each other @ Recorded activity date: 20020725 @ Recorder: Britta Zawada @ Participant: A = F2 (Lunga) @ Participant: B = F1 (Bukiwe) @ Transcriber: Mvuyisi Siwisa @ Transcription date: 20020805 @ Checker: Rusandre Hendrikse @ Checking date: 20020912 @ Anonymised: No @ Activity Medium: face-to-face @ Activity duration: 00:44:30 @ Other time coding: Each section @ Tape: V0105 @ Section: Family affairs @ Section: Crime @ Section: Unemployment @ Section: Closing @ Comment: Medunsa open ended conversation between two adult speech therapy students Bukiwe and Lunga

Slide 13

Transcription header @ Recorded activity ID: V010501 V = Video, 01 = project number 05 = Tape number within this project 01 = Recording number @ Activity type: Informal conversation @ Recorded activity title: Getting to know each other @ Recorded activity date: 20020725 @ Recorder: Britta Zawada

Slide 14

Transcription header, cont @ Participant: A = F2 (Lunga) @ Participant: B = F1 (Bukiwe) F stands for female F1 is unique for Bukiwe in the entire corpus A and B are ID:s for the participants

Slide 15

Transcription header, cont @ Transcriber: Mvuyisi Siwisa @ Transcription date: 20020805 @ Checker: Rusandre Hendrikse @ Checking date: 20020912

Slide 16

Transcription header, cont @ Anonymised: No Indicates whether personal names, etc have been changed to pseudonyms (Yes) or not (No) both in the header and in the conversation @ Activity Medium: face-to-face Normally spoken, face to face, but could also have other values, like telephone conversations.

Slide 17

Transcription header, cont @ Activity duration: 00:44:30 Duration in hours, minutes and seconds @ Other time coding: Each section There is a time line for each section @ Tape: V0105 This is a part of the recorded activity ID

Slide 18

Transcription header, cont @ Section: Family affairs @ Section: Crime @ Section: Unemployment @ Section: Closing @ Comment: Medunsa open ended conversation between two adult speech therapy students Bukiwe and Lunga Any relevant information that is not covered by any of the required headings

Slide 19

The body This is the actual transcription - the background information is in the header Four kinds of lines: $A: uyakhonza kaneneContribution @ Information line At officeSection line # 00:10:00Time line

Slide 20

Sections Family affairs $B: sibabini kuphela esibabalwe sada safunda ke noko sakwazi ukuphangela sikwazi ke noko kuba ndinobhuti wam osebenzayo... Religion $B: uyakhonza kanene $A: ndiyakhonza owu ndiyamthand{a} [4 uthixo ndiyamthanda andisoze ndimlahle undibonisile ukuba mkhulu nantso ke into efunekayo qha ]4 kuphela $B: [4 nantso ke sisi e: e: ]4 $B: nantso ke into efunekayo uthixo ulithemba lethu [5 uthixo ulithemba lethu ulixhadi lethu ]5 uligwiba $A: [5 ulixhadi lethu ulixhadi lethu]5 $B: [6 uligwiba andazi ukuba ndingangendithini ngendiphi na xa uthixo heyi ]6 Situation on their arrival at Medunsa $A: [6 ucinga ukuba ngesiphi na ngesisemedunsa ]6 $B: uye wasithatha khona waza kusibeka kule ndawo...

Slide 21

Contributions Religion $B: uyakhonza kanene $A: ndiyakhonza owu ndiyamthand{a} [4 ndiyamthanda andisoze ndimlahle undibonisile ukuba mkhulu nantso ke into efunekayo qha ]4 kuphela @

Slide 22

Overlaps Religion $B: uyakhonza kanene $A: ndiyakhonza owu ndiyamthand{a} [4 ndiyamthanda andisoze ndimlahle undibonisile ukuba mkhulu nantso ke into efunekayo qha ]4 kuphela $B: [4 nantso ke sisi // e: e: ]4 @

Slide 23

Contrastive stress, pauses and lengthening $B: abanye ke bazihlalele nje: / abanye ABAZANGE bafune sikolo // uyayiqonda ke la meko yokungabikho mzali uqhubayo / uthi aba baza emva kwam bobabini ABAZANGE bafunde kuyaphi // kodwa ke // andigxeki nto kuba ke / ndibakhona ngethuba le ngxaki nobhuti ke [2 abeyinkxaso kakhulu ]2 $A: [2 ya / m: ewe ]2 hayi izinto zikuthixo azikho kuthi nam obu bushuman bam ndiseza kutshata ndiseza kutshata

Slide 24

Unclear speech and glottal stop $M: loo nto ke njengo{ku}ba sekunyanzeleke ukuba ndiye phaya nje (...) ndikwazi ukuncedisa phaya ndiyiphushile ukwenzela ukuba ndibe neclaim endizakuba nayo that is why ndithole because ndiyaclaimer so that at least uba ndiclayimile ndikwazi ukuhamba $T: ke ngoku ke yenye yezinto endifuna ukuyoyenza $M: ngolwesithathu (what she said to me ngoku bendiphaya ngecawe) besingcwaba umfazi kasicaka jama $T: ee andekufuni ukutya

Slide 25

Comment Lines $A: kunetha imvula sinemithwalo engaka nako sisa @ $B: esingazi lo mntwana ngoba kaloku siza apha asazi mntu ukuba wayengekho ngesasitheni na asazi mntu @

Slide 26

Research output Jens Allwood A distributed database (corpus) Networks (homepages) Spoken language corpus activities (seminars, workshops)

Slide 27

TAGGING SPOKEN LANGUAGE SAMPLES PROBLEMATIC ISSUES CONVENTIONS & STANDARDS A P Hendrikse 16/03/04

Slide 28

PROBLEMATIC ISSUES Loans and codeswitching Fixed expressions Spoken language reductions Morphophonological issues Designing a tag set Manual tagging A drag-and-drop tagger Automated tagging

Slide 29

Loans and Codeswitching Non-indigenised codeswitching ndifuna Indigenised but non-standardised codeswitching loans >ndiyakleyimisha? ndiyaklayimisha? ndiyafonisha? ndiyafowunisha?

Slide 30

Fixed Expressions A continuum: Idioms/proverbs prefabricated expressions collocations How fixed is fixed? Into yokuba (*izinto zokuba) Nantso ke (*nantsi ke?) (Ke) kaloku (ke) Bafondini/mfondini Undincedile Ungadinwa nangomso

Slide 31

Fixed Expressions cntd Flagging fixed phrases Into_yokuba Ke_kaloku_ke Morphosyntactic tagging or not? Ke >_kaloku >_ke > > Or Ke_kaloku_ke >

Slide 32

Spoken language reductions Standardised reductions Ngokuba > ngoba Written standard reduction: reconstruction convention {} not used, i.e. *ngo{ku}ba Non-standardised reductions Musa ukuhamba > sukuhamba (wsr) > Suhamba (non-standardised)

Slide 33

Spoken Reductions cntd Reconstruction convention S{uku}hamba Tagged S >{uku >}hamb >a > >

Slide 34

Morphophonological Issues Coalescence Nenkomo > ne >n >komo > Neenkomo > ne >en >komo > Syllabification Ngasendl{w}ini > nga >se >n >dl{w} >ini > Ayikafiki > ayi >ka >fik >i >

Slide 35

Morphophonological cntd Elision Andinamoto > andi >na > >m oto > > Stem modifications Emlanjeni > e >m >lanj >en i > >

Slide 36

Designing a tag set Granularity Lexical categories N, V (Tagging lexical categories is problematic in an agglutinating language) Syntagmatic morphological slots amadodana > a >ma >dod >ana >

Slide 37

Designing cntd Paradigmatic instantiations within a syntagmatic slot gnp = >--- > Word categories nje (wenjenje) > nje >; njalo >; njeya > ke > ke > kaloku > ke > ke_kaloku_ke > e >m >lanj >eni >??

Slide 38

Designing cntd Spoken language expressions Non-word like expressions 2 problems 1.Standardising orthographic representation 2.Tags e: > mh: > uh_uh_uh >

Slide 39

Designing cntd Word-like expressions >thixo > Thixo > Heyi_wethu Nantso_ke Suka_(wena)

Slide 40

Manual tagging Manual tagging necessary for 3 reasons Identifying tagging problems and problematic phenomena and revising the tag set Developing a training corpus Correcting automated tagging errors Manual (typing) tagging not ideal Tedious Error-prone Solution: Drag-and-drop tagger

Slide 41

Drag-and-drop tagger Demonstration of drag-and-drop tagger