INAOE at QAST 2009:Evaluating the usefulness of a

phonetic codification of transcriptions

Alejandro Reyes-BarragánLuis Villaseñor-Pineda

Manuel Montes-y-Gómez

Laboratory of Language TechnologiesNational Institute of Astrophysics, Optics and Electronics

Tonantzintla, Mexico

mmontesg@inaoep.mxhttp://ccc.inaoep.mx/~mmontesg

Our previous work

• We worked in the task of spoken document retrieval.

– Search relevant information for general user queries from a

collection of automatic transcriptions of speech.

• Main challenge is to reduce the impact of transcription

errors in the retrieval accuracy.

– Current Automatic Speech Recognition (ASR) systems have

error rates that vary from 20% to 40%.

Our idea for SDR

• A new document representation based on a phonetic

codification of automatic transcriptions.

– Characterize words with similar pronunciations through the

same phonetic code

– Use Soundex codes to enrich the representation of

transcriptions.

• An example:

– Unix Sun Workstation → (U52000 S30000 W62300)

– Unique some workstation → (U52000 S30000 W62300)

The enriched representation

Automatic transcription

…just your early discussions was roll wallenberg uh any recollection of of uh where he came from…

Phonetic codification

... J23000 Y60000 E64000 D22520 W20000 R40000 W45162 U00000 A50000 R24235 O10000 O10000 U00000 W60000 H00000 C50000 F65000 ...

Enriched representation

{just, early, discussions, roll, wallenberg, recollection, came, E64000, D22520, R40000, W45162, R24235}

• We eliminated stop words and high frequency phonetic codes from

the enriched representation.

• Queries were represented in the same way

– Actions of Raoul Wallenberg

– {actions, raoul, wallenberg, A23520, R40000, W45162}

•

Results in SDR

• Comparing our results against those from the English monolingual task of CL-SR 2007

University Used Information MAP

Ottawa AK1,AK2,ASR04 .0855

Our Method AK1,AK2,ASR06 .0795

Dublín AK1,AK2,ASR06 .0787

Brown AK1,AK2,ASR06 .0785

Chicago AK1,AK2,ASR06 .0571

Amsterdam AK2,ASR06 .0444

Our participation at QAST 2009

• An extension of our previous work

• Aimed to evaluate the usefulness of employing a

phonetic codification in the task of QA in speech

transcriptions.

• Our goal was to improve the retrieval of relevant

passages for each question, and, therefore, the

final answer accuracy– We applied very simple techniques for question classification

and answer extraction.

Architecture of our system

Indexing

Retrieval

NE recognition

Passage segmantation

Phonetic codification

Document processing

Passage representation

Question classification

Phonetic codification

Question processing

Query representation

Information retrieval

Oral DocumentTranscriptions

QuestionTranscription

Answer extraction

Answer selection

CandidateAnswers

Enrichedrepresentationof passages

Enrichedrepresentationof the question

Indexing

Retrieval

NE recognition

Passage segmantation

Phonetic codification

Document processing

Passage representation

NE recognition

Passage segmantation

Phonetic codification

Document processing

Passage representation

Question classification

Phonetic codification

Question processing

Query representation

Question classification

Phonetic codification

Question processing

Query representation

Information retrieval

Oral DocumentTranscriptions

QuestionTranscription

Answer extraction

Answer selection

CandidateAnswers

Enrichedrepresentationof passages

Enrichedrepresentationof the question

Evaluation of passage retrieval(Questions having the answer in the first five passages)

Document Transcription Without phonetic codes Using phonetic codes

Manual 21 24

A 18 24

B 17 22

C 14 17

Document Transcription Without phonetic codes Using phonetic codes

Manual 21 24

A 18 24

B 17 22

C 14 17

Task T1a – Written questions

Task T1a – Spontaneous Oral questions

Results for manual transcriptions

• In TASK T1a, the inclusion of phonetic information was not really

advantageous, it only produced a slightly improvement.

• In TASK T1b, where questions were transcriptions, it was possible

to observe an improvement by using the phonetic codification.

Without phonetic transcription With phonetic transcription

Task Run id#Questions with

at least 1 correct answer

MRR ACC Run id#Questions

with at least 1 correct answer

MRR ACC

T1a (written questions)

INAOE1 54 0.36 27 % INAOE2 51 0.36 28 %

T1b (spontaneous

oral questions)INAOE1 35 0.27 22 % INAOE2 47 0.34 26 %

Results for automatic transcriptions

Task T1a Without phonetic transcription With phonetic transcription

Transcription Run id#Questions with

at least 1 correct answer

MRR ACC Run id#Questions with

at least 1 correct answer

MRR ACC

A INAOE1 41 0.3 23 % INAOE2 42 0.29 22 %

B INAOE1 29 0.22 17 % INAOE2 30 0.22 17 %

C INAOE1 34 0.28 25 % INAOE2 35 0.28 24 %

Task T1b Without phonetic transcription With phonetic transcription

Transcription Run id#Questions with at least 1 correct

answerMRR ACC Run id

#Questions with at least 1 correct

answerMRR ACC

A INAOE1 40 0.3 24 % INAOE2 41 0.29 23 %

B INAOE1 30 0.22 16 % INAOE2 31 0.22 16 %

C INAOE1 33 0.28 25 % INAOE2 34 0.27 23 %

Preliminary Conclusions

• Results indicate that phonetic codes had, in general,

no impact on the answer accuracy.

• Given that phonetic codes improved the passage

retrieval, we may conclude that our answer extraction

method is inadequate.

• We obtained better results using manual transcriptions

because the NER was accurate.

Thank you!Manuel Montes y GómezLanguage Technologies Laboratory

National Institute of Astrophysics, Optics and ElectronicsTonantzintla, México

mmontesg@inaoep.mxhttp://ccc.inaoep.mx/~mmontesg

Soundex codification

• Capitalize all letters in the word and drop all punctuation marks.• Retain the first letter of the word.• Change all occurrence of the following letters to '0' (zero):

  'A', E', 'I', 'O', 'U', 'H', 'W', 'Y'.• Change letters from the following sets into the given digit:

– 1 = 'B', 'F', 'P', 'V'– 2 = 'C', 'G', 'J', 'K', 'Q', 'S', 'X', 'Z' – 3 = 'D','T' – 4 = 'L' – 5 = 'M','N' – 6 = 'R'

• Remove all pairs of equal digits occurring beside each other from the string resulted after step (4).

• Remove all zeros from the string that results from step (5)• Pad the string resulted from step (6) with trailing zeros and return only the

first six positions. The output code will be of the form <uppercase letter> <digit> <digit> <digit> <digit> <digit>.