Download - INAOE at QAST 2009: Evaluating the usefulness of a phonetic codification of transcriptions
INAOE at QAST 2009:Evaluating the usefulness of a
phonetic codification of transcriptions
Alejandro Reyes-BarragánLuis Villaseñor-Pineda
Manuel Montes-y-Gómez
Laboratory of Language TechnologiesNational Institute of Astrophysics, Optics and Electronics
Tonantzintla, Mexico
[email protected]://ccc.inaoep.mx/~mmontesg
Our previous work
• We worked in the task of spoken document retrieval.
– Search relevant information for general user queries from a
collection of automatic transcriptions of speech.
• Main challenge is to reduce the impact of transcription
errors in the retrieval accuracy.
– Current Automatic Speech Recognition (ASR) systems have
error rates that vary from 20% to 40%.
Our idea for SDR
• A new document representation based on a phonetic
codification of automatic transcriptions.
– Characterize words with similar pronunciations through the
same phonetic code
– Use Soundex codes to enrich the representation of
transcriptions.
• An example:
– Unix Sun Workstation → (U52000 S30000 W62300)
– Unique some workstation → (U52000 S30000 W62300)
The enriched representation
Automatic transcription
…just your early discussions was roll wallenberg uh any recollection of of uh where he came from…
Phonetic codification
... J23000 Y60000 E64000 D22520 W20000 R40000 W45162 U00000 A50000 R24235 O10000 O10000 U00000 W60000 H00000 C50000 F65000 ...
Enriched representation
{just, early, discussions, roll, wallenberg, recollection, came, E64000, D22520, R40000, W45162, R24235}
• We eliminated stop words and high frequency phonetic codes from
the enriched representation.
• Queries were represented in the same way
– Actions of Raoul Wallenberg
– {actions, raoul, wallenberg, A23520, R40000, W45162}
•
Results in SDR
• Comparing our results against those from the English monolingual task of CL-SR 2007
University Used Information MAP
Ottawa AK1,AK2,ASR04 .0855
Our Method AK1,AK2,ASR06 .0795
Dublín AK1,AK2,ASR06 .0787
Brown AK1,AK2,ASR06 .0785
Chicago AK1,AK2,ASR06 .0571
Amsterdam AK2,ASR06 .0444
Our participation at QAST 2009
• An extension of our previous work
• Aimed to evaluate the usefulness of employing a
phonetic codification in the task of QA in speech
transcriptions.
• Our goal was to improve the retrieval of relevant
passages for each question, and, therefore, the
final answer accuracy– We applied very simple techniques for question classification
and answer extraction.
Architecture of our system
Indexing
Retrieval
NE recognition
Passage segmantation
Phonetic codification
Document processing
Passage representation
Question classification
Phonetic codification
Question processing
Query representation
Information retrieval
Oral DocumentTranscriptions
QuestionTranscription
Answer extraction
Answer selection
CandidateAnswers
Enrichedrepresentationof passages
Enrichedrepresentationof the question
Indexing
Retrieval
NE recognition
Passage segmantation
Phonetic codification
Document processing
Passage representation
NE recognition
Passage segmantation
Phonetic codification
Document processing
Passage representation
Question classification
Phonetic codification
Question processing
Query representation
Question classification
Phonetic codification
Question processing
Query representation
Information retrieval
Oral DocumentTranscriptions
QuestionTranscription
Answer extraction
Answer selection
CandidateAnswers
Enrichedrepresentationof passages
Enrichedrepresentationof the question
Evaluation of passage retrieval(Questions having the answer in the first five passages)
Document Transcription Without phonetic codes Using phonetic codes
Manual 21 24
A 18 24
B 17 22
C 14 17
Document Transcription Without phonetic codes Using phonetic codes
Manual 21 24
A 18 24
B 17 22
C 14 17
Task T1a – Written questions
Task T1a – Spontaneous Oral questions
Results for manual transcriptions
• In TASK T1a, the inclusion of phonetic information was not really
advantageous, it only produced a slightly improvement.
• In TASK T1b, where questions were transcriptions, it was possible
to observe an improvement by using the phonetic codification.
Without phonetic transcription With phonetic transcription
Task Run id#Questions with
at least 1 correct answer
MRR ACC Run id#Questions
with at least 1 correct answer
MRR ACC
T1a (written questions)
INAOE1 54 0.36 27 % INAOE2 51 0.36 28 %
T1b (spontaneous
oral questions)INAOE1 35 0.27 22 % INAOE2 47 0.34 26 %
Results for automatic transcriptions
Task T1a Without phonetic transcription With phonetic transcription
Transcription Run id#Questions with
at least 1 correct answer
MRR ACC Run id#Questions with
at least 1 correct answer
MRR ACC
A INAOE1 41 0.3 23 % INAOE2 42 0.29 22 %
B INAOE1 29 0.22 17 % INAOE2 30 0.22 17 %
C INAOE1 34 0.28 25 % INAOE2 35 0.28 24 %
Task T1b Without phonetic transcription With phonetic transcription
Transcription Run id#Questions with at least 1 correct
answerMRR ACC Run id
#Questions with at least 1 correct
answerMRR ACC
A INAOE1 40 0.3 24 % INAOE2 41 0.29 23 %
B INAOE1 30 0.22 16 % INAOE2 31 0.22 16 %
C INAOE1 33 0.28 25 % INAOE2 34 0.27 23 %
Preliminary Conclusions
• Results indicate that phonetic codes had, in general,
no impact on the answer accuracy.
• Given that phonetic codes improved the passage
retrieval, we may conclude that our answer extraction
method is inadequate.
• We obtained better results using manual transcriptions
because the NER was accurate.
Thank you!Manuel Montes y GómezLanguage Technologies Laboratory
National Institute of Astrophysics, Optics and ElectronicsTonantzintla, México
[email protected]://ccc.inaoep.mx/~mmontesg
Soundex codification
• Capitalize all letters in the word and drop all punctuation marks.• Retain the first letter of the word.• Change all occurrence of the following letters to '0' (zero):
'A', E', 'I', 'O', 'U', 'H', 'W', 'Y'.• Change letters from the following sets into the given digit:
– 1 = 'B', 'F', 'P', 'V'– 2 = 'C', 'G', 'J', 'K', 'Q', 'S', 'X', 'Z' – 3 = 'D','T' – 4 = 'L' – 5 = 'M','N' – 6 = 'R'
• Remove all pairs of equal digits occurring beside each other from the string resulted after step (4).
• Remove all zeros from the string that results from step (5)• Pad the string resulted from step (6) with trailing zeros and return only the
first six positions. The output code will be of the form <uppercase letter> <digit> <digit> <digit> <digit> <digit>.