performance analysis of bangla speech recognizer model using hmm
TRANSCRIPT
Performance analysis of Bangla Speech Recognizer
model using Hidden Markov Model (HMM)
Submitted by: Md. Abdullah-al-MAMUN
1
OUTLINEOUTLINE What is speech recognition ?What is speech recognition ? The Structure of ASR The Structure of ASR Speech DatabaseSpeech Database Feature ExtractionFeature Extraction Hidden Markov ModelHidden Markov Model
Forward algorithmForward algorithm Backward algorithmBackward algorithm Viterbi algorithmViterbi algorithm
Training & RecogntionTraining & Recogntion ResultResult ConclusionsConclusions ReferencesReferences
2
What is What is SSpeech peech RRecognitionecognition??
In Computer Science, In Computer Science, Speech Speech recognitionrecognition is the translation of is the translation of spoken words into text .spoken words into text . Process of converting acoustic Process of converting acoustic signal captured by microphone to a signal captured by microphone to a set of words.set of words. Speech recognition known as Speech recognition known as “Automatic Speech Recognition (ASR) “Automatic Speech Recognition (ASR) ”, “Speech to Text (STT)".”, “Speech to Text (STT)".
3
Model of Model of BBangla angla SSpeech peech RRecognitionecognition
4
Fig -1 : Simple model of Bangla Speech Recognition
Database Signal Interface
Feature Extraction
Recognition
DatabasesTraining HMM
The Structure of The Structure of ASRASR System:System:
Figure 1 :Functional Scheme of an ASR SystemFigure 1 :Functional Scheme of an ASR System
Speech samples
X Y
S
W*
5
Speech Database:Speech Database:-A speech database is a A speech database is a collection of recorded speech collection of recorded speech accessible on a computer and accessible on a computer and supported with the necessary supported with the necessary transcriptions.transcriptions.-The databases collect the The databases collect the observations required for observations required for parameter estimations.parameter estimations.-In this ASR system, I have used In this ASR system, I have used about 1200 keywords.about 1200 keywords.
6
Classification of Classification of KeywordsKeywords
Bengal Word
Independent
Dependent
Vowel
Consonant
Modifier
Character
Compound
Character
7
DDatabase atabase CCreation reation PProcessrocess
Database
8
Speech Signal AnalysisSpeech Signal Analysis
Feature Extraction for Feature Extraction for ASR:ASR:
- The aim is to extract the voice - The aim is to extract the voice features to distinguish different features to distinguish different phonemes of a language.phonemes of a language.
9
515645465
156156165
156456454
251561565
Feature Extractio
n
MFCCMFCC extractionextraction
Pre-emphasis DFTMel filter
banks Log(||2) IDFT
Speech
signalx(n)
WINDOW
x’(n)
xt (n)
Xt(k)
Yt(m)
MFCCyt(m)(k)
10
MFCC means Mel-frequency cepstral coefficients that representation of the short-term power spectrum of a sound for audio processing.
The MFCCs are the amplitudes of the resulting spectrum.
Speech waveform Speech waveform of a phoneme “\of a phoneme “\
ae”ae”
After pre-emphasis After pre-emphasis and Hamming and Hamming
windowingwindowing
Power spectrumPower spectrum MFCCMFCC
Explanatory ExampleExplanatory Example
11
FFeature eature VVector to ector to P(O|M)P(O|M) via via HMMHMM
12
51564654564
P(O|M)
HMM
For each input word O the HMM generate a corresponding probability P(O|M) that could be computed by the HMM.
HMM ModelHMM Model
13
HMM is specified by a five-tuples λ=( , , , , )S O A B
14
Elements of an HMMElements of an HMM
1) Set of hidden states 1) Set of hidden states S={1.2., … … N}S={1.2., … … N}
2) Set of observation symbols 2) Set of observation symbols O={oO={o11, o, o22, … … o, … … oMM}}
M: the number of observation symbolsM: the number of observation symbols
3) The initial state distribution3) The initial state distribution
4) State transition probability distribution4) State transition probability distribution
5) Observation symbol probability distribution in 5) Observation symbol probability distribution in state j state j
1{ } ( | ), 1 ,ij ij t tA a a P s j s i i j N
{ ( )} ( ) ( | ) 1 ,1j j t k tB b k b k P X o s j j N k M
0{ } ( ) 1i i P s i i N
15
Three Basic Problems in Three Basic Problems in HMMHMM 1.The Evaluation Problem 1.The Evaluation Problem –Given a model –Given a model λλ =(A, B, =(A, B,
π)π) and a sequence of observations O and a sequence of observations O = (o = (o11, o, o22, , oo33,...o,...oMM ) ), what is the probability P(O|, what is the probability P(O|λλ); i.e., the ); i.e., the probability of the model that generates the probability of the model that generates the observations?observations?
2.The Decoding Problem 2.The Decoding Problem – Given a model – Given a model λλ =(A, B, =(A, B, π)π) and a sequence of observation O and a sequence of observation O = (o= (o11, o, o22, , oo33,...o,...oMM ) ), what is the most likely state sequence in , what is the most likely state sequence in the model that produces the observations?the model that produces the observations?
3.The Learning Problem 3.The Learning Problem –Given a model –Given a model λλ =(A, B, π) =(A, B, π) and a set of observations and a set of observations O = (oO = (o11, o, o22, o, o33,...o,...oMM ) ), how , how can we adjust the model parameter can we adjust the model parameter λλ to maximize to maximize the joint probability P(O|the joint probability P(O|λλ)?)?
How to evaluate an HMM?
Forward Algorithm
How to Decode an HMM?
Viterbi Algorithm
How to Train an HMM?
Baum-Welch Algorithm
16
Calculate Calculate PProbability robability ( O| M )( O| M )
Trellis:
0.5
0.3
0.2
P(up)P(down) P(no-change)
0.30.30.4
0.70.10.2
0.10.60.3
0.179
0.036
0.008
0.35
0.02
0.09
0.35*0.2*0.3
0.02*0.5*0.7
0.09*0.4*0.7
0.02*0.2*0.3
0.09*0.5*0.3
0.35*0.6*0.7 0.179*0.6*0.7
0.008*0.5*0.7
0.036*0.4*0.7
0.60.50.4
0.20.30.1
0.2
0.2 transition matrix0.5
0.2230.46add probabilities !
Forward Calculations – Forward Calculations – OverviewOverview
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
0.60.10.3
0.10.10.2
17
Forward Calculations (t=2)Forward Calculations (t=2)
S0
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2
NOTE: that 1 (2)+ 2 (2)is the likelihood of the observation.
1
2
1 1 13 11 2 23 21
2 1 13 12 2 23 22
(1) 1
(1) 0
(2) (1) (1) 0.21
(2) (1) (1) 0.09
b a b a
b a b a
0.60.10.3
0.10.10.2
18
Forward Calculations (t=3)Forward Calculations (t=3)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
1(3)
0.60.10.3
0.10.10.2
19
Forward Calculations (t=4)Forward Calculations (t=4)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
S1
S2
0.60.10.3
0.10.10.2
20
Forward Calculation of Forward Calculation of Likelihood FunctionLikelihood Function
t=1 t=2 t=3 t=4
1(t) 1.0
1 =1
0.21
1(1) a11 b13
+2(1) a21 b23
0.04621(2)a11 b12
+2(2)a21 b12
0.021294
2(t) 0.0
2 =0
0.09 1(1) a12 b13
+2(1) a22 b23
0.0378 0.010206
L(t)
p(K1… Kt)
1.01(1) +2(1)
0.31(2) +2(2)
0.0841(3) +2(3)
0.03151(4) +2(4)
21
Backward Calculations – Backward Calculations – OverviewOverview
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
0.60.10.3
0.10.10.2
22
Backward Calculations (t=3)Backward Calculations (t=3)
S1
S2
TIME 3
0.60.10.3
0.10.10.2
23
Backward Calculations (t=2)Backward Calculations (t=2)
S1
S2
S1
S2
TIME 2 TIME 3 TIME 4
a22=0.5
a11=0.7
a12=0.3a21=0.5
NOTE: that 1 (2)+ 2 (2)is the likelihood the observation/word sequence.
1
2
1
2
1 1 11 12 2 12 12
2 1 21 22 2 22 22
(4) 1
(4) 1
(3) 0.6
(3) 0.1
(2) (3) (3) 0.045
(2) (3) (3) 0.245
a b a b
a b a b
0.60.10.3
0.10.10.2
24
Backward Calculations (t=1)Backward Calculations (t=1)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a21=0.5
TIME 2 TIME 3 TIME 4
0.60.10.3
0.10.10.2
25
Backward Calculation of Backward Calculation of Likelihood FunctionLikelihood Function
t=1 t=2 t=3 t=4
1(t) 0.0315 0.045a11b11 1(1) ++ a12b21 1(1)
0.6b11
1
2(t) 0.029 0.245 a11b11 1(1) +
+ a12b21 1(1)
0.1 b21
1
L(t)p(Kt… KT)
0.03151 1(1) +
2 2(1)
0.2901(2) +2(2)
0.71(3) + 2(3)
1
26
27
Calculate Calculate maxmaxSS Prob. Prob. state sequence state sequence SS
0.35
0.09
0.02
P(up)P(down) P(no-change)
0.30.30.4
0.70.10.2
0.10.60.3
0.147
0.021
0.007
0.35*0.2*0.3
0.02*0.5*0.7
0.09*0.4*0.7
0.02*0.2*0.3
0.09*0.5*0.3
0.35*0.6*0.7 0.147*0.6*0.7
0.007*0.5*0.7
0.021*0.4*0.7
0.5
0.2
0.3
best
Select highest probability !
Viterbi Algorithm – OverviewViterbi Algorithm – Overview
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
0.60.10.3
0.10.10.2
28
Viterbi Algorithm (Forward Calculations Viterbi Algorithm (Forward Calculations t=2)t=2)
S0
S1
S2
S1
S2
1=1
2=0
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2
1 1
2 2
1 1 13 11 2 23 21
2 1 13 12 2 23 22
1
2
(1) 1
(1) 0
(2) max{ (1) , (1) } 0.21
(2) max{ (1) , (1) } 0.09
(2) 1
(2) 1
b a b a
b a b a
0.60.10.3
0.10.10.2
29
Viterbi Algorithm (Backtracking t=2)Viterbi Algorithm (Backtracking t=2)
S0
S1
S2
S1
S2
1=1
2=0
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2
1 1
2 2
1 1 13 11 2 23 21
2 1 13 12 2 23 22
1
2
(1) 1
(1) 0
(2) max{ (1) , (1) } 0.21
(2) max{ (1) , (1) } 0.09
(2) 1
(2) 1
b a b a
b a b a
0.60.10.3
0.10.10.2
30
Viterbi Algorithm (Forward Viterbi Algorithm (Forward Calculations)Calculations)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
0.60.10.3
0.10.10.2
31
Viterbi Algorithm (backtracking)Viterbi Algorithm (backtracking)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
0.60.10.3
0.10.10.2
32
Viterbi Algorithm (Forward Calculations t=4)Viterbi Algorithm (Forward Calculations t=4)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
S1
S2
0.60.10.3
0.10.10.2
33
Viterbi Algorithm (Backtracking to Obtain Viterbi Algorithm (Backtracking to Obtain Labeling)Labeling)
S0
S1
S2
S1
S2
S1
S2
1
2
a12=0.3
a11=0.7
a22=0.5
a21=0.5
TIME 2 TIME 3 TIME 4
S1
S2
0.60.10.3
0.10.10.2
34
Implementing Implementing HMMHMM to speech to speech ModelingModeling
( (TrainingTraining and and Recognition Recognition ))
- Building HMM speech models based - Building HMM speech models based on the correspondence between the on the correspondence between the observation sequences observation sequences YY and the state and the state sequence (sequence (SS). ). (TRAINNING).(TRAINNING).- Recognizing speech by the stored - Recognizing speech by the stored HMM models HMM models and by the actual and by the actual observation Y. observation Y. (RECOGNITION)(RECOGNITION)
Training HMM
Feature Extraction
RecognitionW*Y
Y
S
Speech Samples
35
RECOGNITIONRECOGNITION Process Process Given an input speech Given an input speech S=(sS=(s11,s,s22,…,s,…,sTT)) be the recognized . be the recognized . xxtt be the feature samples computed at time be the feature samples computed at time tt, where the feature , where the feature
sequence from time sequence from time 11 to to t t is indicated as: is indicated as: X=(xX=(x11,x,x22,…,x,…,xt t )).. The recognized states The recognized states S*S* could be obtained by: could be obtained by:
S*=ArgMax P(S,X|S*=ArgMax P(S,X|))..
Dynamic Structure
Search Algorithm
S*
Static Structure
St , P(xt,{st}|{st-1},)
{St-1}
xt
36
ResultResult ((SSpeaker peaker RRecognition)ecognition)
37
Table 1: Speaker recognition result
ResultResult ((IIsolated solated SRSR))
38
Table 3: Result for isolated speech recognition.
Result Result ((CContinuous ontinuous SRSR))
39
Table 3: Continuous Speech recognition result
ConclusionsConclusions No speech recognizer till now has No speech recognizer till now has
100% accuracy. 100% accuracy.
You should avoided poor quality You should avoided poor quality microphone consider using a better microphone consider using a better microphonemicrophone
On important matter is that , On important matter is that , training the computer will provide training the computer will provide an even better experience.an even better experience.
40
Thank YouThank You
41