3.0 Speech Synthesis

Imperial Academy of St Petersberg put the following questions as the subject of their annual prize.

1. What is the nature and character of the sounds of the vowels a, e, i, o, u that make them so different from one another?

2. Can an instrument be constructed like the vox humana pipes of an organ which shall accurately express the sounds of the vowels.

Winner was Kratzenstein 1779

A E I O U

Von Kempelen’s speaking machine (1791)(Wheatstone’s reconstruction

Voder (Voice Operation Demonstrator) 1939

3.0 Speech Synthesis

The emphasis these days is very much on text-to-speech(TTS) synthesis and we shall concentrate on this method.

One can consider TTS systems as consisting essentially of two components – a high-level component and a low-level component – or perhaps as a front end and back end.

Current speech synthesis systems may be classified, depending largely on their low-level component, as

formant parameter systems

waveform concatenation systems

Audio clips of synthetic speech illustratingthe history of the art and technology of synthetically produced human speech.

http://www.cs.indiana.edu/rhythmsp/ASA/Contents.html

http://www.cs.indiana.edu/rhythmsp/ASA/highlights.html

http://www.humnet.ucla.edu/humnet/linguistics/faciliti/demos/vocalfolds/vocalfolds.htm

Human speech

Parametric analysis

Normalisation

SPRUCE Text-to-Speech Synthesis

Build syllable sentence

Parse sentence

Perform prosodic processing

High level output

Text input

DictionaryRules

Pragmatics

Retrieve andconjoin syllables

Assemble output file

Holmes synthesiser

Retrieve andconjoin syllables

Assemble output file

DAC

Syllable inventoryInventory

ofwaveform

units

This may also be referred to as synthesis-by-rule although rules of one sort or another are common to all synthesis systems. For a formant synthesis system the output of the high-level component typically consists of a sequence of allophones together with their duration and pitch, e.g.

DH 7 34 I 5 34 S 8 33

Formant Synthesis

Duration measured in 10ms frames

Pitch coded into the range 1-63

The low-level component uses this input to provide a sequence of frames, each frame containing a set of parameters referring to formant frequencies, formant amplitudes, voicing, fundamental pitch, etc., e.g.

Fn alf f1 a1 f2 a2 f3 a3 ahf s f0 250 33 280 33 1300 32 2680 34 41 36 34 250 37 280 37 1300 36 2680 38 45 36 34 250 40 280 40 1300 39 2680 41 48 36 34 250 42 280 42 1300 41 2680 43 50 36 34 This information is then fed into a formant synthesiser which uses it to generate the appropriate audio output. The formant synthesiser may be implemented in hardware or software.

An example of a formant synthesis system is DECTALK.

RZ R1 R2 R3 R4RN

R1

R2

R3

R4

R5

R6

Glottal filter

Impulse train

Random numbers

LP filter

A1

A2

A3

A4

A5

A6

Pre-emphasis+

Parallel

R5

Synthetic speech output

Cascade

Klatt Synthesiser

Klatt synthesiser

A combined serial/parallel formant synthesiser.

A serial, or cascade, synthesiser is a better model for the production of vowel and vowel-like sounds whereas a parallel synthesiser is better suited to producing nasals, fricatives and stops.

The serial synthesiser specifies the formant centre frequencies and bandwidths.

The parallel synthesiser specifies the formant levels (peak amplitudes) also.

Waveform Concatenation Synthesis

With this system the low-level component generates a speech output file by concatenating units of previously recorded speech. Informationabout the duration and pitch of these units is again supplied by the high-level component. The size of the units is clearly an important consideration and both in terms of amount of storage required and the difficulties involved with joining them together (more about this later).

An example of a waveform concatenation synthesis system is the Lernout and Hauspie TTS system.

cer- -tain- -ly(ser-) (-tan-) (-lee)

What are the options for stored units?

Sentences

Phrases

Words

Syllables

Phonemes

Demi-syllables

Diphones

Stored Speech Units

words

How many? ~ 300,000Perhaps 2000 for most frequently used words.

Formant synthesis1 word ~ 0.5sec12bytes per 10msec frame1 word = 0.5/.010 frames = 50 frames = 600 bytes

300,000 words ~180Mbyte

Waveform concatenation1 word ~ 0.5 secSampling rate = 16Khz1 word = 8K samples = 16K bytes at 2 bytes per sample

300,000 words = 4.8Gbyte

morphemesBasic meaningful units that make up words, essentially roots, prefices, sufficese.g. sail, travel, -ed, -s => sail, travel sailed, travelled sails, travels ~10,000-30,000 entries

Formant synthesis30,000 entries ~ 10-15Mbyte

Waveform concatenation30,000 entries ~ 120Mbyte

syllables

How many? ~ 10,000??

Formant synthesis1 syllable ~ 0.3sec12bytes per 10msec frame1 syllable = 0.3/.010 frames = 30 frames = 360 bytes

10,000 syllables ~3.6Mbyte

Waveform concatenation1 syllable ~ 0.3 secSampling rate = 16Khz1 syllable = 4.8K samples = 9.6K bytes at 2 bytes per sample

10,000 syllables = 96Mbyte

phonemes

Formant synthesis.

1 phoneme ~ 10 frames = 120 bytes

40 phonemes ~5Kbytes

Actually need about 70-80 allophones,giving ~ 10Kbytes

How many? About 40 for English.

Waveform concatenation

1 phoneme ~ 100msec = 0.1sec = 1.6K samples = 3.2K bytes

Total ~ 256K

demi-syllables

s u m t ie m z (sometimes)

These are units of speech obtained by making cuts in the middle of the vowel part of the syllable.The reason for doing this is that coarticulation effects are minimal in the middle of the vowel.The number of demi-syllables is about 4-5000 made up from about 1500 initial demisyllables and 3000 final demisyllables.

s u m - t ie m z

cer- -tain- -ly(ser-) (-tan-) (-lee)

Demi-syllables

diphones

k u n uu (canoe)

These are units of speech obtained by going from the middle of onephone to the middle of another.The reason for doing this is that coarticulation effects are minimal in the middle of the sound.Theoretically there are about 40x40 = 100 diphones but in practice thenumber is about 1200.

qk | ku | un | nuu | uuq

q = silence

cer- -tain- -ly(ser-) (-tan-) (-lee)

Diphones

Words versus phonemes

Trade off between storage and processing

The larger the unit then the more storage space it requiresbut in compensation less effort is required in joining the units together.

Pronunciation Task

Input Text

AllophoneTask

Allophonic Rules

Phonemic Text

Broad Phonetic Text

Affix Tables

Pronunciation Rules

Restricted Text

Phonemic Text

Conversion Task

Exceptions Dictionary

Restricted Text

ProsodyTask

Prosody Table

Broad Phonetic Text

Narrow Phonetic Representation

Lower Phonetic Task Lower Phonetic

Table

Narrow Phonetic Representation

Control Parameters

Phonotactic Tables

Speech

Conversion Task

Pronunciation Task

Prosody Task

Lower Phonetic Task

Input textExceptions Dictionary

Phonotactic Tables

Affix Tables

Pronunciation Rules

Prosody Table

Lower Phonetic Table

The JSRU TTS SystemSpeech Output

This task converts unrestricted text to restricted text. Unrestricted text consists of non-English words, abbreviations (e.g. Dr.), unusual pronunciation, words to be spelt (e.g. BBC), etc. This task also gets the phonetic form of words that are in the dictionary and deletes any redundant white space.

The date is 1/10/97. …the first of October, ninety ninety seven.

I can’t do it. …I cant do it. Or I cannot do it

The price is £23.99. … twenty three pounds ninety nine pence

St. George St. …Saint George Street

well-behaved …well behaved

CONVERSION TASK

PRONUNCIATION TASK

1. Look up entire word in the dictionary.If found then exit from pronunciation task.

2. Remove suffixes.After removing a suffix look up remainder of word in the dictionary. If found go to 5 else try to remove a further suffix.Continue until no further suffixes can be removed.

3. Remove prefixes.

4. Perform pronunciation rules on the stem.

5. Replace prefixes and suffixes in their phonetic form, of course,obtained from lookup tables.

6. Perform stress assignment by applying the stress rules.

7. Divide the word into syllables.

1. Look up the entire word in the dictionary. If found then exit from pronunciation task.

sentence “sen-tanssensitive “sen’si-tivtransport “traans-poatcough “kof

 Conversion of words to phonemes is far from regular and is very context dependent in English. Some languages are better than others in this respect.

 George Bernard Shaw is quoted as saying that fish is spelt GHOTI??

 The advantage of using a dictionary is that it can include information on syllables, stress, syntactic types, etc. This is much more easily obtained using a lookup table than by using algorithms. For example, stress markers and syllable boundaries are more easily identified. But, of course, there will always be some words which are not in the dictionary, however large one makes it.

2. Remove suffixesAfter removing a suffix look up the remainder of the word in the dictionary. If found then go to 5 else try to remove a further suffix. Continue until no further suffixes can be removed.

e.g. wonderfully -> wond - er - ful – ly 

First remove –s

parties -> party -s

The ‘e’ needs to be deleted for words like dishes because otherwise the final ‘e’ lengthens the previous vowel (magic ‘e’ rule). e.g. fades -> fade -s

fishes -> fish -s

All other suffixes treated consistently.

Example of function for -s removal (not definitive)

lastchar = Sif (prevchar=S) then

returnelse if (prevtwochars = IO) then

remove Selse if (prevchar=vowel but not E) then

returnelse if (prevchar=E) then

remove Sif (prevprevchar=I) then

replace IE by Yelse if (prevprevchar=H) and (prevprevprevchar T) then

delete Eelse if ((prevprevchar=S) and (prevprevprevchar=S))

or ((prevprevchar=Z) and (prevprevprevchar=Z)) thendelete E

SS

IOS

IES -> Y

HES but not THES

SSES/ZZES

loss

folios

parties

batches/bathes

losses/buzzes

AS alias

A table of suffixes contains for each suffix:         its spelling         a list of letters that may precede          a list of suffixes which may precede          an indication of whether it can replace a final 'e'         its pronunciation         its stress class  For example, "ICAL" "" anything can precede it "ENT possible preceding suffixes 0 cannot replace a final 'e' "ikal" phonetic spelling -2 mainstress placed two syllables to the

left of the last syllable in the suffix

 e.g. radical stress on 'rad' identical stress on ‘ent'

The JSRU system has a list of 39 suffixes e.g. ED,NT, FUL, OUS, ALIC, IBLE, EN, etc.

 After a suffix has been removed, the stem is checked to see if it is long enough(at least 3 letters) and whether the final consonant cluster is pronounceable.

 If the suffix could have replaced a final 'e' then this 'e' must be added. The algorithm for deciding whether or not an 'e' has been removed is not simple.e.g.alternation -> alternate + ioninteraction -> interact + ion

Rule here is that if stem ends in vowel + consonant then an 'e' should be added. What about taxable? OK when 'x' replaced by 'ks'.Having removed a suffix, the reduced word is looked up in the dictionary. If it’s not found then an attempt is made to remove another suffix.

 e.g. wond - er - ful - ly

3. Remove prefixes in a similar fashion to the suffixes.

Similar procedure to removal of suffixes but with simplification that a prefix doesn't alter the stem.Table of prefixes contains for each prefix:         its spelling          list of letters which may follow it          list of prefixes which may follow it          its pronunciation          stress placement control  For example,"AL" *aeioul none of a, e, i, o, u, l "" any"awl" phonetic spelling0 prefix may be stressed

 e.g. always, already

4. After removal of suffixes and affixes we are left with the stem of the word.

Perform pronunciation rules on the stem.

 Pronunciation of the stem is worked out by a set of rules, unless it's found in the dictionary.

Three sets of rules

The first set consists of 105 rules, each one contains

         string to be changed to phonetic form

         left context

         right context

         replacement string

For example,

{"A", "", "consY>", "ai"} lazy

{"OUGH", "", "", "ou"} bough

{"AUGH", "", "", "aw"} daughter

{"C", "", "E", "S"} lace

{"PH", "", "", "F"} phase

What about laughter?

What about rough, cough, though?

The order of the rules is important. e.g.

Special procedures need to be called in some cases:

{"Q", "", "", "KW"} freeqent freekwent

{"E", "cons", "Q", "EE"} freqent freeqent

{"QU", "", "", "Q"} frequent freqent

N.B. Every rule is tried at every position in the word. Hence, time taken

depends very much on the number of rules.

{"vowel", "", ""consE", "@001"} Magic 'e' procedure

{"E", "", ">", "@002} final 'e' procedure

{"@003", "", "", ""} double letter procedure

5. Replace the prefixes and suffixes – in their phonetic form, of course, which can be obtained from lookup tables. This may result in some adjustments which are handled by the second set of rules. e.g.  {"c", "", "i", "s"} precious

{"c", "", "", "k"} practically

{"ig", "", "n-", "ie"} assignment

6. Perform stress assignment by applying the stress rules – these can be quite complicated. Make adjustments to final pronunciation e.g. reduction of some unstressed vowels.  The stress rules are based on MIT rules and are quite complicated. (From text to speech: the MITalk system Allen et al)

In the first (cyclic) phase several rules are applied in sequence first to the stem then to the stem + suffixes taken one at a time Prefixes are considered as part of the stem   Some affixes(e.g. -ING) do not affect the stress pattern so rules are omitted, while others(e.g. -ION) force stress onto a particular vowel, usually the one before the affix.

In the second (non-cyclic) phase one vowel is selected for mainstress and all others are reduced to secondary or no stress.

Stress Rules

Main Stress Rule (cyclic)

1. V -> [1-stress] / X – C0 {[short v] C01 / V} {[short V] C0 / V}

where X contains all prefixes and the symbol ‘-’ indicates the position of the vowel to be stressed C0

1 matches zero or one consonant C0 matches any number of consonants (including none) {..} denotes a list of alternative patterns separated by slashes /.

Assign 1-stress (primary stress) to the vowel in a syllable which precedes a weak syllable followed by a morph-final syllable containing a short vowel and zero or more consonants.

e.g. difficult - > d ” i f i k a l t X – C0 {..} {…}

V -> [1-stress] / X – C0 {[short v] C01 / V} {[short V] C0 / V}

where X contains all prefixes and the symbol ‘-’ indicates the position of the vowel to be stressed

Assign 1-stress to the vowel in a syllable preceding a vowel followed by a morph-final syllable containing a short vowel and zero or more consonants.e.g. secretariat -> sekret“eir ee aat X – C0 {..} {..}

Assign 1-stress to the vowel in a syllable preceding a vowel followedby a morph-final vowel.e.g. oratorio -> orat“oar ee oa X – C0 {..} {..}

2. V -> [1-stress] / X – C0 {[short V] C0 / V} where X contains all prefixes

Assign 1-stress to the vowel in a syllable preceding a short vowel and zero or more consonants

e.g. edit -> “ed it bitumen -> bity”uum en agenda -> aj”en da

3. V -> [1-stress] / X – C0

where X contains all prefixes

Assign 1-stress to the vowel in the last syllable

e.g. stand -> st”aand parole -> paar”oal

Stressed Syllable Rule (cyclic)

Alternating Stress Rule (cyclic)

Destressing Rule (non-cyclic)

Compound Stress Rule (non-cyclic)

Strong First Syllable Rule (non-cyclic)

Cursory Rule (non-cyclic)

Vowel Reduction Rule (non-cyclic)

After the stress assignment a third set of rules is applied. e.g.{"\141r", "", "cons", "\141"} far gone versus far away (141 = 'schwa'){"ir", "", "cons", "er"} dirt versus direct  

and some unstressed vowels are reduced.e.g. aa aai i u a o a  bottom => "bot-am

7. Divide the word into syllables. The rule used in the JSRU for this is pretty basic – not based on any principles of morphology. It simply places syllable boundaries half way between vowels.

e.g.

frequent -> freekwent -> freek – went

boundaries -> bound - reez

This task adds intonation and timing to the phonetic text. The task processes complete breath groups.

The output consists of a list of phonemes with corresponding pitch and duration.

PROSODY TASK

In November the region’s weather was unusually dry.

Do you want to travel to London?

Who is the Prime Minister of the Bahamas?

Lift the safety cover and press the red button.

Stress – two kinds•word stress•sentence stress

Word stressOne syllable in a polysyllabic word is given more prominence than the others.All words have one syllable with primary stress.

electricity university

tomorrow

controversy icecream

electricityuniversitytomorrow

controversy

icecream (or icecream?)

Examples

What time is the next train to London?

What time is the next train to London?

What time is the next train to London?

There’s John cycling down the road.

Sentence stressThis refers to the way in which one word is singled out in a sentence as the focus, or nuclear stress

Rhythm

Rhythm is about how we time the delivery of the sentence.

In English we are supposed to time the stressed syllables so thatthey occur equidistantly in time.

English is said to be a stress-timed language

The stressed syll–a-bles-in Eng-lish o - ccur e - qui- dis-tant-ly in time

# # # # # #

# # # # # # # # # # # # # # # #

clo pe die se lec tro niquescysenLe sontpleinesD’in for ma tions

Les encyclopedies electroniques sont pleines d’informations

French is said to be a syllable-timed language D Crystal 1995

•Set up the tone group boundaries.•Calculate durations of phonemes•Expand phonemes into lower phonetic segments•Set the pitch contour

Tone group boundaries are marked by a content word to the left and a function word to the right - provided this does not create a tone group without a content word.

JSRU PROSODY TASK

Utterance

Tone group 1 Tone group 2 Tone group N…….

/u u u / S u u S u u ……/ Smax/ u S u/ u /HeadPre-head Nucleus Tail

Projects represent a substantial part of your marks and you should spend some time choosing a project which will show you in your best light.

Projects represent // a substantial part // of your marks // and you should spend // some time choosing // a project // which will show // you in your best light.

“Pro-jects rep-res-”ent // a sub-”stant-ial “part // of your “marks // and you should “spend // some “time “choos-ing // a “pro-ject // which will “show // you in your “best “light.

Unstressed syllable weight 1Secondary stress “ 2Primary stress “ 3Emphatic stress “ 4

Tone groups are amalgamated such that all tone groups have a weight >=9.

“Pro-jects ‘rep-res-”ent // a ‘sub-”stant-ial “part // of your “marks //

and you should “spend // some “time “choos-ing // a “pro-ject

// which will “show // you in your “best “light.

Pitch contours are worked out for syllables according to their type, and the tone group type

Tone groups are classified as being of•Type 1 ends with a fullstop•Type 2 ends with a question mark•Type 3 ends with an exclamation mark•Type 4 any other tone group

10 10 5

6 8 5

5 9

See overheads!!!!

Baseline

When all syllable patterns have been calculated they are superimposed on a falling baseline

, ,

The value at the end of the phrase (indicated by the ,) is half that atthe beginning, and starting value for next phrase is 1.2 times the final value of previous one

TimingEach phoneme is represented by 1, 2 or 3 elements each of which hasan inherent duration and a minimum duration

Then actual duration = minimum + f * (inherent – minimum)

Rules for values of f (derived mainly by Klatt) are typically of the form

• For vocalic phonemes in a syllable which is not the last in a phrase multiply f by 0.6 (f initially set to 1)

• For phonemes with no stress or with secondary stress, multiply f by a factor as follows:

vocalic, word medial syllable 0.7other vocalic 0.85liquid or glide before a vowel 0.40all others 0.85

This task converts the output from PROSODY TASK into control parameters for the hardware synthesiser.

LOWER PHONETIC TASK

The table for each phonetic element contains information relating to howtransitions between target values are calculated. For each parameter in thetable there are entries for Its target value The proportion of the target of the dominated element used in deriving the boundary value   A fixed contribution to the boundary value  Transition duration within the dominant element      Transition duration within the dominated element

Target Fixed

Contribution

Proportion Internal

Duration

External Duration

Rank

W 760 380 50 4 4 8

E 1900 950 50 3 3 2

It sends a frame of parameters to the synthesiser every 10msecs.

The boundary value is calculated as Fixed Contribution of Dominant element + Target of dominated element * proportion of dominant element i.e. 380 + 1900 * 0.5 = 1330

760 Hz

Target value for W

1330 Hz

1900 Hz

Target value for E

Internal duration4 frames

External duration4 frames

Target Fixed Contribution Proportion Internal Duration External Duration Rank

W 760 380 50 4 4 8

E 1900 950 50 3 3 2

Target Fixed Contribution

Proportion Internal Duration

External

Duration

Rank

DH 1300 650 50 2 5 6

A 1420 710 50 3 3 2

W 760 380 50 4 4 8

E 1900 950 50 3 3 2

L 880 440 50 0 6 11

Z 1720 860 50 2 5 6

DH A W E

1300

1360

1090

1330

1420

760

Boundary TargetValues Values

DH|A 650 + 1420 * 0.5 = 1360 DH 1330A|W 380 + 1420 * 0.5 = 1090 A 1420W|E 380 + 1900 * 0.5 = 1330 W 760

7 4 8 12

Tomorrow will be starting off grey and rather murky.

:"to-ma-rou /wil /bey :"star-ting /of :"grai /and :"raa-dha

:"mer-kee.

T 9 37TY 2 37TZ 3 37 O 10 37 M 9 35 A 4 35 R 4 34

OU 7 34OB 7 33 W 4 34 I 5 34

LP 4 32 B 5 33BY 1 33EY 5 33 S 11 36 T 6 35

TY 2 35AR 14 35 T 6 31TY 2 31 I 5 31NG 9 30 O 6 32 F 6 31 G 6 34GY 2 34 R 2 34AI 12 34AJ 9 32 A 4 32 N 6 31 D 2 30DY 1 30 R 7 33

AA 10 32DH 7 31 A 5 30 M 11 32ER 14 32 K 6 25KY 2 23 I 10 23QQ 51 22 Q 42 21

* Output from SYNTH: * fn alf f1 a1 f2 a2 f3 a3 ahf s f0* T 9 37 0 Frame 0 250 6 190 6 1780 6 2680 6 6 1 37 250 10 190 10 1780 10 2680 10 10 1 37 250 13 190 13 1780 13 2680 13 13 1 37 250 15 190 15 1780 15 2680 15 15 1 37 250 15 190 15 1780 15 2680 15 15 1 37 250 15 190 15 1780 15 2680 15 15 1 37 250 13 190 13 1780 13 2680 13 13 1 37 250 10 190 10 1780 10 2680 10 10 1 37 250 6 190 6 1780 6 2680 6 6 1 37* TY 2 37 0 Frame 9 250 15 190 15 1780 40 2680 51 58 1 37 250 15 190 15 1780 40 2680 51 58 1 37* TZ 3 37 0 Frame 11 250 6 190 6 1780 23 2680 28 40 1 37 250 10 190 10 1780 27 2680 32 44 1 37

BT1

Spruce 2

L&H 2

AT&T 2

Festival 2

BT 2

Mbrola 2

L&H 1

Spruce 1

Festival 1

AT&T 1

Mbrola 1