Speech Based Optimization Speech Based Optimization of Hearing Devicesof Hearing Devices

Alice E. Holmes, PhDAlice E. Holmes, PhDCollege of Public Health & Health College of Public Health & Health

ProfessionsProfessions

Research TeamResearch TeamAlice E. Holmes Alice E. Holmes – AudiologyAudiology– UF Communicative DisordersUF Communicative Disorders

Rahul ShrivastavRahul Shrivastav – Speech ScienceSpeech Science– UF, Communication Sciences and DisordersUF, Communication Sciences and Disorders

Lee KrauseLee Krause– EngineerEngineer– AudigenceAudigence

Purvis BedenbaughPurvis Bedenbaugh– NeuroscienceNeuroscience– East Carolina University East Carolina University

The ProblemThe Problem

Programming is based on electrically Programming is based on electrically measured dynamic ranges of pulsed measured dynamic ranges of pulsed stimuli (non-speech)stimuli (non-speech)

Current programming methods have Current programming methods have numerous optionsnumerous options

PurposePurpose

The goal is to understand speech, the tuning The goal is to understand speech, the tuning of the device should be based on speech and of the device should be based on speech and not tones.not tones.

Development of a standard metric to Development of a standard metric to understand the strengths and weaknesses of understand the strengths and weaknesses of the individual CI user.the individual CI user.

An automated process needs to exist to An automated process needs to exist to optimize the CI mapping strategy based on optimize the CI mapping strategy based on input from the speech feature error matrix.input from the speech feature error matrix.

5

Vision born in 2002 after Lee Krause received cochlear implant

Strong grounding in clinical audiology, speech intelligibility through 5-year Univ. of Florida & Audigence interaction

Strong technical team assembled to solve complex problems using optimization theory

Innovative Patent Protected Solution Positioned for Success

Speech Science Technology

7,206,416 - Speech based optimization of Digital Hearing devices (US, Australia, EPO)•CIP Telephony domain (filed published)

Patient classification (filed Sept 2008 -US)Reduction in test time (filed Sept 2008 -US)Optimization Algorithms I -- Optimization Algorithms II (filed Aug 2008 -US)

Cochlear ImplantsCochlear ImplantsComprehensive approach for the Implant domainComprehensive approach for the Implant domain– Standard/automated fitting approach Standard/automated fitting approach – Improved device performance Improved device performance – Integrated rehabilitation Integrated rehabilitation – Patient population results used to drive future researchPatient population results used to drive future research– Supports telemedicineSupports telemedicine– Supports multiple languages Supports multiple languages

Original approach was designed as a two stage process Original approach was designed as a two stage process – First level of optimization focused on the signal (rate, loudness growth, FAT)First level of optimization focused on the signal (rate, loudness growth, FAT)– Second level of optimization was the fine tuning (individual sensor gain, Second level of optimization was the fine tuning (individual sensor gain,

frequency range) frequency range) Current work focused on individual parametersCurrent work focused on individual parametersFuture work focused on getting to optimization in less timeFuture work focused on getting to optimization in less time– Business rulesBusiness rules– Data miningData mining– Full automated (speech recognition)Full automated (speech recognition)

CI user speech feature battery test Develop Optimized

Map for CI device Using:

- Fuzzy logic - Genetic Algorithms - Model Field Theory

Audiologist updates CI Map

Re-evaluate the CI user with adjusted map repeat until map is optimized

Speech feature to CI device map parameter knowledge base

Speech based Optimization of Cochlear Implant Processor patient Study

Benefits: -Optimized Map for CI user -Improved hearing performance -Improved quality of life -Reduced cost of tuning procedure

Perform standard CI user evaluation: -HINT -CNC Monitor

Session results

Confusion Error Matrix

Monitor Optimization results

CI user with optimized map

Overview Overview

Cochlear Implant

Device, DBrain, B

Input Signal, Sinp

Output Signal, Sout

Intermediate Signal, Sint

inp int

int out

inp out

D S S

B S S

B D S S

inp outWe want :

. . . .

S S

i e B D I

Almost nothing is known about the function B

Stimuli (Stimuli (SSinpinp))

Chosen from a set of phonemes Chosen from a set of phonemes

Each phoneme is characterized by a set of Each phoneme is characterized by a set of 9 auditory distinctive features for English 9 auditory distinctive features for English [Jakobson, Fant & Halle, 1963][Jakobson, Fant & Halle, 1963]

Each feature is weighted by importance in Each feature is weighted by importance in the languagethe language

Device Parameters (D)Device Parameters (D)

Any device parameterAny device parameter-- E.g. Signal processing strategies, stimulation rate, pulse width, -- E.g. Signal processing strategies, stimulation rate, pulse width, threshold (T), comfort values (C), number of channels or maxima, threshold (T), comfort values (C), number of channels or maxima, deactivation of selected channels, frequency allocation, gain, global deactivation of selected channels, frequency allocation, gain, global T and C modifiers, Q-value, base level, jitter, channel orderingT and C modifiers, Q-value, base level, jitter, channel ordering

All of these parameters together characterize the function All of these parameters together characterize the function DD

Adjusting certain parameters to decrease errors in one feature might Adjusting certain parameters to decrease errors in one feature might lead to an increase in error in another featurelead to an increase in error in another feature-- How to adjust the parameters such that the overall performance is -- How to adjust the parameters such that the overall performance is enhanced?enhanced?-- An optimization problem-- An optimization problem

Artificial IntelligenceArtificial Intelligence

What to optimize?What to optimize?-- Minimize error function-- Minimize error function

From patient experiments, we can get data for different From patient experiments, we can get data for different values of the parameters and the corresponding errorsvalues of the parameters and the corresponding errors

-- -- The dimension of this data is equal to the number of The dimension of this data is equal to the number of independent parametersindependent parameters-- Many parameters, hence very high dimension leading -- Many parameters, hence very high dimension leading to the “curse of dimensionality”to the “curse of dimensionality”

i ii

ii

wn

w

th

th

: weight of the feature

: # errors in the feature

i

i

w i

n i

Artificial IntelligenceArtificial Intelligence

To reduce the complexity of the To reduce the complexity of the problemproblem-- Patient-independent knowledge should -- Patient-independent knowledge should be available (e.g. as rules)be available (e.g. as rules)-- Patient-specific knowledge should be -- Patient-specific knowledge should be statistically extracted from the statistically extracted from the performance of each patientperformance of each patient-- Model field theory approach-- Model field theory approach

Initial Clinical TrialInitial Clinical Trial20 adults with 20 adults with

– N24 or New Freedom implantsN24 or New Freedom implants– Freedom ProcessorsFreedom Processors

Adjusted the following parametersAdjusted the following parameters– RateRate– Loudness growthLoudness growth– Frequency allocation tables Frequency allocation tables Outcome measures

– CNC lists in quiet – BKB-SIN– Subjective questionnaire

Subject DemographicsSubject Demographics

GenderGender

MaleMale FemaleFemale

N=7N=7N=13N=13

Age (Years)Age (Years)

MeanMean S.D.S.D. RangeRange

57.357.319.919.9

24-8224-82

Length of CI Use (months)Length of CI Use (months)

MeanMean S.D.S.D. RangeRange

25.625.628.9728.975-1155-115

Type of CIType of CI

N24N24 New FreedomNew Freedom

N=3N=3N=17N=17

Initial Clinical TrialInitial Clinical Trial

The Optimization program was designed to interface with a customized version of Cochlear Corp. Custom Sound so that programming changes recommended by the algorithm could be tested seamlessly. All stimuli were presented through a direct connection to the speech processor and at a constant level across all test sessions (approximately 60 dBA). 3 Sessions – two weeks apart3 Sessions – two weeks apart

Session 1Baseline performance obtained using subject’s current map on outcome measures The T- and C- values were obtained at multiple pulse ratesThe optimization routine completedMap with lowest NWE was selected and programmed in to processer for use until next session

Optimization Optimization (Clarujust™ ) )A series of VCV syllables were presented & verbal responses were recorded by the researcher. NWE for the processor setting was calculatedThe next combination of FAT, PR & LG was automatically recommended & tested.Procedure was repeated for 30 minutes

Session 2Session 2

Outcome performance using Opt 1 was evaluated using CNC lists in quiet and BKB-SIN measurements as reported in Session 1.

The Optimization procedure described above was then repeated to obtain Optimization 2 (Opt 2) and programmed into their speech processor.

Subjects were asked to use the optimized map until Session 2.

Session 3Session 3

Outcome performance using Opt 2 was evaluated using CNC lists in quiet and BKB-SIN measurements as reported above.

Subjects then chose the maps that they wanted saved in their speech processors for regular/everyday use.

Subject Map Parameters (Number of Subjects)

 

Stimulation Rate

 

250 500 720 900 1200 1800

Baseline 1 5 3 8 2 1

Opt 1 2 4 8 4 1 1

Opt 2 2 2 5 4 2 5

 

Loudness Growth (LG)

 

10 15 20 25 30

Baseline 1 18 1

Opt 1 6 2 5 5 2

Opt 2 4 3 3 5 5

 

Frequency Allocation Table (FAT)

188-7938

188-7438

188-6938

188-6563

188-6063

188-5938

188-5813

Baseline 19 1

Opt 1 6 1 8 1 2 2

Opt 2 4 6 4 2 1 1 2

CNC Word Scores

-15

-5

5

15

25

35

45

55

65

75

85

95

118 106 110 114 101 103 116 102 120 121 104 119 113 112 105 109 108 107 117 115

Subject Number

Per

cen

t C

orr

ect

-15

-5

5

15

25

35

45

55

65

75

85

95

Opt 1 gain

Opt 2 gain

Series3

Baseline

Opt 1

Opt 2

CNC Phoneme Scores

-30

-10

10

30

50

70

90

110

130

150

170

190

210

230

250

270

290

118 106 110 114 101 103 116 102 120 121 104 119 113 112 105 109 108 107 117 115

Subject Number

Ph

on

emes

co

rrec

t

-30

-10

10

30

50

70

90

110

130

150

170

190

210

230

250

270

290

Opt 1 gain

Opt 2 gain

Series3

Baseline

Opt 1

Opt 2

BKB-SIN Scores

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

14

16

18

20

114 106 120 118 110 101 102 103 104 121 119 113 108 105 107 116 112 109 117 115

Subject Number

Sig

nal

-to

-No

ise

Rat

io (

SN

R)

in d

B

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

14

16

18

20

Opt 1 gain

Opt 2 gain

Series3

Baseline

Opt 1

Opt 2

CNC Word ANOVA-R• Significant difference among

the three conditions using Greenhouse-Geisser analysis (p < 0.004).

• Further trend analyses indicated a significant ascending omnibus trend from baseline (p < 0.004)

• Pairwise comparisons significant differences between baseline and Opt 1 (p < 0.025) and between Baseline and Opt 2 (p < 0.015).

CNC Word RAU Opt 2 Session 3CNC Word RAU Opt 1 Session 2CNC Word RAU Baseline

Me

an

60.00

40.00

20.00

0.00

Error bars: +/- 2 SE

CNC Phoneme ANOVA-R• Significant difference among

the three conditions using Greenhouse-Geisser analysis (p < 0.008).

• Further trend analyses indicated a significant ascending trend from baseline (p < 0.015)

• Pairwise comparisons showed significant differences between base line and Opt1 (p < 0.003) and between Baseline and Opt 2 (p < 0.04).

CNC Phoneme RAU Opt 2 Session 3

CNC Phoneme RAU Opt 1 Session 2

CNC Phoneme RAU Baseline

Me

an

100.00

80.00

60.00

40.00

20.00

0.00

Error bars: +/- 2 SE

BKB-SIN ANOVA-R• Significant difference

among the three conditions using Greenhouse-Geisser analysis (p < 0.03).

• Further trend analyses indicated a significant ascending quadratic trend from baseline (p < 0.009).

• Pairwise comparisons showed significant differences between baseline and Opt 1 (p < 0.03)

BKB Opt 2 Session 3BKB Opt 1 Session 2BKB Baseline

Me

an

12.50

10.00

7.50

5.00

2.50

0.00

Error bars: +/- 2 SE

Subjective ResultsSubjective Results

At the end of this clinical trial, 17 out of 20 patients preferred to continue using one of their optimized maps.

Subjective ratings in various situations were also obtained from each subject (Holden, et al, J Am Acad Audiol 18:777–793, 2007)

Subjective Performance in 19 Listening Situations

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50

Conversation on the Telephone

Message on the Answering Machine

News on TV

Movies/Dramas/Sitcoms on TV

Radio in the Car

Radio at Home

Lyrics to Music

Conversation at Dinner Table

Conversation in Quiet with One

Conversation in Quiet with Several

Conversation in a Car

Conversation at Social Gathering

Conversation at Restaurant

Conversation with Cashier

Conversation with a Child

Conversation Outside

Someone in the Distance

Church Service

Meeting in a Large Room

Subjective Performance

Optimization 2

Optimization 1

Baseline

ConclusionsConclusions

• The optimization method used in this study resulted in improved subject performance in all outcome measures.

• Speech perception was significantly better in word and phoneme identification with optimized maps.

• In addition, subjects performed better in noise using the optimized maps.

• Subjective tests suggest that patients preferred the optimized maps in their daily lives.

What is Next?What is Next?

One patient has been successfully mapped from One patient has been successfully mapped from his initial hook-uphis initial hook-upContinue to refine process with CI technologyContinue to refine process with CI technologyCurrently doing a clinical trial with hearing aid Currently doing a clinical trial with hearing aid programmingprogrammingFuture applicationsFuture applications– HybridsHybrids– Audiologic rehabilitationAudiologic rehabilitation– Cell phonesCell phones– ????????

Thank you to the students involvedThank you to the students involved

Hannah SiburtHannah Siburt

Kevin StillKevin Still

Elyse SwartzElyse Swartz

Bekah GathercoleBekah Gathercole

AcknowledgmentsAcknowledgments

This project is funded by Audigence, Inc. and the Florida High Tech Corridor Council.

We wish to thank Cochlear Corporation for supplying the fitting software platform and for their extensive and timely technical support.

We also want to thank our subjects for their willingness to participate in the experiment.