nih (nidcd) initiative to develop real-time portable signal … · 2019-03-09 · nih (nidcd)...
TRANSCRIPT
NIH (NIDCD) initiative to develop real-time portable signal processing tools for advancing research on hearing loss compensation
1
Unless otherwise indicated the information provided is based on the author’s personal analysis and opinions relative to the NIH (NIDCD*1) initiative
*1 The National Institute for Deafness and Other Communication Disorders
Roger Miller & Amy Donahue (2016 IHCON)
“These research tools will
• lower barriers for hardware and software development,
• and facilitate translation of these advances into widespread use with hearing aids, cochlear implants, and consumer electronics devices.“
.
2
Projects funded
Six awards were given starting July 2016, and one July 2017
3
RO1 AwardSmartphone‐Based Open Research Platform for Hearing Improvement Studies Issa Panahi, Nasser Kehtarnavaz, and Linda Thibodeau University of Texas, Dallas, USA
Android and iOS systems, running C shells to:
• Use the phone as a remote microphone
• Wired and wireless version
• Use the phone itself as a hearing aid
• Wired and wireless version
• 4
RO1 Award
A Real-time, Open, Portable, and Extensible Speech Lab
Hari Gurudadri, UC San Diego
•Initial MHA implementation on Os X and Linux Computers
•Portable platform based on Snapdragon 8016 and Debian Unix.
•Single microphone Receiver-in-the-canal BTE delivery system
•http://openspeechplatform.ucsd.edu/index.html 5
SBIR Award: Development of an Open Speech Signal Processing Platform Ray Weber & Ross Snider, Flat Earth, Inc, Bozeman, MT, USA
• Based on the SoC FPGAs which allows for a deterministic low-latency signal
processing
• Applications run on top of Linux, which is running on the dual ARM Cortex A9 CPU
• Applications interact with the FPGA fabric
• I/O daughter cards:
• analog and digitals mics; speaker driver; headphone driver; BT Transceiver; Wi-Fi
Transceiver, etc.
https://flatearthinc.com/open-speech-platform-0
6
SBIR Award: Open Hardware and Software System for Speech and Signal Processing
Rafael Delgado, Intelligent Hearing Systems, Miami, FL
7
• Users can interface directly to the DSP system hardware
using a Dynamically Linked Library (DLL) and Visual
Control Library (VCL).
• Languages that support dlls include most high level
programming languages such as C++, Delphi (Visual
Pascal) and Visual Basic. Other development packages
that also support the use of dlls include MatLab and
LabView.
http://ohsspds.ihsys.info
SBIR Award: Open Source System for Audio Processing
Odile Clavier, Creare Inc., Hanover, NH, USA
8
•1st generation hardware: Teensy 3.6 + custom audio interface board •Relies on Arduino software development environment
•Next generation (Rev D, this spring)): ARM M4
https://tympan.org
Open Portable Platform for Hearing Aid Research
Caslav (Chas) Pavlovic1, Hendrik Kayser2, Paul Maanen3, Tobias Herzke3, Volker Hohmann1,3, Prakash S.R.1,4, and Reza Kassayan11 BatAndCat Corporation 2 University of Oldenburg 3 HörTech gGmbH Oldenburg 4 EarLens Corporation
Research reported in this presentation was supported by the NIDCD of the NIH. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
The BTE Case Design supported by EarLens Corporation
We report on two closely related projects:RO1 Award R01DC015429 Volker Hohmann and Chas Pavlovic July 2016
SBIR Award R44DC016257 Chas Pavlovic July 2017
Hardware Software
Software: Open Master Hearing Aid (open MHA)
11
REQUIREMENTS
• end to end latency (fully loaded)< 10 ms • programmable in C++ • Linux, MacOS, Windows environment • hardware-independence
openMHA - current status
12
Hearing aid processing modules, release 4.8.0: • calibration • multi-band dynamic compressor • adaptive feedback cancellation • single-channel noise reduction • adaptive differential microphone • binaural coherence filter • binaural beamforming algorithms • sound source localization
Lin
calibrationF B
Rin
B F
ADMADM
coherence
DC
Lout Rout
13
Designed for different user groups
Clinicians who
• use “Plug and Play” software modules
• change processing parameters at high level
14
Designed for different user groups
Audiological Researchers and Application Engineers
• set up measurement tools and customize algorithms
• access configuration interface as a text file
nchannels_in = 2 fragsize = 64 srate = 44100
# MHA library name mhalib = transducers
# IO plugin library name iolib = MHAIOFile
mha.plugin_name = overlapadd
mha.calib_in.peaklevel = [90 90] mha.calib_out.peaklevel = [90 90]
mha.overlapadd.fftlen = 256 mha.overlapadd.wnd.len = 128 mha.overlapadd.plugin_name = mhachain
# list of plugins mha.overlapadd.mhachain.algos = [ ... fftfilterbank ... dc ... combinechannels ... ]
# Frequency bands mha.overlapadd.mhachain.fftfilterbank.f = [200 2000]
# gaintable data in dB gains mha.overlapadd.mhachain.dc.gtdata = [[10 -10 -30];[20 -25 -50];[10 -10 -30];[20 -25 -50]]
# input level for first gain entry in dB SPL mha.overlapadd.mhachain.dc.gtmin = [0]
# level step size in dB …
15
Plugin developers who can • Develop and implement new
plugins in the openMHA framework
• For example a new C++ routine for noise reduction
Designed for different user groups
openMHA – structure Basic framework • MHA host application • toolbox with plugins • communication interfaces (I/O) • Matlab/Octave GUI for audiological usage
Documentation • manuals for different usage scenarios • example configuration files for algorithms
16
openMHA
plugins IO
audio backend(Jack, File, TCP)
MHA hostapplication
control applications(e.g., Matlab, Octave)
Hardware platforms 1. Desktop setup - off the shelf; available now
A simple version demonstrated today: • Windows, Linux or MacOs • Headphones that provide a good sound
isolation (e.g. audiometric audio caps) • A lapel microphone
More complex versions may include: • Multichannel audio interface for multiple
microphones • Actual hearing aid hardware for audio
transmission (speakers; multiple microphones etc .)
17
Hardware platforms 2. Single board computer and own codec board (showed today)available now• also serves as a development
board for fully portable version• features three codecs with 6
input and 6 output channels • preamplifiers • supports 2 hearing aids (with
directional microphones) and one binaural headset
18
19
Hardware platforms 3. PORTABLE PLATFORM: Single board computer and own codec board shown today Available; May 2019
L BTE R BTE:
Front and Back Mic Receiver in the canal
Each BTE:
Headphone Mic and Speaker
20
3. Portable platform - May 2019
Hardware platforms
21
Binaural BTE Sound delivery systemAvailable Summer 2019
Hardware platforms
• High end ID and mechanical/acoustical design : Earlens case adapted for MHA purposes
• Front and back MEMS microphones • Differential amplifiers • 8 wire low-noise cable
Preliminary measurements
22
Battery Life: 6 hours
Equivalent Electrical Input Noise: MEMS mike: 26 dB SPLPower supply and circuit: 20 dB SPL (does not include the earpiece cable noise) Expected total (microphone, cable, circuit and power supply) under 28 dB SPL
openMHA software runs successfully with the following latencies: direct routing through JACK server (MHA):3.2 ms with MHA in time domain: 3.9ms with STFT and overlap-add: 5.2 ms with full processing load: under 10 ms
23
24
Libraries and a fitting example available Summer 2019
Device Control and Fitting System
• WiFi at the moment; BLE summer 2019
• Example gui: Earprint format
• Earprint: A self-fitting system (Pavlovic et al. US 8,112,166; now property of HIMPP)
Other related interests - that we may use MHA for and implement in the MHA libraries:
A. Fitting
25
Fitting Process Initial Fitting (formula):
Fine Tuning
Further Specific Environment Adjustments
Audiologist
Audiologist Self
Self
Self
Self Adjustment Difficulties
1. N dimensional space of parameters affecting the preference (N definitely more than 2).
2. Multiple local maxima and minima.
26
27
Device Control and Fitting System
• Each of the three levels require a different parameter space
• The space needs to be arranged so that there is a smooth transition from one cell to another
• Multidimensional EarPrint is possible and needed
All 3 levels: Initial, Fine Tuning, and Environment adjustment
Earprint: Used by Sound ID, SoundHawk, Starkey
28
Device Control and Fitting System
• EarPrit has the same accuracy as any of the other best psychophisical scaling procedures (CS, ME) or the direct speech recognition testing - but in 10% of the time (Ramani, Michael, Pavlovic, JASA, 2011)
• CS, ME, and the Objective Speech Recognition testing for the same accuracy take the same amount of time (Pair Comparisons efficiency decreases rapidly as the number of stimuli increases, Purdy and Pavlovic, 1992)
• FUTURE: Test time and accuracy of all this methods can be substantially increased by the use of DNN.
• It is highly likely that, as data accumulates, the fine tuning to different environments will become, with the assistance of the DNN, totally automatic. • A person walks into a bar • The Environment data sample and/or the location Geo tag
are sent to the Cloud • Adjustments calculated by a Cloud algorithm based on:
• person’s previous adjustments in similar environments • and/or adjustments done by individuals with similar
hearing profile, age, etc. • and/or adjustments typically done in this particular
environment (GPS environment tag)
29
DNN and automated fitting
Other related interest that we will further explore in the MHA B.Connectivity
• BT has become almost ubiquitous in hearing aids • current uses: phone calls; remote mics; TV, etc. • WiFi in larger wearables • possibility to connect to Cloud to access big data
storage and and super computer power
30
THE MERGER OF HEARIG AIDS AND WEARABLES:
• Wearables: devices that feature a number of functions; such as playing music, heart rate monitor, blood pressure, oxygen level, instant language translation, etc.
• Trend: hearing aids are getting many features of wearables and wearables are getting hearing aid functionality (i.e. become “hearables”)
31
Nuheara Bragi
Market Size US (annual sales):
• Hearing Aids: 8 M customers • Headphones: 400 M customers
32
Price
• Hearing aids: $2000 • Headphones: $200
33
What do the following facts suggest?
• Equally sophisticated HW • 50 times bigger headphone volume • 10 time smaller headphone price • and, legal backing for OTC sales at least in the
US
34
Over-the-Counter Hearing Aid Act of 2017
35
Plus the future:
• Wearables becoming smaller and smaller • Backed by huge volume and huge competition • Hearing aid will become a feature of
wearables
36
but “hearing aids should be:
Invisible or hide effectively as parts of desirable objects”
What is a desirable object?
What if it provides extraordinary hearing even for normal hearing people?
In this regard we have evolved a long way:
• 2004: SoundID Device: 1st hearable - low sales
• 2008: SoundID BT headset secret hearable: “pass-through mode” separately sold remote microphone solid sales
In this regard we have evolved a long way:• 2004: SoundID Device: 1st hearable - low sales
• 2008: SoundID BT headset secret hearable: “pass-through mode” separately sold remote microphone high sales
• 2017: super high sales
• High quality stereo earphone
• Connects to phone • Personal binaural sound
amplifier • 3 microphones on each side • Direct wireless link between
multiple devices for voice communications (“Grid”) 40
Bat&Cat headset
Connectivity:person to person
Proprietary and Confidential – BatAndCat Corporation © 2015
Grid - example
41
Scaling speech understandingResults (simulated and actual restaurant)
0 SNR
-10 SNR
Specially Distributed Microphone Grid (BatAndCat: in development):
43
The AI based technologies can utilize the combined signals from distributed microphones to virtually eliminate noise and focus on the target speaker
Spatially Distributed Microphones
44
• microphones and transmitters present already in ubiquitous audio capable devices (Amazon Echo, Apple HomePod, smart phones, future watch phones)
• low cost microphones-transmitters integrated in home automation devices (switches, thermostats, lamps)
• objects specifically designed for this purpose (remote microphones)
For more information:
http://www.openmha.org/https://github.com/HoerTech-gGmbH/openMHAhttps://github.com/HoerTech-gGmbH/Cape4allhttp://batandcat.com/nih-supported-projects.html
45