opportunistic sensing with mic arrays on smart speakers for
TRANSCRIPT
![Page 1: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/1.jpg)
Opportunistic Sensing with Mic Arrays on Smart Speakers for
Distal Interaction and Exercise Tracking
Anup Agarwal, Mohit Jain, Pratyush Kumar, Shwetak Patel
![Page 2: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/2.jpg)
Smart SpeakersNew class of voice-only devices offering hands-free interaction
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 2
![Page 3: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/3.jpg)
Smart SpeakersNew class of voice-only devices offering hands-free interaction
35.6M smart speakers sold in US in 2017, 129% more than 2016
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 3
Q4
![Page 4: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/4.jpg)
Smart Speakers
To increase the device’s range for recognizing voice commands from across the room using beamforming
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 4
Sonos One 6Apple Homepod 6 Amazon Echo 7
![Page 5: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/5.jpg)
Smart Speakers
To increase the device’s range for recognizing voice commands from across the room using beamforming
Beamforming: The signals from the each mic are combined in a way that signals coming from a certain direction in space interfere constructively while others interfere destructively.
Delay-and-Sum beamforming
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 5
Sonos One 6Apple Homepod 6 Amazon Echo 7
![Page 6: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/6.jpg)
ProblemFor certain scenarios voice-only interaction may not be sufficient.
• For instance, when you are busy on phone and want the smart speaker to shut up (without saying it aloud)
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 6
![Page 7: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/7.jpg)
ProblemFor certain scenarios voice-only interaction may not be sufficient.
• For instance, when you are busy on phone and want the smart speaker to shut up (without saying it aloud)• Perform a simple hand gesture (similar to stop sign) to shut it up.
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 7
![Page 8: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/8.jpg)
ProblemFor certain scenarios voice-only interaction may not be sufficient.
• For instance, when you are busy on phone and want the smart speaker to shut up (without saying it aloud)• Perform a simple hand gesture (similar to stop sign) to shut it up.
No notification• User needs to explicitly ask a smart speaker to give notifications.
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 8
![Page 9: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/9.jpg)
ProblemFor certain scenarios voice-only interaction may not be sufficient.
• For instance, when you are busy on phone and want the smart speaker to shut up (without saying it aloud)• Perform a simple hand gesture (similar to stop sign) to shut it up.
No notification• User needs to explicitly ask a smart speaker to give notifications. • The smart speaker detects when a person entered the room, and starts
proactive notifications.
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 9
![Page 10: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/10.jpg)
Aim
Leverage the mic array in smart speakers for opportunistically sensing gestures
and classifying and counting exercises
Smart Speakers for Distal Interaction and Exercise Tracking20/04/18 10
![Page 11: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/11.jpg)
Aim
Leverage the mic array in smart speakers for opportunistically sensing gestures
and classifying and counting exercises
(without speaking aloud)Smart Speakers for Distal Interaction and Exercise Tracking20/04/18 11
![Page 12: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/12.jpg)
Doppler Shift
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 12
![Page 13: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/13.jpg)
Doppler Shift
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 13
![Page 14: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/14.jpg)
Doppler Shift
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 14
![Page 15: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/15.jpg)
System
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 15
![Page 16: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/16.jpg)
System
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 16
![Page 17: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/17.jpg)
System
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 17
![Page 18: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/18.jpg)
System
fr = ft * (c+v)/(c-v)
fr = frequency recorded by mic {farthest from pilot in the interval [ft-2, ft+2] kHz above 5dB threshold}ft = pilot tone frequency c = speed of sound in airv = speed of body movement towards the mic
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 18
![Page 19: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/19.jpg)
System
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 19
![Page 20: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/20.jpg)
System
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 20
veloctiy-time curve divided into 5s chunks
with 200 ms stride length
![Page 21: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/21.jpg)
Hardware
MiniDSP UMA-8 circular USB mic array
7 MEMS microphones
Radius 43 mm
Sampling rate 48kHz (Fs)
Capturing 24 bits per sample
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 21
![Page 22: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/22.jpg)
Data Collection: 1
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 22
0.5m
1 62 203 4 5 7 …..
![Page 23: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/23.jpg)
Data Collection: 1
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 23
0.5m
20 markers, 0.5 m away
Forward (pushing hand away from body)Backward (pulling hand towards the body)
10 times at each marker
Two pilot tones: 20 kHz and 6kHz
12 participants (10 male, 2 female)
1 62 203 4 5 7 …..
Age = 22.4 ± 4.3 years Weight = 73 ± 10.1 kgs Height = 172.5 ± 8.7 cm
![Page 24: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/24.jpg)
Results: 1
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 24
![Page 25: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/25.jpg)
Results: 1
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 25
![Page 26: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/26.jpg)
Data Collection: 2
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 26
![Page 27: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/27.jpg)
Data Collection: 2
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 27
10 exercises, 20 repetitions each2.5m from the device20 kHz pilot tone
17 participants (15 male, 2 female)
Age = 26.4 ± 4.4 yearsWeight = 73.6 ± 12.3 kgsHeight = 174 ± 9.6 cmsAverage fitness = 3.4 ± 0.8Daily exercise = 6/17Exercise 2-3 times a week = 4/17
![Page 28: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/28.jpg)
Results: Exercise Identification
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 28
![Page 29: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/29.jpg)
Results: Exercise Identification
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 29
99.8% on the training set 95.9% on the evaluation set
![Page 30: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/30.jpg)
Results: Confusion Matrix
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 30
![Page 31: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/31.jpg)
Results: Confusion Matrix
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 31
![Page 32: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/32.jpg)
Results: Exercise Counting
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 32
x x x
![Page 33: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/33.jpg)
Results: Exercise Counting
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 33
1.Cross
Stretch
3.Folded Shoulder
Rotation C
4.Folded Shoulder
Rotation AC
6.Jumping
Jacks
7.Leg
Raise
8.Shoulder
Rotation C
9.Shoulder
Rotation AC
Accuracy (m)
85.7 91.3 94.7 86.7 97.0 95.0 92.2
sd 15.8 16.2 5.1 19.0 4.8 3.7 6.6
91.8% accuracy
x x x
![Page 34: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/34.jpg)
Limitations & Future Directions
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 34
6 mics43 mm radius
![Page 35: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/35.jpg)
Limitations & Future Directions
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 35
6 mics43 mm radius
24 mics43 mm radius
![Page 36: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/36.jpg)
Limitations & Future Directions
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 36
6 mics43 mm radius
6 mics8.8 mm radius
24 mics43 mm radius
![Page 37: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/37.jpg)
Conclusion
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 37
Accurately tracking hand movement gestures (96.8%) from a distance of 2.5m
Classifying 10 exercises accurately (96%)
Counting 7 exercises accurately (91.8%)
![Page 38: Opportunistic Sensing with Mic Arrays on Smart Speakers for](https://reader033.vdocuments.site/reader033/viewer/2022051420/627e3de555bb7d47ac1a1956/html5/thumbnails/38.jpg)
Thank You!
Mohit JainIBM Research, India: [email protected]
University of Washington, Seattle USA: [email protected]
20/04/18 Smart Speakers for Distal Interaction and Exercise Tracking 38