rethinking hci with neural interfaces · 2020-06-14 · rethinking hci with neural interfaces...
TRANSCRIPT
![Page 1: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/1.jpg)
Rethinking HCIwith Neural Interfaces
@CTRLlabsco
Adam Berenzweig, Director of R&D, CTRL-labs@madadam | [email protected]
![Page 2: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/2.jpg)
![Page 3: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/3.jpg)
![Page 4: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/4.jpg)
1. A Brief History of User Interface Paradigm Shifts2. Introduction to Neural Interfaces3. Designing for Neural Interfaces
![Page 5: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/5.jpg)
![Page 6: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/6.jpg)
![Page 7: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/7.jpg)
![Page 8: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/8.jpg)
![Page 9: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/9.jpg)
![Page 10: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/10.jpg)
![Page 11: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/11.jpg)
![Page 12: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/12.jpg)
![Page 13: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/13.jpg)
![Page 14: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/14.jpg)
1. A Brief History of User Interface Paradigm Shifts2. Introduction to Neural Interfaces3. Designing for Neural Interfaces
![Page 15: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/15.jpg)
Human output30 bits/s
Human input10,000,000 bits/s
![Page 16: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/16.jpg)
![Page 17: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/17.jpg)
Cortical BMI is hard
![Page 18: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/18.jpg)
![Page 19: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/19.jpg)
Surface EMG neural interface.Biophysical signal to control signal.
Neural activity EMG Control
![Page 20: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/20.jpg)
In the future you will connect your nervous system to everything.
![Page 22: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/22.jpg)
1. A Brief History of User Interface Paradigm Shifts2. Introduction to Neural Interfaces3. Designing for Neural Interfaces
![Page 23: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/23.jpg)
Neural Interaction Design Challenges1. Robustness & Reliability2. Novel Interactions3. Ergonomics
a. Touchless Control vs. Spatial Computing4. Learnability & Customization
a. Fun to learn, and it learns you.5. Fit to Environment6. Props
![Page 24: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/24.jpg)
![Page 25: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/25.jpg)
Neural Interaction Design Challenges1. 3- and 6-DOF navigation2. Activation, “wake-word” & mode switching3. Simultaneous continuous & discrete control4. Hand-object interaction in XR.5. Text Input!!
![Page 26: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/26.jpg)
Ergonomics & Comfort1. Smaller is better -- motionless is best?2. Anatomically informed3. Variation is essential for comfort.4. Force: brief or light.
![Page 28: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/28.jpg)
Hand-Object Interaction in XR1. Grasping, manipulating.2. Spatial Computing vs.
Touchless Control, i.e. direct vs indirect.
3. Haptics & feedback.
Use your arm, or use The Force?
![Page 29: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/29.jpg)
Novel InteractionsCo-evolution of software and controllers.
Control these with an xbox controller… or your hand?
![Page 30: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/30.jpg)
Novel InteractionsCo-evolution of software and controllers.
Control these with an xbox controller… or your hand?
![Page 31: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/31.jpg)
Text Input!!1. One-handed vs two2. QWERTY-or-not3. Visual, audio feedback4. Swipe, chords, …?
We can do better than this...
![Page 32: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/32.jpg)
Props1. Constrain hand shape &
muscle activation2. Something to push against3. Feels cool4. Optional scaffolding?
![Page 33: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/33.jpg)
![Page 34: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/34.jpg)
https://www.reddit.com/r/gaming/comments/84wjsk/inverted_mouse/
![Page 35: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/35.jpg)
![Page 36: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/36.jpg)
![Page 37: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/37.jpg)
![Page 38: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/38.jpg)
2019 2020
…
![Page 39: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/39.jpg)
Hardware
SDK & API
CTRL-kit developer platform.
@CTRLlabsco
ctrl-labs.com
![Page 41: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/41.jpg)
[Extra slides]
![Page 42: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/42.jpg)
Neurocontrol: EMG → motor neuron activity
![Page 43: Rethinking HCI with Neural Interfaces · 2020-06-14 · Rethinking HCI with Neural Interfaces @CTRLlabsco Adam Berenzweig, Director of R&D, CTRL-labs @madadam | adam@ctrl-labs.com](https://reader034.vdocuments.site/reader034/viewer/2022042804/5f5053e90bdb9c273b6232ff/html5/thumbnails/43.jpg)
Tool / object control
Spoken language
The richest signals are at the forearm.