machine touch for dexterous robotic and prosthetic hands
TRANSCRIPT
![Page 1: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/1.jpg)
Machine Touch forDexterous Robotic and Prosthetic Hands
Gerald E. Loeb, M.D.
Professor of Biomedical EngineeringUniversity of Southern California
Chief Executive OfficerSynTouch Inc., Los Angeles, CA
![Page 2: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/2.jpg)
Tactile feedback is essential for dexterity.Humanlike tactile sensing is NOT about force sensors:
1. Exploratory movements2. Mechanical properties
3. Multimodal sensing
![Page 3: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/3.jpg)
Biological Transducers
Cutaneous Touch
Proprioceptive Touch
ActuatorForce
Golgi Tendon Organ
Position& Velocity
Muscle Spindle
Vibrations
Meissner Corpuscle
Pacinian CorpuscleForce &
Deformation
Merkel Disc
Ruffini Ending
Thermal
Tree Nerve Ending
Enabled by Exploratory Movements…
![Page 4: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/4.jpg)
Engineered Transducers
ActuatorForce
Golgi Tendon Organ
Position& Velocity
Muscle Spindle
Strain Gages
Position Encoders
Cutaneous Touch
Vibrations
Meissner Corpuscle
Pacinian CorpuscleForce &
Deformation
Merkel Disc
Ruffini Ending
Thermal
Tree Nerve Ending
“Taxel”Normal Force
Arrays
Enabled by Exploratory Movements…
Proprioceptive Touch
![Page 5: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/5.jpg)
The BioTac Design Approach
Cutaneous Touch
Vibrations
Meissner Corpuscle
Pacinian CorpuscleForce &
Deformation
Merkel Disc
Ruffini Ending
Thermal
Tree Nerve Ending
Biomimetic Mechanics
BioTac developers and SynTouch foundersJeremy Fishel, Nick Wettels, Gary Lin, Ray Peck, Matt Borzage
Contributing researchersRoland Johansson, Veronica Santos, Dipayon Roy, Blaine Matulevich, Vikram Pandit,
Danfei Xu, Zhe Su, Lorenzo Smith, Todd Erickson, Morelle Arian, Alex Blaine, Meghan Jimenez, Rahman Davoodi, Kelsey Muller, Alexandra Llic
![Page 6: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/6.jpg)
Deformation Sensing
Large Tangential Force No Tangential Force
• Forces deform skin and fluid
• Impedance changes are sensed by electrodes
• Raw data can be used with machine learning techniques to extract features:• Tri-Axial Force• Point of Contact• Radius of Curvature• …
ElectrodeArray
Wettels, Popovic, Santos, Johansson, Loeb. Advanced Robotics (2008)
Wettels, Smith, Santos, Loeb. IEEE Intl Conf Biomed Robotics and Biomechatronics (2008)
![Page 7: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/7.jpg)
Impedance: Probing about an Electrode
Localized, wide dynamic range: 0.03 – 50 N
![Page 8: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/8.jpg)
Point of Application Computation(post Lowess Filtering)
![Page 9: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/9.jpg)
Tri-axial force vectors extraction on BioTac
![Page 10: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/10.jpg)
Force vector extraction and control
0 5 10 15 20 25-5
0
5X
Fo
rce (
N)
root mean squared error = 0.4210 N
0 5 10 15 20 25-20246
Y F
orc
e (
N) root mean squared error = 0.4823 N
0 5 10 15 20 25-226
10
Z F
orc
e (
N)
time (s)
root mean squared error = 0.4461 N
Actual Force
Predicted Force
![Page 11: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/11.jpg)
The skin enveloping lateral impedance electrodes provides compliance and radius-of-curvature discrimination.
![Page 12: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/12.jpg)
Apical Tuft as a Vernier Tilt Sensor
7 9 8
10
30°
Spherical region
Cylindrical region
Flat Surface
Roll
Pitch Z X
Y
12 2
E8 E9
(c)
(a)
(b)
Su, Z., Schaal, S. and Loeb, G.E. Surface tilt perception with a
biomimetic tactile sensor BioRob 2016, Singapore
![Page 13: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/13.jpg)
Exploring Complex Shapes:Deconvolving Contour and Friction
E9 E10
![Page 14: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/14.jpg)
BioTac3 nm
Humans200 nm
Pressure Transducer
FluidPath
Similar size, shape and structure as the
human fingertip
Similar mechanical properties and dynamic response as the human finger
Flexible skinover liquid “pulp”
Textured Object
Vibration Sensitivity
![Page 15: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/15.jpg)
Jeremy Fishel (SynTouch) 15
Fingerprints Greatly EnhanceSliding Vibration Amplitude
Hydro-AcousticPressure Sensor(Hydrophone)
IncompressibleLiquid
Elastomeric Skinwith Fingerprints
Thermistor
•Can detect micro-vibrations for slip •and object texture discrimination
![Page 16: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/16.jpg)
Fingerprints Enhance Vibration Spectra
![Page 17: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/17.jpg)
When is Tactile Sensing Necessary?
• Dexterous Manipulation = “Perception for Action”– Contact timing– Grip adjustment– Slip detection
• Object Characterization = “Action for Perception”– Identify without vision– Anticipate handling properties
• Utility of related objects = “Affordances”
![Page 18: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/18.jpg)
Incipient Slip and Grip Adjustment
Jeremy Fishel, SynTouch
![Page 19: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/19.jpg)
Incipient Slip Detection During Tremor-Grip
![Page 20: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/20.jpg)
Incipient Slip Detection During Tremor-Grip
![Page 21: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/21.jpg)
Perception for Action: Fragile Grasp
OttoBock Health Care
Prosthetic hands typically operate with the motors stalling on objects with high forces (up to 100N).
It is very difficult to grasp fragile objects without intense visual feedback to control finger position.
![Page 22: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/22.jpg)
Blaine Matulevich, Vikram Pandit, Jeremy Fishel
Biomimetic Inhibitory Reflex for
Fragile Grasp
0 1 2 3 4 5 60
2
4
6
8
10
Preamplified Net EMG Signal (V)
Mo
tor
Com
ma
nd
(V
)
Contact
No Contact
![Page 23: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/23.jpg)
Performanceon cracker
task Speed
Accuracy
![Page 24: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/24.jpg)
Ejection
Displacement
Translate wrist or adjust finger
to avoid:
Detect contact:
Part Misalignment Mitigation
![Page 25: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/25.jpg)
Actions for Perception
Identifying Objects by Touch
Source: S. Najarian, et al. (2009) “Artificial Tactile Sensing in Biomedical Engineering”.
:
25
![Page 26: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/26.jpg)
Test: Artificial Texture Discrimination
Texture Discrimination
How?
Exploratory MovementsWhich Ones?
Fishel, J.A. and Loeb, G.E. Bayesian exploration for intelligent identification of textures. Frontiers in Neurorobotics, 2012
![Page 27: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/27.jpg)
Artificial Texture Discrimination Requirements: Biomimicry
• Tactile Vibration Sensitivity:
near human performance
• Texture Exploratory Movements:
inspired from human behavior
• Relevant Texture Properties:
from language humans use
• Intelligent Exploratory Strategies
inspired by theories of biological behavior
✓
![Page 28: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/28.jpg)
Forc
e
Velocity
Not Enough Power(no signals)
Too Much Power(wears skins)
PowerµForce´velocity
Best for TractionDiscrimination
1)
Best for FinenessDiscrimination
3)
Best for RoughnessDiscrimination
2)
Pilot Study – Which Movements are Best?
![Page 29: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/29.jpg)
Rough:
Smooth:
Texture Properties (from Language)
• Traction: sticky / slippery
Thrust (from motor)
Inspired from psychophysical literature on texture discrimination
FFTs:
Coarse
Fine
Vibration:
• Roughness: rough / smooth
Vibration Power
• Fineness: coarse / fine
Center Frequency
![Page 30: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/30.jpg)
Bayesian Inference
P A |B( ) =P B | A( )P A( )
P B( )Bayes’ Theorem
How do people decide which movements to make when exploring textures?
Used to update probabilities of textures after a movement and observation is made…
![Page 31: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/31.jpg)
2)
Heavy
Pick Up: Is it Heavy?
Theory: Bayesian ExplorationUnknown Object: 1)
Rectangular, Large
Grasp: Figure out shape/size
It’s a Brick!
.
Brick
Block of Wood
Block of Foam
![Page 32: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/32.jpg)
Bayesian Exploration
32
![Page 33: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/33.jpg)
117 Textures!
Papers
Glasses
Art Supplies Woods PlasticsMetal Finishes
Rubbers Leathers Cloths Weaves
Furs
Foams
Vinyls
5 Trials EachMovement
![Page 34: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/34.jpg)
5 Trials EachMovement( )
![Page 35: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/35.jpg)
1st
Movement
PossibleFuture
Movements
( )
Better Movement for Discrimination
Confusion
![Page 36: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/36.jpg)
Silicone RubberCorrect!
Textured VinylCorrect!
Cotton
Suede
Velcro BalsaIncorrect!
Smooth CardstockCorrect!
BalsaVelcro
Sometimes difficulty distinguishing two materials that humans cannot distinguish either
Sometimes rapid identification
Sometimes tests a few hypotheses before a correct identification
![Page 37: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/37.jpg)
Silicone RubberCorrect!
Textured VinylCorrect!
Cotton
Suede
Velcro BalsaIncorrect!
Smooth CardstockCorrect!
BalsaVelcro
Sometimes Difficult to Distinguish between
two: Smooth Cardstock and Balsa Wood
Sometimes, Quick Identification
Sometimes: Tests a few hypotheses, before a correct
identification
1)
2)
3)
4)
16 Validation TexturesAverage: 95.4% Correct ClassificationMedian Movements to Converge: 5
![Page 38: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/38.jpg)
Applications
If you can feel the difference, we can quantify it!
Quantifying TouchMeasures 15 properties related to:
Texture, Friction, Compliance, Thermal Properties and Adhesion
![Page 39: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/39.jpg)
39
Representing the World in the Brain
![Page 40: Machine Touch for Dexterous Robotic and Prosthetic Hands](https://reader033.vdocuments.site/reader033/viewer/2022043020/626bc87493de980c8f7cecad/html5/thumbnails/40.jpg)
Bayesian Action&Perception:
Representing the World in the Brain
G.E. Loeb and J.A. Fishel, 2014
Frontiers in Neuroscience, doi: 10.3389/fnins.2014.00341
Recently expanded to 500 materials
explored by 5 movements
to create 15 perceptual dimensions,
resulting in
MORE ACCURATE and FASTER
performance.
Avoids “curse of dimensionality”