embodied cognition - booch hicss50
TRANSCRIPT
![Page 1: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/1.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 1
Embodied Cognition
Grady Booch
![Page 2: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/2.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 2
Imagine unleashing Watson in the physical world. Give it eyes, ears, and touch, then let it act in that world with hands and feet and a face, not just as an action of force but also as an action of influence. This is embodied cognition: by placing the cognitive power of Watson in a robot, in an avatar, an object in your hand, or even in the walls of an operating room, conference room, or spacecraft, we take Watson's ability to understand and reason and draw it closer to the natural ways in which humans live and work. In so doing, we augment individual human senses and abilities, giving Watson the ability see a patient's complete medical condition, feel the flow of a supply chain, or orchestrate the tasks in a day in the life of an individual.
![Page 3: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/3.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 3
Embodied Cognition Defined
§ An embodied cognition must– Be in and of the world– Reason– Learn– Have identity
§ An embodied cognition is not– Simply an STT -> NLC -> action pipeline
![Page 4: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/4.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 4
Robotics
Raw curiosity
More computational
power
Advances in AI algorithms and architectures
Better/smaller sensors and
actuators
Better batteries
3D printing
Advances in neural implants
New military doctrine
Demographic changes
Economic need
Advances in neural implants Technology push
Demand pullRaw curiosity
![Page 5: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/5.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 5
An Agent May Be Embodied In A Robot…
![Page 6: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/6.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 6
Dependent Autonomous
![Page 7: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/7.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 7
Fixed place Mobile
![Page 8: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/8.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 8
Thing-facing Human-facing
![Page 9: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/9.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 9
Predictable Responsive Social
![Page 10: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/10.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 10
Architecturally, robots may be classified along one of several dimensions
![Page 11: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/11.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 11
Biologically-organized
Symbolically-organized
![Page 12: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/12.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 12
Thin Client Thick Client
![Page 13: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/13.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 13
Thin Client Thick Client
By way of context, Watson is largely a symbolically-organized thin client cognitive system
Biologically-organized
Symbolically-organized
![Page 14: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/14.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 14
An Avatar…
![Page 15: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/15.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 15
A Space…
![Page 16: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/16.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 16
Or An Object
![Page 17: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/17.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 17
EmbodiedCognition
17
UseCases
Concierge(Robot, Avatar,
Space)
Whereistheelevator?
Retail(Robot, Avatar,
Space)
Doyouwannabuildasnowman?
Elder Care(Robot, Avatar,
Space)
I’vefallenandIcan’tgetup!
Cobot(Robot)
Getmeascrewdriver.Manufacturing
(Robot)
Watchmedothis.
Transportation(Robot, Avatar,
Space)
Openthepodbaydoors,Watson.
Boardroom(Avatar, Space)
Helpmedecide. Companion(Avatar, Device)
Let’splayagame.
![Page 18: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/18.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 18
Self
18
Sensors&Percep,on
Actuators
Agents
Goals&Planning
Pla5ormManagement Infrastructure
MetaManagement
Models
VoluntaryBehavior&Skills
InvoluntaryBehavior&Reflexes
Others
World
Self
Architecture
![Page 19: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/19.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 19
PointsofTechnicalConfluence
19
Collaboration
Whosaidthat?
Sensory Fusion
Ifeelsad.
Theory of Mind
IneedthePhillipsheadscrewdriver.
Context
Givemethatone;no,Imeanthatone!
Embodied Conversation
It’sgoodtoseeyouagain,Sandia!
Learning
I’vejustpickedupafaultintheAE35unit.
Devices
InwhatroomwasAlyssaworkingyesterday?
Knowledge Representation
![Page 20: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/20.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 20
EmbodiedCognition
§ Team‒ Leader:GradyBooch;Co-Leader:JeffKephart‒ KeyGlobalTeamMembers:SamAdams,RachelBellamy,RayChancey,JonConnell,StefanHamer,Kohichi Kajitani,SkyMatthews,NatMills,RobertMoore,AntonRiabov,FrancescaRossi,MartinRufli,Ryuki Tachibana,HironobuTakagi,JohnVergo
§ Theme‒ DeliveraplatformthatmakesthecognitivepowerofWatsonmanifestintherealworldwithsensorsandactuators,servingasanembodiedpersonathatorchestratestheagencyofthatsystemtoaugmentandcollaboratewithhumansusingvisual,vocal,andtactileformsofcommunication
‒ Deploythisplatformintheformofrobots,avatars,spaces,andobjects‒ Focusonhigherlevelsofreasoningandlearning‒ Enablepluggableagents,goals,plans,skills,reflexes,classifiers,andmodelssoastofacilitateavibrantdeveloperandsolutionecosystem
§ ShortTermGoal(oneyear)‒ GotomarketwiththeWatsonGroupandWatsonIoT,deliveringtheSelfkernel,services,andexamplar solutions.
§ LongTermGoal(threeyears)‒ Becomethedefactoindustrystandardforhigherorderembodiedcognitivesolutions
§ Gaps&Issues‒ There’smoreworktobedonethanIBMhasskillsorresourcestodoalone‒ Maniacalfocusisessential:therearemanymovingpartsthatcutacrossorganizationsboundaries
20
ExecutiveSummary
Sensors&Percep,on
Actuators
Agents
Goals&Planning
Pla5ormManagement Infrastructure
MetaManagement
Models
VoluntaryBehavior&Skills
InvoluntaryBehavior&Reflexes
Others
World
Self
Self
![Page 21: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/21.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 21
EmbodiedCognitionCore Technology Topic Project Name Contact/Lab ~ Team SizePlatform Self Booch/Almaden & Chancey/Austin 1 + 4
Goals & Planning
Planning Riabov/Yorktown 2
Embodied Action Moore/Almaden 1
Multi-modal Contextual Interaction Bellamy/Yorktown 5
Embodied Decision Support Systems Rossi/Yorktown 5
Models of the World, Others, and Self
Spatial Intelligence Rufli/Zurich 4
People Intelligence Hoory/Haifa 5
Spatial Awareness Kephart/Yorktown 4
Theory of Mind TBD
Learning Learning by Example Connell/Yorktown 2
Sensors & Perception/ActuatorsPhysical Interaction Tachibana/Tokyo 5
Edge Intelligence Hamer/Australia 2
Platform & Meta Management Tooling TBD
21
Solution Area Project ContactSpaces Watson for Buildings/Watson for Retail TBD/cross-laboratory
Robots, Avatars, & Spaces Watson Labs Chancey/Austin
Robots & Spaces Watson IoT Matthews/Durham
Robots Industrial Robots Kajitani/Tokyo
Spaces Elder Care Takagi/Tokyo & Adams/Raleigh
Spaces M&A Vergo/Yorktown
Spaces Boson Mills/Cambridge
Objects Cognitive Objects Bellamy/Yorktown
ProjectPortfolio
![Page 22: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/22.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 22
EmbodiedCognition
22
Collaborators
Solution Area Project ContactRobots Cooperative Robots Carnegie Mellon University
Spaces Multimodal Interaction Rensselaer Polytechnic Institute
Avatars Baby/X University of Auckland
Robots TBD University of California, San Diego
Robots TBD University of California, Berkeley
Robots TBD Massachusetts Institute of Technology
Robots TBD Stanford Research Institute
![Page 23: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/23.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 23
Background
§ In November 2015, Rob High, Ray Chancey, and I were brainstorming about a suitable grand challenge for the 2016 Embodied Cognition GTO.
§ We settled on a very grand one: build a system to support mission control, flight operations, and the scientific agenda of NASA’s Mission to Mars; the requirements of this mission lead to the concept of Watson in the walls (an embodied cognition for the Orion spacecraft and the Mars habitat) together with robots and other devices powered by Watson (namely, Robonaut and its peers).
§ Recognizing this was a 20+ year research effort, I set out to devise an architecture that would not only support this Mission to Mars but the journey of which would yield a middleware that would attend to a variety of use cases, from concierge to retail to elder care and many others.
§ Ray named this architecture “Self”.
![Page 24: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/24.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 24
Self Basic Principles
§ Augment human capabilities
§ Learn, don’t program
§ Theory of mind and social intelligence
§ Self understanding
§ Platform agnostic
§ Embodiment as a robot, an avatar, a device, or a space
§ Deployment as middleware with microservices in the cloud
![Page 25: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/25.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 25
Self Significant Design Decisions
§ Self is a hybrid architecture, encompassing explicit symbolic computation in the center together with neural networks at the edges.
§ Inspired by Minsky’s Society of Mind, behavior takes place in the context of multiple concurrent agents that communicate opportunistically via blackboards and deterministically via peer to peer connections.
§ Inspired by Brooks’s subsumption architecture, behavior takes place in a hierarchy of cognition, from involuntary reflexes to voluntary skills to goals and planning.
§ We maintain a clear separation of concerns among perception, actuating, models, and behavior.
§ As much as possible, behavior is either taught or is learned, not programmed.
§ As much as possible – driven by these separation of concerns, the needs of packaging, and performance – all components are made manifest as RESTful microservices.
§ As much as possible, plans, skills, and reflexes are extensible.
§ Self is intentionally full of strange loops: components of Self are also parts of the models of itself.
§ Self is intentionally fractal: an instance of Self may have models of others, which themselves are other instances of Self.
![Page 26: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/26.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 26
Self Markitecture
IBM Confidential 26
Sensors & Perception
Actuators
Agents
Goals & Planning
Platform Management Infrastructure
Meta Management
Models
Voluntary Behavior & Skills
Involuntary Behavior & Reflexes
Others
World
Self
![Page 27: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/27.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 27
Embodied cognition represents the next generation of Watson
![Page 28: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/28.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 28
Embodied cognition represents the extension of IT into the physical world, bringing cognition to the edge of the Internet of Things
![Page 29: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/29.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 29
At this touch point we encounter the intersection of big data and human presence, mediated by cognitive systems that reason
![Page 30: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/30.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 30
Text
Watson can be the back end to a variety of embodied cognitive systems.
![Page 31: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/31.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 31
Text + Audio
![Page 32: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/32.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 32
Text + Audio
This is the current state of an embodied Watson(the low hanging fruit, wherein platforms such as Pepper serve as animated speakers)
![Page 33: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/33.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 33
Text + Audio
There is much, much more that is possible(if we extent the embodiment to encompass greater perception and action in the world)
![Page 34: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/34.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 34
Text + Audio + Vision
![Page 35: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/35.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 35
Text + Audio + Vision + Emotion
![Page 36: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/36.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 36
Text + Audio + Vision + Emotion + Motion
![Page 37: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/37.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 37
Text + Audio + Vision + Emotion + Motion + Physical interaction
![Page 38: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/38.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 38
Text + Audio + Vision + Emotion + Motion + Physical interaction
Socially-intelligent cooperative robot
![Page 39: Embodied Cognition - Booch HICSS50](https://reader030.vdocuments.site/reader030/viewer/2022020108/587587561a28ab901c8b5023/html5/thumbnails/39.jpg)
IBM Confidential | Do Not Distribute © 2016 IBM Corporation 39