ken forbus presented “software social organisms: implications for measuring ai progress.”
TRANSCRIPT
Software Social Organisms:Implications for Measuring AI Progress
Kenneth D. ForbusNorthwestern University
Big Picture
• Achieving human-level AI = learning how to create sufficiently smart software social organisms
• No single test will be sufficient to measure progress
• Evaluation should be ability to participate in our culture– Apprentices– Provides multiple dimensions within which progress
can be measured• Modalities, range of domains, levels of knowledge…
Today’s AI systems
can be fast and effective.
But they requires careful design for
narrow circumstances,maintenance by highly trained
personnel
What if AI systems
were as robust, trainable, andtaskable as mammals?
Companion Cognitive Architecture• Approach: Understand how to build Software
social organisms that– Work with people using natural modalities– Learn and adapt over extended periods of time– Maintain themselves– Capable of relating to people as apprentices, not tools
• Companion architecture incorporates other hypotheses as well– Centrality of analogy, qualitative representations
• But the idea of software social organisms is more general, and is a path to human-level AI
Why Organisms?• Autonomy
– We do not have direct access to the internal representations of children or our collaborators
– We communicate via language, sketching, …
• Adaptation– Incremental learning in everyday life– Building up knowledge over weeks, months, years
• New interest in interactive task learningKirk & Laird,
2013Hinrichs & Forbus
2013
Why Social?
• Being social is a driver of intelligence– e.g. Tomasello’s research
• Cultural knowledge is central in human life– We can all think about things we never experienced
• Plate tectonics, the Revolutionary War, …
– Stories are an important component of human learning
• e.g. Winston’s research
• Smart organisms want to cooperate and collaborate as well as compete– AI has started to model this, e.g. Grosz’s research
Human-level AI
• Will end up being social organisms• Sufficiently smart software social organisms will
be human-level AIs – Sufficiently smart = capable of learning to perform a
sufficiently broad range of tasks that people do, with the same or better levels of performance
Apprenticeship
• Natural trajectory for bringing people into a role– Also provides one trajectory for evaluations
• Start as a student– Lots of “book learning”, interaction– Tests to gauge learning– Legitimate peripheral participation
• Expand range of responsibilities– Joint work– Negotiated roles
• Autonomous operation with a community– Performs well on its own, or with others at their level– Can ask for help, and help others
Substrate Capacities Needed for Human-like Social Organism
• Autonomy– Its own needs, drives, and capabilities for acting and learning
• Operates in environments that support shared focus– Each participant has some information about what others can sense– Participants can make focus of attention known to each other easily
• Language sufficient to express beliefs, hypotheticals• Ability to build models of the intent of others
– Implies learning the types of goals they can have, and how available actions feed into those goals
– Models of needs, drives as wellsprings of particular goals
• Strong interest in interactions with conspecifics– Helping and teaching
Implications for Evaluation• No single test will work
– Won’t measure adaptability and breadth– Can always be hacked
• Need to evaluate via ability to participate in our culture– Do real work– Interact with people appropriately– Learn and adapt about its work, others, and itself– Without AI experts constantly fiddling with its internals
Scalable Evaluation Dimensions• Modalities
– Text, sketching, speech, vision, mobility, …
• Starting point– Nothing versus some prior knowledge versus lots
• Level of domain knowledge– Elementary school Professional life
• Range of tasks responsible for– Single task versus multiple– Interactive task learning
• Learning ability– About its job, its people, and itself– Amount and quantity of data required
Summary
• Human level AI = sufficiently smart software social organisms
• Apprenticeship trajectory provides framework for many kinds of evaluation– Initially a student, learning from instructors through
carefully designed exercises– Working as an assistant to a mentor, with increasing
responsibility as it learns– Working autonomously, reporting back periodically for
debriefing, review, and retasking
Check out QRGLab channel on YouTube