1

Simulation Semantics: A Framework to explore the links between Language, Cognition, and Action

Jerome A. FeldmanICSI/UC Berkeley

ONR Cognitive Science & Human Robot Interaction6.1 and 6.2 Program Review

August 2015

2

OBJECTIVES

Perform and integrate basic research and converging evidence from cognitive linguistics, computation, and neural information processing to build cognitive models of language acquisition and use.

Provide an operational computational framework of action/simulation semantics to investigate the interaction between language, action, and cognition.

Action/Simulation semantics offers the possibility to build systems for computing with natural language that come close to human performance levels. This is necessary for joint action in complex naval scenarios with a mix of human and artificial agents.

ECG - NLU Beyond the 1980s

 

1. Much more computation

2. NLP technology

3. Construction Grammar: form-meaning pairs

Conceptual compositionality + Idioms, etc.

4. Cognitive Linguistics: Conceptual primitives, Metaphor, etc.

ECG = Embodied Construction Grammar; 6 uses of formalism

5. Constrained Best Fit : Analysis, Simulation, Learning

Analysis uses Bayesian (form, meaning and context) best fit

6. Under-specification: Meaning involves context, goals, etc.

SemSpec = Semantic/Simulation Specification

7. Simulation Semantics: Meaning as action/simulation

8. CPRM= Coordinated Probabilistic Relational Models; Petri Nets ++

Action formalism works as a generative model

9. Domain Semantics; Need rich semantics on the Action side

10. General NLU front end: Modest effort to link to a new Action side

  

Slide 3

Active representations

• Many inferences about actions derive from what we know about executing them

• X-net representation based on stochastic Petri nets captures dynamic, parameterized nature of actions

• Used for acting, recognition, planning, and language

Walking: bound to a specific walker with a direction or goal

consumes resources (e.g., energy)may have termination condition

(e.g., walker at goal) ongoing, iterative actionwalker=Harry

goal=home

energy

walker at goal

How do we specify an event? Formalized event schema

• Key elements– preconditions, resources, effects, sub-events– evoked by frames (alternatively: predicates, words)

• Contrast with Event Recognition/Extraction, other NLP work

5

ISA

hasFrame hasParameter

construedAs

composedBy

EVENT

COMPOSITE EVENT

FRAMEActorThemeInstrumentPatient

CONSTRUALPhase (enable, start, finish, ongoing, cancel)Manner (scales, rate, path) Zoom (expand, collapse)

RELATION(E1,E2)SubeventEnable/DisableSuspend/ResumeAbort/Terminate Cancel/StopMutually ExclusiveCoordinate/Synch

EventRelation

CONSTRUCTSequenceConcurrent/Conc. SyncChoose/AlternativeIterate/RepeatUntil(while)If-then-Else/Conditional

PARAMETERPreconditionsEffectsResources - In, OutInputsOutputsDurationGrounding

Time, Location

AccomplishmentsAction Language Understanding System

• Demonstrate utility through a series of scalable prototypes – that show the ability of the system to handle increasingly complex

language– in a general way across multiple tasks and environments – to support communication in communities comprised of both

human and artificial agents

• Current Goal: Implement a prototype system that can follow instructions and synthesize actions and procedures expressed in natural language. – This requires the system to analyze natural language and

translate this language in context into a coordinated network of actions and complex commands.

Slide 6

Integrated Pilot System for Action Synthesis

7

DiscourseAnalyzer ECG Grammar

SemSpecSpecializer

ApplicationProblem Solver

API ~ Morse etc.

Actions

CompiledN-Tuples

World

Situation

Ontology

Robot1, move North!

Videos

8

9

ISSUES

• The 2013 PI, Professor Srinivas Narayanan, joined Google in Zurich as of February 1, 2014.

• Professor Jerome Feldman, who was Co-PI, has become the PI. Prof. Narayanan continues to advise on the project and there has been no significant disruption.

• In early 2015, the companion IARPA program was abruptly cancelled, causing significant curtailment of the metaphor work, but not the core ONR mission.

10

CONCLUSIONS & NAVAL RELEVANCE

• The main conclusion is that it is now feasible to build practical LCAS (Language Communication with Autonomous Systems) realizations. ICSI has received a provisionary patent on this technology.

• Our NLU work is deeply interdisciplinary and is having effects on several fields.

• One explicit Naval requirement is an improved ability to communicate with robots and other complex equipment using natural language. We have made significant progress in this area and propose to directly address Navy tasks.

11

CURRENT AND FUTURE WORK

We will continue to develop both the theory of action/simulation semantics and its application to Naval mission-relevant LCAS. Our actionability theory is being discussed in a wide range of forums and is potentially disruptive.

We will expand our general working LCAS interface to complex systems and the specific applications to multi-agent robot control and spoken language integration.

One main effort moving forward is to expand the full system to industrial scale. Analysis and staffing for this challenge has begun.

12

RECENT PUBLICATIONS, PATENTS, PRESENTATIONS, & AWARDS

• The team had five papers accepted at the International Cognitive Linguistics Conference held in June 2013.

The 2013 PI, Professor Narayanan and collaborators won the Semantic Web Science Association’s ten year highest impact paper award. Prof. Narayanan, was the keynote speaker at the First Workshop on Metaphor in NLP at NAACL in June 2103

PI Prof. Jerome Feldman, Invited seminar talk, Redwood Center for Theoretical Neuroscience, UC Berkeley.Jerome Feldman gave invited talks on our Actionability theory early in 2014 at: The Conference on Theory of Mind, The Soft Computing Round Table, and at the UC Irvine Cognitive Science seminar.

CoPI Jerome Feldman and PI Srini Narayanan organized a workshop on Actionability held at ICSI Berkeley in 2013.

Katia Shutova organized a workshop on metaphor in NLP at NAACL in 2013 and again in 2014.

13

RECENT PUBLICATIONS,

Srini Narayanan (2013). “A neural computational model of metaphor learning using spiking networks and STDP”, Neural Computation (accepted)

Dodge, Ellen. “A Compositional Constructional Analysis of ‘Hitting’ Verb Argument Realization Patterns and Their Meanings”. Talk presented at the International

Cognitive Linguistics Conference (ICLC) 12, Edmonton, AB. June 27, 2013. Srini Narayanan (2013). A Cognitive Model of Scales. Proceedings of the

International Conference Cognitive Linguistics Conference, June 2013.

Dodge, Ellen K.; Petruck, Miriam R. L. “Representing Caused Motion in Embodied Construction Grammar”. Association for Computational Linguistics, Baltimore, MD. June 26, 2014

Sweetser, E. (with Dancygier, B.): Figurative Language Cambridge U. Press 2014

Feldman, Jerome “Affordances, Actionability, and Simulation” Affordances in Vision for Cognitive Robotics, Berkeley, July 2014

14

TRANSITION PLAN

Our applied efforts in semantics –based Natural Language Understanding (NLU) have reached a stage where transition to military and/or commercial use is feasible.

It is possible that the current LCAS pilot system could be used for focused development of Naval systems and we would be happy to assist in such efforts. We are actively working with ONR staff to identify Navy and performer partners.

For wide spread military and/or commercial use, we need to scale up and harden the current system and also expand to spoken language and multi-agent tasks. We have begun this effort in cooperation with other research and industrial groups .

Simulation SemanticsJerome Feldman ICSI/UC Berkeley

Award Period 01/2011 – 01/2016Project Objectives:

Technical Approach: Accomplishments/Impact/Transitions: ECG Workbench and Grammar Analyzer Computational Model of Simulation

Semantics Extension to metaphor, other languages Integrated System - Language control of

mobile robots, etc. - LCAS LCAS Provisional Patent granted in 2015 Transition to Spoken Language Many publications, invited talks, workshops Collaborations with government, industry,

academics

Develop and test cognitive models of language use.Operational framework of action/simulation semantics LCAS- Language Communication with Autonomous Systems

Action/Simulation uses probabilistic best-fit a) Coordinated Probabilistic Relational Models (CPRM) b) Embodied Construction Grammar Message-based LCAS applicationsDemonstrate utility with a series of scalable prototypes Build community based on action/simulation semantics

Situation model

Problem Solver

n-tuplesSpecializer

Task API (MORSE simulator)

SemSpec 

Analyzer(ECG)

TEXT

USERObservations

Ontology

Simulated

World

Extras

16

Computing with Natural Language

• An integrated system combining –Deep semantic analysis of language in context with–A scalable simulation model

• Best-fit Language Analyzer– Embodied Construction Grammar (ECG)

• Construction Parser – John Bryant PhD Thesis 2008– Eva Mok PhD Thesis 2009– Ellen Dodge PhD Thesis 2010

• Scalable Domain Representation– Event Models

– Steve Sinha PhD Thesis 2008– Joe Makin PhD Thesis 2008

– Coordinated Probabilistic Relational Models – Leon Barrett PhD Thesis 2010– Steve Doubleday Thesis 2015

Embodied Construction Grammar: ECG

• ECG serves:1. as a technical tool for linguistic analysis

2. to specify shared grammar, conceptual conventions of a linguistic community

3. as a computer specification for implementing linguistic theories

4. as a representation for models and theories of language acquisition

5. as a front-end system for applied language-understanding tasks

6. as a high-level functional description for biological and behavioral experiments

ECG Workbench

ECG Workbench:● Based on Eclipse● Takes advantage of and fully integrates with

Eclipse RCP (Rich Client Platform)● Makes it easy to enter, edit and check

consistency of ECG grammars● Can analyze text licensed by the grammar,

producing a SemSpec (Semantic Specification)

● Download: http://www.icsi.berkeley.edu/~lucag

Additional ExamplesSimple Commands:Robot1, move North!Robot1, move to the big red box!Robot1, move behind a red box!Robot1, push the blue box North!

Serial Processes:Robot1, move to the blue box then return!Robot1, move to the big red box then move the small red box North!Robot1, move to the green box then push another one East!

Conditionals + Referent Resolution:Robot1, if the green box is near the small red box, move to it!Robot1, if the blue box is near the small red box, move to the green box then push it!

Questions:which boxes are near the green box?which boxes are red?is the box near the blue box green?where is the green box?

Definitions:define tour QL1 and QL2 and QL3 as move to QL1 then move to QL2 then move to QL3.Robot1, tour Box1 and Box2 and Box3!

Same Examples in SpanishSimple Commands:Robot1, muévete al norte!Robot1, muévete detrás de una caja roja!Robot1, empuje la caja azul al norte!

Serial Processes:Robot1, muévete a la caja azul y regresa!Robot1, muévete a la caja roja y grande y muévete a la caja roja y pequeña!

Conditionals + Referent Resolution:Robot1, si la caja verde está cerca de la caja roja y pequeña, la empuje al norte!Robot1, si la caja pequeña es roja, muévete a la caja verde y la empuje!

Questions:es la caja cerca de la caja azul verde?está la caja verde cerca de la caja roja y pequeña?cuáles cajas son rojas? cuáles cajas están cerca de la caja roja y pequeña?dónde está la caja verde?

Robot1, move North then return!

Robot1, move North then return!

Analyzer

Robot1, move North then return!

Integrated Pilot System for Action Synthesis

25

DiscourseAnalyzer ECG Grammar

SemSpecSpecializer

ApplicationProblem Solver

API ~ Morse etc.

Actions

CompiledN-Tuples

World

Situation

Ontology

Robot1, move North!

Parse serial processes

Initialize executable template

Assign values in SemSpec to template

Specializer

Parameters of message

First part of N-tuple received by Solver; contains information for first part of serial process (“move North”)

Second part of N-tuple received by Solver; contains information for second part of serial process (“return”)

Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading='north', goal=None)

N-Tuple

[Struct(protagonist='robot1_instance', heading='north', goal=None, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing'),

Struct(protagonist='robot1_instance', heading=None, goal={'location': 'home'}, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing')]

Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading=None, goal={'location': 'home'})

Solver

Unpacks N-tuple passed from Specializer.

Determines heading of movement vector (“move North”).

Returns to “home” position, stored in prior state variable.

Morse (Robot Simulator)

Moves robot

Gets the new state of the world

Updates internal model of the world

Gets the updated state of the world

Moves the robot in simulation (and waits for it to arrive)