1

Context Machine

Perry McDowell Michael Capps

Mike Zyda Andrzej Kapolka

2

Outline

The Context Machine

Literary Inspirations

Payoff and Impact

Goals and Objectives

Technical Approach

Accomplishments

Demo

Future Work

3

Literary Inspirations

Diamond Age, Neal Stephenson

Futuristic novel of a young girl who discovers an interactive book which plays scenes demonstrating proper behavior of a young lady

She is able to drastically change her life by adhering to lessons learned in the book

4

Literary Inspirations

Starship Troopers, Robert A. Heinlein Futuristic novel of a young man who joins

the Mobile Infantry and fights in a intergalatic war

The soldiers wear an automated suit which greatly enhances their lethality and survivability

Goal is to give soldiers the same improvement mentally that the suits gave physically

5

The Context Machine

We are building a Context Machine that determines the “context” of a situation for the user and provides advice/guidance.

We anticipate that this augmentation will become essential to future soldiers.

6

How the Context Machine Helps Users

The context machine is envisioned to improve flow of information to users in several ways: Informing users of things they can not observe Informing and filtering for users things they don’t have the

time to observe Explaining situations and devices users can’t understand Providing an perceptive interface for the user

7

Where Does the Context Machine Fit in AugCog?

LocationLocation

ActionsActions

PeoplePeople

ObjectsObjects

Outer World Outer World AbstractionAbstraction

Determine Determine Context Context and COAand COA

Output to Output to UserUser

User’s User’s CommandsCommands

Goal 1Goal 1

Goal 4Goal 4 Goal nGoal n

Goal 2Goal 2

Goal 3Goal 3

PreferencesPreferences

Visual Visual DisplayDisplay

Aural Aural DisplayDisplay

8

Payoff/Impact

A mechanism for accepting symbolic information from a sensory stream and inserting it into a robust dynamic architecture for determining and representing context

Provides expert advice in limited domains

Capable of learning new and novel situations

Monitors user’s goals and assists where appropriate

Adaptable to individual users

9

How to Build the Middle Without the Ends

We used a game engine to create a world where we could test the Context Machine’s ability to determine context The engine simulates the system that extracts

data from the real world and puts it into the symbolic information

Game Engine Game Engine WorldWorld

Game Engine Game Engine DisplayDisplay

Context Context MachineMachine

10

And Now, a Word From My Sponsor…

MOVES Institute has a new War Games Lab Uses game technology to rapidly build high fidelity simulations

20 artists and programmers from the game industry Plus plenty of cheap graduate student labor

Work closely with numerous game companies Unlimited license for Epic Games’ UNREAL engine Currently building products or demonstrations for US Army,

SOCOM, DTRA, DOE, ONR, NRO, and others Vague website at http://www.wargameslab.com/

12

Technical Approach

We are pursuing a concurrent and intertwined two-tracked approach: Track 1 involves developing hypothesis of

context Literature search of current research into context Develop a taxonomy of context elements Define “necessary and sufficient” for creating context-

aware applications Track 2 involves creating context-aware

applications to test our hypotheses These applications begin simply in specific domains

and increase in complexity and generality

13

What We Have Accomplished

Wrote a technical report which consists of an in-depth literature review of the research previously done on context-aware systems. Summarizes the major efforts in context-

aware computing to date Presents both the definitions of context and

the attributes used to determine context Determines a taxonomy to be used in future

investigations into context

14

What We Have Accomplished

Created a simulation demonstrating assistance furnished by a Context Machine Acts as a base for our later investigations Demonstrated rudimentary context sensing

capability Demonstrated ways the Context Machine can

increase cognitive throughput Serves as a test platform to test techniques to

improve the user’s cognition

15

Specific Accomplishments

Created a pathing system which can determine the best path for a user to follow

Created an imperfect mental model to reflect the facts known by the Context Machine

Use that information to determine context and advise user of appropriate course of action

16

Demo

17

Demo

Goal of the simulation is to transit the treeline on the left of the camp, go to the roof of the building (“A”) and observe a convoy passing down the road to the right.

A

Ingress Convoy Location

18

Directional Guidance

Arrows provide directional guidanceIntend to add other information to arrows

19

Improving Recognition

Augmented reality cursor points out important information that might otherwise not be noticed

20

Displays Information Regarding Mental Model

Bar indicates degree of certainty as to the identification of an object/person in the Context Machine’s mental model

21

Remembers Information

After identifying objects, the Context Machine remembers them so user can concentrate on other things

22

Finds Paths Based Upon Current Info

Pathing can be based on a wide variety of criteria

24

Contact Information

Perry McDowellThe MOVES InstituteNaval Postgraduate School833 Dyer Road, Code CS/mpMonterey, CA 93943-5118(831) 656-4075mcdowell@nps.navy.milhttp://www.movesinstitute.org/~augcog/