MACHINE-MEDIATEDCOMMUNICATION

3D environments,(sensors, projectors: sight, sound, touch, other sensory modes)

information capture, display

personalagents multimodal

fusion

storage

media conversion, multilingual translation

information representation

computationmanagement

architectures, protocols

agentsdB’s

libraries

storageWeb pages

grid-based computation

Other Sites

HumanInterface

GlobalNetwork

CommunicationMediator

security

security

Multimodal Human-Machine Interface

Force Feedback Glove

Gaze Tracker

Microphone

Array

Multimodal Human-Machine Interface

Real-world trial with NJ National Guard

Portable Interactive Command Console (PICC)

Steerable microphone arrayelement

Stereo face tracking cameras

Gaze tracker

Source locator microphone

Loudspeakers

Light pen

Flatpaneldisplay

Sensors

HQ/VEHICLE

Robotic Vehicles Emergency Responder in the Field

Internet

User interface for interaction and collaboration with robots and humans

NSF Equipment Grant EIA#98-18313 Center for Advanced Information Processing, Rutgers University, Piscataway, NJ 08854.

PI: J.L. Flanagan, co-PIs: J. Wilder, I. Marsic, M. Krane

“Move this to there.”

FRAMEWORK FOR MULTIMODAL LANGUAGE

SENSORY SIGNAL

IDENTIFY ACTION

ESTIMATE MEANING

GENERATE RESPONSE

FORM SENTENCE

CREATE “WORDS”

CLASSIFY SEGMENTS

SEGMENT CONTINUA

EXTRACT FEATURES

Transduce intelligence

Options library

Semantic rules; parse

sentence

Multimodal language

generation

Syntax rules; word n-grams

Lexicon; grammar;

symbol n-grams

Symbol library

Source – receiver

constraints

Transform domain

FUSION OF MODALITIES