1 command and natural language command and natural languages hieroglyphs (a 5000 year old...

1Command and Natural Language

Command and Natural Languages

Hieroglyphs (a 5000 year old technology) versus icons (the technology of the day)

Special languages

• Mathematics

• Music

• Chemistry

• Computers (interactive, user/programming communication)

2

Command and Natural Language

Command and Natural Languages

Goals of a language

• Precision

• Compactness

• Ease of writing and reading

• Completeness

• Speed in learning

• Simplicity in reducing errors

• Ease of retention over time

3


• Command and Natural Languages

• Tools that accelerated language development

• Printing Press versus Computer

• Print Press provided widespread dissemination of written work• Computer provides widespread dissemination of words via networks• Computers provide tools to manipulate language• Language is a tool to manipulate computers

4


• Command and Natural Languages

• Computer Languages

• Modest influences on spoken natural languages• Huge impact on the development of formal written languages• Early computers were designed to perform mathematical computations• Future is to operate on the real world

• Directing robots• Controlling manufacturing• Guiding spacecraft

5


Language Program Size # of Programmers Use

FORTRAN small mostly single engineering

COBOL mostly large mostly multiple business

ALGOL mostly large mostly multiple business

PL/1 small to large mostly multiple business

Pascal small to large single/multiple academic

BASIC small to large single/multiple business

LISP small few artificial intelligence

APL small few statistics

PROLOG Mostly small few artificial intelligence

ADA mostly large mostly multiple government

C++ mostly large multiple business

JAVA Mostly large multiple business, cross platform

C# small to large single/multiple business, cross platform

Perl Small Small web scripting

Languages that tell computers what to do

6


• Command and Natural Languages• Scripting Languages

• HyperCard• Revolution

• Database Query Languages• Structured Query Languages (SQL)• The goal of the user is to create a result, not a

program (3 to 20 lines)• Boolean operations (AND, OR, NOT)

• Command Languages• Ephemeral• Produce an immediate result on an object• Example command line input to a browser:

http://www.google.com/

• New command languages have declined as a result of direct manipulation

• Typical format: Verb followed by a noun with qualifiers

– PRINT MYFILE 3 COPIES– UPD CAD TN=555-555-5555 BILLTN=302-555-1234

7


• A UI Design Approach to Command Language Interfaces

Study the user’s task domain to determine the functionality required of the application

Derive a list of task actions and objects

Represent the items as the low-level interface syntax• Update Customer Record• Copy File_A File_B

Determine the frequency of specific tasks and facilitate the execution of those tasks

Determine whether destructive actions are reversible (e.g., delete)

Determine error messages

Transition diagram of how specific commands take the user to specific states

8


• Transition Diagrams

Logon

iUPD

iDEL

iCPY

iCAD

iCPRa

iCADa

iCAD

iNUM

iCAR

oNewNum

iCADb

oNoCAD

iCAD oNewCar

oNewCAD

iCPRb oNewCPR

oNoCPRiCPR

9


• UPD CAD=555-555-5555 NUM=444-444-4444 NewNum=333-333-3333

Logon

iUPD

iDEL

iCPY

iCAD

iCPRa

iCADa

iCAD

iNUM

iCAR

oNewNum

iCADb

oNoCAD

iCAD oNewCar

oNewCAD

iCPRb oNewCPR

oNoCPRiCPR

10


• CPY CPR=555-555-5555 CPR=444-444-4444

Logon

iUPD

iDEL

iCPY

iCAD

iCPRa

iCADa

iCAD

iNUM

iCAR

oNewNum

iCADb

oNoCAD

iCAD oNewCar

oNewCAD

iCPRb oNewCPR

oNoCPRiCPR

11


• DEL CAD=555-555-5555

Logon

iUPD

iDEL

iCPY

iCAD

iCPRa

iCADa

iCAD

iNUM

iCAR

oNewNum

iCADb

oNoCAD

iCAD oNewCar

oNewCAD

iCPRb oNewCPR

oNoCPRiCPR

12


• Command Organization• Simplest – each command carries out a single task

• Commands plus arguments• COPY FILE_A,FILE_B• DELETE FILE_A• PRINT FILE_A,FILE_B,FILE_C

• Delimiters of blanks or commas

• Keywords• COPY FROM=FILE_A TO=FILE_B

• Options• PRINT/3 PRINTER_LOC_A FILE_A

• Command to check seat availability on a flight on August 21 from Washington’s National Airport to LaGuardia Airport departing about 3:00PM

• A0821DCALGA0300PM• Parsed: A 0821 DCA LGA 0300PM• Speed

13


• Command Organization• Tree Structure• 5x3x4 = 60 tasks based on 5 commands and 1 rule

Action Object Destination

CREATE File File

DISPLAY Process Local Printer

REMOVE Directory Screen

COPY Remote Printer

MOVE Remote Printer

14




CREATE File File



COPY Remote Printer

MOVE Remote Printer

15




CREATE File File



COPY Remote Printer

MOVE Remote Printer

16




CREATE File File



COPY Remote Printer

MOVE Remote Printer

17


• Natural Language Interaction (NLI) • Users communicating in a familiar language (e.g., English) to a computer, and

receiving response from the computer in the same language

• Potential application - Checkbook maintenance • Pay to University of Delaware on 4/18/06 $1525.00• Show me all checks made out to the Deer Park• Which checks did I write this week

Previously direct manipulation interfaces have had more success in the market

18


Natural Language Queries (NLQ) Natural Language front end for databases The user types an English-like sentence The result is a structured database search

19


Natural Language Question Answering (NLQA)

“Who was the first Prime Minister of the European Union?”

Don’t attempt an exact answer• Users make incorrect assumptions

– There is a president of the European Union– There is a European Commission

Better to return a list of results pages

20


Text-database searching Used by Lycos, Google, AltaVista Stemming (attempt to reduce a word to its stem or root form) Relevance ranking by word-frequency analysis Filtering of common words (e.g., “the”, “of”, “in”) Latent semantic indexing

• Uses a term-document matrix which describes the occurrences of terms in documents• It is a sparse matrix whose rows correspond to documents and whose columns

correspond to terms, typically stemmed words that appear in the documents. • A typical example of the weighting of the elements of the matrix:

21


Natural Language Text Generation (NLTG) The computer generates readable reports

• “White-blood-cell count is 12,000”• “This value exceeds the normal range of 3,000 to 8,000 by 50%”• “Further examination for systemic infection is recommended”

Natural Language, Adventure Games and Instructional Systems Example user commands

• TAKE ALL OF THE KEYS• PICK UP THE SWORD

Instructional software• Guide students through the learning process• Algebra and Geometry• Feedback is provided in conversational language

22


Applications in mobile devices – Voice Commands

Call Recognizes verb Accepts name or number as input Requests clarification by providing a list of names Confirms dialing Useful for dialing while driving

Send Text Message Recognizes verb Accepts name or number as input Opens the text composer

Applications in mobile devices – Voice Read Out

23


Natural Language is applicable to situations where the user cannot actually touch the computer or device Voice input Synthesized voice output

24


• Natural Language Interaction (NLI) vs. Direct Manipulation

• Users seek predictable responses

• Users get discouraged having to engage in clarification dialogs

• Visually oriented interaction exploits• The computer’s capacity for rapid display of information• The human’s ability for rapid visual recognition

25


• Natural Language Interaction (NLI)• Turing Test

• “Computer program whose conversation is indistinguishable from a human’s”

ELIZA - emulates a Rogerian psychotherapist.

ELIZA has almost no intelligence whatsoever, only tricks like string substitution and canned responses based on keywords. The illusion of intelligence works best, however, if you limit your conversation to talking about yourself and your life.

http://www.manifestation.com/neurotoys/eliza.php3

AskJeeves

• http://www.ask.com/

• Invites natural language questions

• Does keyword extraction

• Returns web-page result sets

26


• Natural Language Interaction (NLI) vs. Direct Manipulation

• Users seek predictable responses

• Users get discouraged having to engage in clarification dialogs

• Visually oriented interaction exploits• The computer’s capacity for rapid display of information• The human’s ability for rapid visual recognition

27


• NLI Future Vision – 2001: A Space Odyssey

Astronauts set off in a spaceship controlled by HAL 9000, a revolutionary computer system that is every bit mankind's equal, and perhaps his superior.

HAL endangers the crew's lives for the sake of the mission, and eventually, kills almost all of the astronauts in the film. The astronauts have to overcome the computer.

HAL behaves in the most human fashion of all of the characters.

• He reaches human intelligence levels,

• Develops human traits of paranoia, jealousy and other emotions. Humans act like machines

• Cooly performing their tasks in a mechanical fashion,

• Even under extreme duress.

HAL, develops an acute emotional crisis because he can not accept evidence of his own fallibility.

28


• NLI Future Vision – 2001: A Space Odyssey

Check out this video to see the natural communication between the astronauts and the computer.

2001 Space Odyssey Video• Ease of communication• Personal assessment of Dave’s art work• Determining what is a “personal” question• HALs superior statement of human error to computer error• Preparing the crew psychological report• HAL providing a hard copy of a report• Astronauts talking in private about whether to trust HAL• HAL reading lips

http://www.youtube.com/watch?v=Be8Gbqdox68

29

• AI – by Steven Spielberg

• Global warming lead humans to near to extinction

• Humans create robots known as mechas which can emulate thoughts and emotions.

• David is designed to resemble a human child and to virtually feel love for its human owners.

• Humans then become extinct

• Humanoid’s use David’s memories, to reconstruct the Swinton home

• The humanoids also create a clone Monica (his mother). She can only live for one day.

• David spends the happiest day of his life playing with Monica and Teddy.

Human Robot Interaction – AI by Spielberg

30

http://www.youtube.com/watch?v=xRhCz0ELrKs

Human Robot Interaction – AI by Spielberg

31

• More questions and answers• The role of facial expressions?• Context?• Personality?• The role of psychology in applying what is known about behavior to robots?• Education?• Living family trees?

http://www.youtube.com/watch?v=IhVu2hxm07E&feature=related

http://www.youtube.com/watch?v=pkpWCu1k0ZI

Human Robot Interaction

32

http://www.pbs.org/wgbh/nova/tech/social-robots.html


33

• Human-robot interaction (HRI) • The interdisciplinary study of interaction dynamics between humans and

robots• Robots name first used by Capek’s in the play R.U.R.

• http://www.youtube.com/watch?v=mEG7MhltHWk

• Robots roles in today’s society• factory automation• service applications • medical care • Entertainment

• Asimov in I, Robot (1950) used the term ”robotics” • Three laws of robotics:

• A robot may not injure a human being or, through inaction, allow a human being to come to harm.

• A robot must obey orders given it by human beings except where such orders would conflict with the First Law.

• A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

• http://www.youtube.com/watch?v=WJRfWyTpJRQ&list=PL2FA31FC4B426806D

From: Human-Robot Interaction - David Feil-Seifer & Maja J Mataric (2009)


34

• Philip K. Dick’s novel Do Androids Dream of Electric Sheep (1968)• Created replicants are humanoid robots that look and act like humans,• special tests are devised to determine if an individual is a human or a replicant

• Test is related to the Turing Test• Asks probing questions that require human experiences and capacities in order to answer correctly.

• George Lucas’ Star Wars movies (1977) • C3P0 and R2D2• http://www.youtube.com/watch?v=1rorneEGPso

• Star Trek: The Next Generation (1987-1994), • Data is an android with super-human intelligence but no emotions• http://www.youtube.com/watch?v=qcqIYccgUdM


35


36

•Embodiment •Impact of physical embodiment on social interaction•Bartneck - the robot was more engaging to the user than a simulated agent

•Anthropomorphism •Differences in interaction between anthropomorphic and non-anthropomorphic

robots• Children with autism respond to simple mobile car-like robots as well as to

humanoid machines. http://www.youtube.com/watch?v=lZSDnvOoX84• Some studies find that humanoid robots are overwhelming and intimidating • Other studies find that humanoid robots have therapeutic benefit• Biomimetic, (anthropomorphic form) allow human-like gestures and direct

imitation movements, • Non-biomimetic form preserves the appeal of computers and mechanical

objects.

•Simplicity/Complexity of Robot Design •Wainer

• The more realistic or complex a robot was, the more watchful it seemed• Participants were less likely to share personal information with a realistic or

complex robot.


37

•Other Attributes• Reeves and Nass

• Users experienced a stronger sense of social presence from the agent when the voice type and personality matched, than when they did not

• Tapus and Mataric• When a robot’s expressive personality matched the user’s personality, task

performance was better than when the personalities were mismatched

• Robles used agents that gave feedback for a speed-dating application

• Takeuchi• How cultural norms and customs affect the use of robot systems


http://www.youtube.com/watch?v=hKAkLUd1jFg

38

•Assistive robots

• Gives aid or support to a human user• Intended for schools, hospitals, homes

• Socially assistive robotics (SAR) • Elder care, education, social and cognitive disorders, and rehabilitation• Post-operative cardiac surgery recovery and Post-stroke rehabilitation • Rehabilitation

• Motivating rehabilitation through mutual storytelling • A robot and a user construct a story, which, when acted out, require

the user to perform physical therapy exercises.• Feeding, brushing teeth, getting in and out of bed, getting into and out of a

wheelchair, and adjusting a bed for maximum comfort• Envision robots as part of a ubiquitous computing system

• Combines cameras and other sensors in the environment and computer controlled appliances (such as light switches, doors, and televisions)

• The Huggable - a robot outfitted with several sensors to detect different types of touch

• http://www.youtube.com/watch?v=nJXkL7bcQR0


39

• Paro - an actuated stuffed seal which behaves in response to touch and sound.• Goal is to provide the benefits of pet-assisted therapy in nursing

homes that cannot support pets• Studies have shown lowered stress levels in residents interacting with

this robot, as well as an overall increase in the amount of socialization among residents in the common areas of the same facility.

• http://www.youtube.com/watch?v=N5WEbw7DOMY• http://www.youtube.com/watch?v=PNw4oicWmWU

• Diagnosis and socialization of children with autism spectrum disorders (ASD).• Robots can observe children in ways that humans cannot

• Eye-tracking studies have shown remarkable promise when evaluating children for the purposes of diagnosing ASD.

• Robots are a more comfortable social partner for children with ASD than people.

• These robots encourage social behavior, such as dancing, singing, and playing


40

Educational Robotics •Goal to develop robot hardware and software to improve education from elementary school onward

•Traditional robot competitions•Botball•FIRST •RoboCup http://www.youtube.com/watch?v=acrS0kPY-J8

•What need is it filling?•NAO for special education

•http://www.youtube.com/watch?v=LoqU0igk_qQ


41

Ethical Issues for HRI

•The technology to support how a robot can make eye contact is comparatively simple •Evaluating how the person is affected by the robot’s gaze and behavior is difficult.

•Does the person get bored or frustrated? •Does the user consider the robot helpful and effective? •Is the robot trusted to perform its intended tasks?

•Challenges •Unintended uses of the robot

•Unintended situations that might be encountered•If the person needs emergency attention, what is the robot’s responsibility?


42

•How safe is the robot itself, and how safe can the robot make life for its user?

• The ability to maneuver about a scene without unwanted contact or collisions. • Safety also refers to protection (as much as it is possible) of a robot’s user and of the

robot itself.• CASERO- a robot used in a nursing home

• http://www.youtube.com/watch?v=dx0zxr3D_zU

•Scalability

• How well does a robot perform with users from the general population?• How many people can be helped by such a robot• How does speech recognition perform when the speaker has recently suffered a

stroke? • Can the robot adapt its behavior to different users?


43

•Social interaction •The primary means of interaction •The driving force behind the design

•Autonomy

•Autonomy is not desirable in situations such as medication dispersion and therapy monitoring

• The user must clearly retain authority• Rehabilitation should terminate if the user is in pain

•Imitation

•The Total Turing Test • A system communicating in human-like ways (text, speech, facial expressions)

tries to fool a human user into believing it is human •A robot’s personality can effect a user’s compliance

• When exhibiting a serious personality, the robot can provoke a greater degree of compliance than displaying a playful personality

•When the robot’s extroversion/introversion personality traits matched the user’s, task performance was improved

•The Uncanny Valley theory suggests that as a robot becomes very similar in appearance to a human, that robot appears less, rather than more, familiar


44

• Privacy

• In contrast to ubiquitous systems] where a user has no idea of when the system may be watching, robots are tangible and their perception limited and observable.

• A robot can be told to leave when privacy is desired, and the user can observe when privacy is achieved.

• Because of its synthetic nature, a robot is perceived as less of a privacy invasion than a person, especially in potentially embarrassing situations.

• Measuring Social Success • Does the robot successfully achieve the desired social identity?• When the robot is intended to be playful, do users find the robot playful? • If the robot is supposed to be a social peer, do users act as if it were a social peer? • The social identity of the robot (both the personality and the role of the robot) has an

effect on the user’s task performance


45

• How the Robot Perceive the Person

• Emotion recognition• Vocalizations• Speech• Language• Motor acts• Gestures• GSR• Heart Rate• Radio frequency identification (RFID) tags

• Positioned to track and observe children in school hallways to detect when users were in social range, and who they were interacting with over time to help the robot determine appropriate social responses


46

Facebook Facial Recognition Security Or, hold up a photo

Mood Recognition

The API will estimate your mood automatically, with 5 values: Happy, Sad, Surprised, Angry, and Neutral.


47

• Success Relative to Human Caregiver • How does the robot perform relative to a human performing the same task?

• In rehabilitation tasks, measure functional improvement• Overall learning measures such as grades, tests, or evaluations can be used. • Spirometry task robot instructs a cardiac surgery patient to do breathing exercises

compliance with a human was a suitable can be measured. • A key role of assistive HRI is to provide care where human care is not available.

• Cost/Benefit Analysis

• Rehabilitation tasks where a robot could serve as a therapist• Giving advice on specific movements• Motivational coach• Monitoring progress• Learning aid, or as a demonstration, showing a user how to do specific exercises.

• Low-cost robot vacuum cleaners • Some treated the robot as if it were a member of the household, with status

roughly between the vacuum cleaner and a pet. • Others treated it strictly as a device with a purpose.

• Does the use of the robot change the cost/benefit ratio of providing such care?• Doe it make such care available where it was not previously possible?


48

• Impact on Caregivers In some cases, the goal is to make the user’s or caregivers’ job easier and more manageable.

• By automating the prompting and monitoring of spirometry, which must be performed ten times per hour for the critical post-surgery period, the robot made it possible for caregivers to attend to other tasks and provide more individualized services.

• Does the job condition of the caregiver improve as a result of the robot? • How well does the caregiver work with the robot?


49

•Existing Quality of Life Measures

•Dementia Care Mapping (DSM)•Robots to observe and measure socially-sensitive care•Eye-contact•Favorable attention

•Does the robot result in a general increase in the quality of life as perceived by the user?

•Impact on the Role in Community/Society

•When fish tanks were introduced into a nursing home environment to test the effects on residents, observers found an overall increase in nutrition on the part of the participating residents

•The introduction of new objects of social interest into an environment can thus change the dynamics of the community.


50

• Paro

• Reduction of stress proteins in the urine of the participants. • Residents were in the common areas longer and socialized more.

• Robovie • Used a robot to stimulate social interaction among a group of elementary school

students • The robot was able to elevate a student’s status in the group by giving him/her

special information

• Does the robot increase or decrease the amount of socialization in its user community?• Are changes in community due to a robot positive or negative?


http://www.youtube.com/watch?v=RuZFZay7XWU

1 command and natural language command and natural languages hieroglyphs (a 5000 year old...

Documents