Command and Command and Natural LanguagesNatural Languages

Human Computer InteractionHuman Computer Interaction

CIS 6930/4930CIS 6930/4930

Section 4188/4186Section 4188/4186

IntroIntro► Languages are a natural way Languages are a natural way

to communicateto communicate► Communication with systemsCommunication with systems

Initially, Initially, programming programming languageslanguages

Scripting languagesScripting languages Database queryDatabase query Command languagesCommand languages

► With menus and DM, why have With menus and DM, why have languages? For some tasks,languages? For some tasks, NaturalNatural FasterFaster For tasks with many options, most For tasks with many options, most

effectiveeffective Small footprint (screen, power, Small footprint (screen, power,

size)size) Logistics: Generating help, Logistics: Generating help,

verification, etc.verification, etc.

IntroIntro► Languages negatives?Languages negatives?

User memoryUser memory Could be crypticCould be cryptic Retention, learning, Retention, learning,

frustrationfrustration

► Ex. Web addressesEx. Web addresses Class web pageClass web page

► Initiate vs. respond (ex. Initiate vs. respond (ex. Unix)Unix)

Functionality to Support Users’ Functionality to Support Users’ TaskTask

► People use systems to People use systems to accomplish a task. accomplish a task.

► How do you build a How do you build a command structure to command structure to support this?support this?

► Identify user tasksIdentify user tasks Usually create 1 to 1 for Usually create 1 to 1 for

functionality with actions functionality with actions and objectsand objects

Common error: Too Common error: Too many actions and objectsmany actions and objects► Overwhelms usersOverwhelms users► More code, more errors, More code, more errors,

more cluttermore clutter Insufficient actions: very Insufficient actions: very

frustrating!frustrating!

Functionality to Support Users’ Functionality to Support Users’ TaskTask

► Create a list of tasksCreate a list of tasks Use a column for Use a column for

frequency of expected frequency of expected useuse

High frequency tasks High frequency tasks should be easiest to should be easiest to remember and carry outremember and carry out

► Careful thought into Careful thought into user baseuser base Ex. do you need macros?Ex. do you need macros?

► Transition diagram Transition diagram helps (Fig 8.1)helps (Fig 8.1)

Command-Organization Command-Organization StrategiesStrategies

►Strategies to create commandsStrategies to create commands►Agreeing on a interface concept aides Agreeing on a interface concept aides

retention, learning, and problem solvingretention, learning, and problem solving►Not that straightforwardNot that straightforward

Ex. Load/Save, Read/Write (notes vs. Ex. Load/Save, Read/Write (notes vs. folders), Open/Close (files vs. notes)folders), Open/Close (files vs. notes)

►Common mistake: Choose a computer Common mistake: Choose a computer metaphor instead of a domain metaphormetaphor instead of a domain metaphor Ex. e-mailEx. e-mail

Command Command Organization Organization

StrategiesStrategies

► Simple Command SetSimple Command Set # of commands = # of tasks# of commands = # of tasks Ex. MUDsEx. MUDs

► Ex. Look, go, moveEx. Look, go, move Cons: Large # of commands Cons: Large # of commands

► Ex. VIEx. VI► Commands plus Commands plus

arguments/optionsarguments/options Each command is followed by >=0 Each command is followed by >=0

argumentsarguments► Ex. Copy X YEx. Copy X Y► Include keyword labels: Copy Include keyword labels: Copy

From=X To=YFrom=X To=Y Pros: Pros: readability, fewer semantic readability, fewer semantic

errors, better for noviceserrors, better for novices Cons:Cons: increased syntax errors, increased syntax errors,

slower for expertsslower for experts► Hierarchical command Hierarchical command

structurestructure Tree structure of commands (like Tree structure of commands (like

menus)menus) Let’s create one for filesLet’s create one for files

► Create,display,remove,copy, moveCreate,display,remove,copy, move► File, process, directoryFile, process, directory► File, printer, screenFile, printer, screen

Easy to write tutorialsEasy to write tutorials

Benefits of StructureBenefits of Structure► Study:Study: Error rates for UNIX Error rates for UNIX

3 to 53% (Hanson ’84)3 to 53% (Hanson ’84) Common commands too! (18% for Common commands too! (18% for mvmv, 30% for , 30% for cpcp)) Experts gain some (perhaps sadistic) fulfillment and club Experts gain some (perhaps sadistic) fulfillment and club

‘inclusion’ by understanding complex command languages‘inclusion’ by understanding complex command languages► BenefitsBenefits

LearningLearning MemoryMemory Problem solvingProblem solving

► Elegancy vs. ConsistencyElegancy vs. Consistency Apply ‘edit’ vs. ‘revise, change, replace’, etc.Apply ‘edit’ vs. ‘revise, change, replace’, etc. Reduces errorReduces error

► Other examplesOther examples Some commands are two characters, others notSome commands are two characters, others not What is a binary decision? On/Off, True/False, etc.What is a binary decision? On/Off, True/False, etc. Multiple design groupsMultiple design groups

► Solution: Solution: Create a guidelines document. Good for managers Create a guidelines document. Good for managers and designersand designers

Benefits of StructureBenefits of Structure► Study: Study: Benefits to argument ordering consistency (Barnard ’81)Benefits to argument ordering consistency (Barnard ’81)

Ex. Source or ID is always a certain argumentEx. Source or ID is always a certain argument► Symbols vs. KeywordsSymbols vs. Keywords

Which is better: FIND:/TOOTH/;-1 or BACKWORD TO “TOOTH”Which is better: FIND:/TOOTH/;-1 or BACKWORD TO “TOOTH” What about for different grade of users? (Novice, Familiar, Expert)?What about for different grade of users? (Novice, Familiar, Expert)? Study: Study: Table 8.1 (Ledgard ’80)Table 8.1 (Ledgard ’80) Clarity overrides speedClarity overrides speed

► Study: Study: (Carroll ’82)(Carroll ’82) Effect of congruency [meaningful pairs] and hierarchies on Effect of congruency [meaningful pairs] and hierarchies on

performanceperformance Ex. Open/Close Left/RightEx. Open/Close Left/Right Memory and problem solving improved w/ congruencyMemory and problem solving improved w/ congruency Error rates reduced w/ congruent hierarchyError rates reduced w/ congruent hierarchy Results: Results:

► Congruency = very goodCongruency = very good► Hierarchy = good for large command setsHierarchy = good for large command sets

► Good things to have: positional and grammatical consistency, Good things to have: positional and grammatical consistency, congruent pairing, hierarchical formcongruent pairing, hierarchical form

Naming and AbbreviationsNaming and Abbreviations

►Let’s look at UNIXLet’s look at UNIX mkdir (make directory)mkdir (make directory) ls (list directory)ls (list directory) cd (change directory)cd (change directory) rm (remove file)rm (remove file) pwd (print working directory)pwd (print working directory)

►What’s wrong with these What’s wrong with these abbreviations?abbreviations? No standard method to derive them!No standard method to derive them! Standards are important aidStandards are important aid

Specificity vs. GeneralitySpecificity vs. Generality► Specific – more descriptiveSpecific – more descriptive► General – more familiar and easier to understandGeneral – more familiar and easier to understand► Study:Study: 2 week training session 2 week training session

Resulted in specific > general (Barnard ’81)Resulted in specific > general (Barnard ’81)► Study:Study: (Black and Moran ’82) – pg. 328. Different (Black and Moran ’82) – pg. 328. Different

terms for insert/deleteterms for insert/delete Infrequent, discriminating: insert/deleteInfrequent, discriminating: insert/delete Frequent, discriminating: add/remove Frequent, discriminating: add/remove Infrequent, nondiscriminating: amble/perceiveInfrequent, nondiscriminating: amble/perceive Frequent, nondiscriminating: walk/viewFrequent, nondiscriminating: walk/view General: alter/correctGeneral: alter/correct Nondiscriminating nonwords: GAC/MIKNondiscriminating nonwords: GAC/MIK Disciminating nonwords: abc-adbc/abc-acDisciminating nonwords: abc-adbc/abc-ac Best = infrequent, discriminating wordsBest = infrequent, discriminating words Worst – generalWorst – general Not bad – nonsenseNot bad – nonsense What does this teach us? (distinctive-ness is a plus)What does this teach us? (distinctive-ness is a plus)

Abbreviation StrategiesAbbreviation Strategies► Should be easy to express with input device Should be easy to express with input device

Keyboard, pen (PDA), speech recognition, mouseKeyboard, pen (PDA), speech recognition, mouse► Error rates increase w/ more complex commandsError rates increase w/ more complex commands

Shift, Ctrl (plus harder for disabled or motor-damaged users)Shift, Ctrl (plus harder for disabled or motor-damaged users) Brevity is good, but must weigh w/ retention and learningBrevity is good, but must weigh w/ retention and learning Study:Study: (Landauer ’83) novices don’t mind typing out full (Landauer ’83) novices don’t mind typing out full

names [increases confidence] (<5 to 7 uses)names [increases confidence] (<5 to 7 uses)► Abbreviation StrategiesAbbreviation Strategies

Simple truncation – commands must be distinguishableSimple truncation – commands must be distinguishable Vowel dropVowel drop First and last letterFirst and last letter First letter of each wordFirst letter of each word Standard abbreviations – familiarityStandard abbreviations – familiarity Phonics – XQTPhonics – XQT

Abbreviation GuidelinesAbbreviation Guidelines1.1. Simple Simple primary ruleprimary rule2.2. Secondary ruleSecondary rule abbreviations should be denoted abbreviations should be denoted

by some distinguishing characterby some distinguishing character3.3. Minimal use of secondary ruleMinimal use of secondary rule4.4. Users should know the rulesUsers should know the rules5.5. Truncation should be used, except when too many Truncation should be used, except when too many

similar actionssimilar actions6.6. Fixed-length is preferable to variable lengthFixed-length is preferable to variable length7.7. Computer generated messages should NOT use Computer generated messages should NOT use

abbreviationsabbreviations8.8. Should be greater than >2 savings for abbreviationsShould be greater than >2 savings for abbreviations9.9. Consider a command menu.Consider a command menu.

1.1. Ex. Imaging Control [really benefits only intermittent users]Ex. Imaging Control [really benefits only intermittent users]

10.10. Underscore critical letter (e.g. Windows)Underscore critical letter (e.g. Windows)

Natural Languages in Natural Languages in ComputingComputing

►One (popular) trend is to communicate with One (popular) trend is to communicate with the computer using natural languagesthe computer using natural languages This involves both This involves both input input and and outputoutput

►Why is this hard?Why is this hard? Subtleties (mood, accent, culture)Subtleties (mood, accent, culture) Context sensitiveContext sensitive Large user baseLarge user base

► Currently:Currently: Very restricted domains (stock trading phone Very restricted domains (stock trading phone

system)system) Processed input and/or outputProcessed input and/or output Formatted texts (weather reports, tech reports, etc.)Formatted texts (weather reports, tech reports, etc.) Can’t do: poems, freeform conversationsCan’t do: poems, freeform conversations Rough translations help w/ getting the ‘jist’ of most Rough translations help w/ getting the ‘jist’ of most

thingsthings►Ex. language learnersEx. language learners

Natural Language InteractionNatural Language Interaction► NLI – Star Trek-type cognitionNLI – Star Trek-type cognition► Pros: Pros:

Don’t have to remember syntax or menu conventionsDon’t have to remember syntax or menu conventions► Cons: (besides harder)Cons: (besides harder)

Not necessarily fasterNot necessarily faster Not necessarily a goal for every type of app.Not necessarily a goal for every type of app.

► Ex. Air traffic controlEx. Air traffic control► Not knowing the extent of capabilities hampers novice or Not knowing the extent of capabilities hampers novice or

intermittentintermittent► Experts like precise commandsExperts like precise commands► Data input/output types and rates vary greatly! 1:1000Data input/output types and rates vary greatly! 1:1000

► Combine with the OAI model and provide a visual Combine with the OAI model and provide a visual representation of optionsrepresentation of options

► Overzealousness is hamperingOverzealousness is hampering► How can a system handle the high error rates with How can a system handle the high error rates with

most NLI?most NLI?

Natural Language InteractionNatural Language Interaction► Ex. Use NLI for finances Ex. Use NLI for finances

(Shneiderman ’80)(Shneiderman ’80) ‘‘Pay $33 to University of Florida’Pay $33 to University of Florida’ 91% accuracy91% accuracy Why isn’t it used now?Why isn’t it used now?

► Quicken, et. al., doesn’t use NLIQuicken, et. al., doesn’t use NLI► Faster, easier to understand, visuals Faster, easier to understand, visuals

helphelp

► Loebner Prize (’91) – Turing TestLoebner Prize (’91) – Turing Test ((

www.loebner.net/Prizef/loebner-prize.html))

researchers aren’t that researchers aren’t that enthusiasticenthusiastic

► Mainstream – HAL, ELIZAMainstream – HAL, ELIZA► Current: Current:

Dialog interaction is too difficultDialog interaction is too difficult Rigorous evaluation of NLIRigorous evaluation of NLI Identify keywords in documentsIdentify keywords in documents Visual recognition is just fasterVisual recognition is just faster Speech RecSpeech Rec

► Problems: Predictable responsesProblems: Predictable responses

►Summary: sometimes Summary: sometimes developers believe NLI should developers believe NLI should operate w/o Direct Manipulation. operate w/o Direct Manipulation. This would be a mistake for many This would be a mistake for many appsapps

Natural Language Queries and Natural Language Queries and Question AnsweringQuestion Answering

► Instead of full NLI, look at a subsetInstead of full NLI, look at a subset Natural Language QueriesNatural Language Queries

► Easier to parseEasier to parse► Ex. AskJeevesEx. AskJeeves► If input to a database, it could be constrained enoughIf input to a database, it could be constrained enough

But is it better than SQL?But is it better than SQL?► Study: Study: SQL was faster (Small ’83, Jarke ’85)SQL was faster (Small ’83, Jarke ’85)

► Case study: INTELLECT Case study: INTELLECT Search financial mainframe databases in the 80sSearch financial mainframe databases in the 80s 400 installations400 installations Text input for queryText input for query Helps because keywords are well defined (like cities)Helps because keywords are well defined (like cities) Used fields to help structure inputUsed fields to help structure input Used structured output to help train users on structured Used structured output to help train users on structured

inputinput► Ex. PRINT THE CHECK NUMBERS WITH PAYEE = MICROSOFTEx. PRINT THE CHECK NUMBERS WITH PAYEE = MICROSOFT

Novices still had a hard time, ideal user: knowledgeable Novices still had a hard time, ideal user: knowledgeable intermittent userintermittent user

Natural Language Queries and Natural Language Queries and Question AnsweringQuestion Answering

► Other products: Other products: Symantec’s Q&A (late 80s)Symantec’s Q&A (late 80s) Microsoft’s English Query Microsoft’s English Query

(’99)(’99)► NLQA (Answering)NLQA (Answering)

Return a set of potential Return a set of potential answersanswers► Instead of an natural Instead of an natural

language answerlanguage answer► Reduce accuracy of Reduce accuracy of

response response ► Let the user huntLet the user hunt

Requires users to be Requires users to be domain knowledgeabledomain knowledgeable

Domain of search could Domain of search could make things difficult (terms make things difficult (terms like year or pay)like year or pay)

Questions need to be well Questions need to be well formed (not guaranteed)formed (not guaranteed)

Text-Database SearchingText-Database Searching► Text-Database searching using NLQText-Database searching using NLQ

Court documentsCourt documents Photo/multimediaPhoto/multimedia NewsNews

► Spectrum of approachesSpectrum of approaches Understanding QueryUnderstanding Query

► Finding synonymsFinding synonyms► Reduce noise wordsReduce noise words► Handling singulars vs. plurals (Handling singulars vs. plurals (stemmingstemming))► Misspellings, pronouns, specific wordsMisspellings, pronouns, specific words

ExtractionExtraction► Breaks down query into fields, does typical database lookupBreaks down query into fields, does typical database lookup► Good for large databases (legal, medical, etc.) with formatted Good for large databases (legal, medical, etc.) with formatted

queriesqueries Study:Study: (Voorhees ’02), NLQ seems to provide rapid (Voorhees ’02), NLQ seems to provide rapid

learning and progresslearning and progress Provide more relevant searches vs. just keywordsProvide more relevant searches vs. just keywords Still not returning exact search resultStill not returning exact search result Potentially faster (ex. user has partial information)Potentially faster (ex. user has partial information)

Natural Language Text Natural Language Text GenerationGeneration

►Prepare structured reports using NLPrepare structured reports using NL►Goal: create stories?Goal: create stories?►Sports game recaps, willsSports game recaps, wills►What’s the source?What’s the source?

DatabaseDatabase Interactive systemInteractive system

►Natural language could help doctors Natural language could help doctors (they don’t want to switch gaze), (they don’t want to switch gaze), militarymilitary

Adventure Games and Adventure Games and Instructional SystemsInstructional Systems

► Recall old Zork or King’s Quest Recall old Zork or King’s Quest games?games? Problems: didn’t get the phrasing just Problems: didn’t get the phrasing just

right…right… Pros: Pros: The ‘exploration’ is a plus since The ‘exploration’ is a plus since

it aids to the experienceit aids to the experience Cons: Cons: Too much exploration is Too much exploration is

frustratingfrustrating► Instructional TutorialsInstructional Tutorials

AutoTutor (Glassner) pg. 340AutoTutor (Glassner) pg. 340► Uses agents to help studentsUses agents to help students► A better interface for learning?A better interface for learning?

Cognitive Tutor (Carnegie Learning)Cognitive Tutor (Carnegie Learning)► Teach math, geometry, algebra, etc.Teach math, geometry, algebra, etc.

Provide feedback and guidance w/ NL Provide feedback and guidance w/ NL using accepted pedagogy approachesusing accepted pedagogy approaches► Helps students (Study: Di Eugenio ’02)Helps students (Study: Di Eugenio ’02)