1
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL Modeling
Psyc645 -- Week 10
Wayne D. Gray and Thomas Mayfield
2
George Mason UniversityHuman Factors & Applied Cognitive Program
Next Week -- TUESDAY!!
NGOMSL#1 Produce NGOMSL methods from the trace of the VCR (my trace not yours)
Are no pops in NGOMSL. Verifies may be suitable replacements. Yours to
decide.
NGOMSL#2 Create NGOMSL methods for the ManTel task
Should be one set of methods that can be applied to both interfaces and to
1 or 2 queries
Obviously, not all methods need to be used in each analysis
Produce an NGOMSL trace for Dialog-Box 1 query and PopUp 2 queries
Readings Paul Green’s 1999 Human Factors Ergonomics Society conference article
3
George Mason UniversityHuman Factors & Applied Cognitive Program
Psyc645 -- Reprise of GOMS lecture 1
Levels of analysis based on Newell’s Time Scale of Human Action
Important to concept is that behavior at one level can be decomposed into
behavior at next lower level
Show what is added at various levels
Important to the argument that I am trying to make to convince you that analyses
at increasing lower levels entail decreasing degrees of generality (or increasing
degrees of specificity)
Programming VCR is a generally understood concept, but programming my VCR is
very different than programming George Lukas’ VCR. Setting Tom’s watch or Mary
Ann’s watch is very different than setting the clock in my car
Implications?
Behavior is not as unconstrained as we would like to believe. As we get more and
more specific in our description of behavior we find more and more constraints on
behavior.
4
George Mason UniversityHuman Factors & Applied Cognitive Program
Psyc645 - Overview of GOMS lecture 2
NGOMSL analyses Unlike KLM, NGOMSL is a serious notation for discussing
interactive behavior at levels lower than the unit task
Not simply a tool for analyzing design but a vehicle for
representing low-level performance
Today we will work through several examples
5
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL
Natural Language GOMS based on structured natural language notation and a procedure
for constructing them
models are in program form
Control Structure: Hierarchical goal stack
Serial or parallel: Serial
Level of Analysis: In principle, as necessary for your design question
6
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL - why?
More powerful than KLM. Much more useful for
analyzing large systems
More built-in cognitive theory
Provides predictions of operator sequence, execution
time, and time to learn the methods
7
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL - Overall Approach
Step 1: Perform goal/subgoal decomposition
Step 2: Develop a method to accomplish each goal List the actions/steps the user has to do (at as general and high-level as
possible for the current level of analysis)
Identify similar methods/collapse where appropriate
Step 3: Add flow of control (decides)
Step 4: Add verifies
Step 5: Add perceptuals, etc.
Step 6: Add mentals for retrieves, forgets, recalls
Step 7: Add times for each step
Step 8: Calculate total time
8
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL - Example
Car clock
Presetting radio stations simple
perverse
9
George Mason UniversityHuman Factors & Applied Cognitive Program
Provides predictions of methods and operators used to complete
a task. If you provide estimates of operator-duration, you can
get predictions of error-free expert performance time.
NGOMSL--Car Radio example (1)
10
George Mason UniversityHuman Factors & Applied Cognitive Program
set clockset mode
set hourset minute
release knobturn knobdepress knob (and keep it depressed)
press tune “-” buttonpress tune “+” button
NGOMSL-- Car Radio example (2)
Goal/subgoal hierarchy
11
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- Car Radio example (3)
Assumptions User's hands are NOT on the buttons at the beginning.
Time to move hand & arm to time change level is estimated by
2ft "reach" from Barnes (1963): 410 msec.
D = device time, time for clock to move forward one number, =
500 msec (we estimate this on the next slide)
I am assuming a 12-hr clock. Seems good assumption for a car
clock.
Radio is off at beginning and end.
Begin knowing the time you want to set the clock to.
12
George Mason UniversityHuman Factors & Applied Cognitive Program
Right Hand
Cognitive Operators
Visual Perception
Minimum time required to perceive clockTime=targetTime and to release the toggle lever
perceive info (X)
290
verify info (X)
50
initiate release (X)
50
release (X)
100
NGOMSL-- Car Radio example (4)
D = device time, time for clock to move forward one number, =
500 msec
13
George Mason UniversityHuman Factors & Applied Cognitive Program
set clockset mode
set hourset minute
release knobturn knobdepress knob (and keep it depressed)
press tune “-” buttonpress tune “+” button
NGOMSL-- Car Radio example (2)
• (Duplicate slide) (Duplicate slide)
• Note that method for accomplishing the top level goal will have three Note that method for accomplishing the top level goal will have three subgoals and one “release” physical operatorsubgoals and one “release” physical operator
14
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- Car Radio example (5)
stmt time oper
op time
tot time assumptions
Method for goal: SET-CLOCK 0.1 0.10
Step 1Accomplish goal: set Mode to set Clock 0.1 0.10
Step 2
Decide: If curHr≠targetHr Then: Accomplish goal: Change Hour display 0.1 0.10
Step 3
Decide: If curMin≠targetMin Then: Accomplish goal: Change Minute display 0.1 0.10
Step 4 Release "vol/on/off" knob 0.1 B 0.10 0.20
Step 5 Return with goal accomplished 0.1 0.10
Need methods for each of these three subgoalsNeed methods for each of these three subgoals
15
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- Car Radio example (6)
stmt time op
op time
tot time assumptions
Method for goal: set Mode to set Clock 0.1 0.10
Step 1 Recall "vol/on/off" is modeSwitch 0.1 M 0.50 0.60
retrieval from LTM or equivalent (location and perception of labels?) needed
Step 2
Decide: If location of modeSwitch is not known then Locate modeSwitch 0.1 M 1.20 1.30
For this example we assume that location is NOT known. If known then tot. time for this step would be the stmt time, 0.10 sec
Step 3Reach & Grasp "volume/on/off" knob 0.1 R 0.48 0.58
source: Barnes, 1963, reach = 410 msec, grasp = 070 msec
Step 4 Turn knob 0.1 T 0.34 0.44 source: Barnes, 1963
Step 5Home thumb to "volume/on/off" knob 0.1 H 0.40 0.50
400 msec seems too high here, might use 100 msec for button press instead.
Step 6 Press and hold knob 0.1 B 0.10 0.20Step 7 Return with goal accomplished 0.1 0.10
16
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- Car Radio example (7)
stmt time oper
op time
tot time assumptions
Method for goal: Change <time> display 0.1 0.10
Step 1
Decide: If location of setTime switch is not known then Locate setTime switch 0.1 M 1.20 1.30 assume that loc is not known
Step 2
Decide: If finger is NOT on setTime switch then: Reach to setTime switch 0.1 R 0.41 0.51
Step 3Retrieve-from-LTM that <time> = "+ or -" 0.1 M 0.50 0.60
retrieval from LTM or equivalent (location and perception of labels?) needed and takes ≈ 500 msec
Step 4 Determine current setting 0.1 P 0.32 0.42
based upon cpm-goms analysis it should take 420 msec for eye to move to clock and for user to perceive the hour.
Step 5Press and hold setTime switch to <time> 0.1 B 0.10 0.20
cpm-goms calc. is ≈250 msec, go with KLM
Step 6Hold until <curTime> = <targetTime>, (500 x #digits) W
System response time is 500 msec/digit, this is enough time for Ss to perceive and initiate keyUp response.
Step 7 Verify display <time> 0.1 M 1.20 1.30Step 8 Return with goal accomplished 0.1 0.10
17
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- Car Radio example (8)
time for SetClock goal 0.70time for setMode goal 3.82
time for setHour from 1 to 10 9.03
time to setMin from 56 to 50 31.53total 45.08
18
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting a station (1)
Locate Station
Select AM/FM
Initiate seek
Identify StationTerminate
seekSet Mode to
RadioetcSet PreSETEnter PreSet
MODEEnter PreSet
ButtonExit PreSet
MODEGeneral goal hierarchy for presetting a stationPreSET Network
Show
19
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting a station (2)
Note differences between this and previous slide.
Locate Station
Select AM/FM
PreSET Network Show
Set Mode to Radio
etc
Set PreSET
Goal hierarchy for optimal design
press & hold preset button until sound returns
seek & identify
20
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting a station (3)
MODEL 1: SETTING A PRESET TO A NETWORK SHOW - OPTIMAL DESIGN
Method for goal: preset network showStep1: Decide: IF radio is not playing, THEN Accomplish goal:
set-mode-to-radioStep2: Decide: IF show N.E. intended show, THEN Accomplish
goal: locate-stationStep3: Accomplish goal: set-presetStep4: Return with goal accomplished
Method for goal: locate-stationStep1: Decide: IF band N.E. desired band, THEN Accomplish
goal: select AM/FMStep2: Accomplish goal: seek&identifyStep3: Return with goal accomplished
Method for goal: select AM/FMStep1: Reach to button for desired bandStep2: Press button for desired bandStep3: Verify band correctStep4: Return with goal accomplished
21
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting a station (4)
Method for goal: seek&identifyStep1: Reach to right side of "Seek" button Step2: Press "Seek" buttonStep3: Decide: IF show N.E. intended show, THEN Goto 2Step4: Verify show correctStep5: Return with goal accomplished
Method for goal: set-presetStep1: Reach to desired "Preset" buttonStep2: Press and hold "Preset" buttonStep3: Wait until sound returnsStep4: Remove finger from "Preset" buttonStep5: Return with goal accomplished
22
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting perverse (1)
Locate StationSelect AM/FMInitiate seekExit seek
MODE
Set Mode to Radio
etcAssign new PreSETEnter
PreSet MODE
Enter PreSet ButtonExit
PreSet MODE
Goal hierarchy for perverse designPreSET Network ShowDelete old PreSETEnter delete MODE
Enter PreSet Button
Exit delete MODE
Set PreSETEnter seek mode
seek & identify
23
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting perverse (2)
ssEJ
– TUNE +
PUSH CLOCK
ON OFF
FMAM
– +BASS
– +TREBLE
L R
BALANCE
L R
FADE2:41
1 2 3
4 5 6
Radio for Perverse Preset Design
SEEK MODE
ERASE PRESET
MEMORIZE PRESET
ssSEEK
24
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting perverse (3)
25
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting perverse (4)
Method for goal: initiate-seekStep1: Accomplish goal setMode-SEEKStep2: Accomplish goal: seek&identifyStep3: Accomplish goal: exit seek modeStep4: Return with goal accomplished
Method for goal: setMode-<mode>Step1: Reach for <mode> buttonStep2: Press <mode> buttonStep3: Verify <mode> correctStep4: Return with goal accomplished
Method for goal: seek&identify (s/a optimal)Step1: Reach to right side of "Seek" button Step2: Press "Seek" buttonStep3: Decide: IF show N.E. intended show, THEN Goto 2Step4: Verify show correctStep5: Return with goal accomplished
Method for goal: set-presetStep1: Accomplish goal: delete-previous-presetStep2: Accomplish goal: assign-new-presetStep3: Return with goal accomplished
26
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL-- PreSetting perverse (5)
Method for goal: delete-previous-presetStep1: Accomplish goal: setMode-deletePreSet-on
(setMode-<mode>)Step2: Accomplish goal: enter-preset-buttonStep3: Accomplish-goal: setMode-deletePreSet-off
(setMode-<mode>)Step4: Return with goal accomplished
Method for goal: assign-new-presetStep1: Accomplish goal: setMode-preset-on (setMode-
<mode>)Step2: Accomplish goal: enter-preset-buttonStep3: Accomplish-goal: setMode-preset-off (setMode-
<mode>)Step4: Return with goal accomplished
Method for goal: enter-preset-buttonStep1: Reach to desired "Preset" buttonStep2: Press "Preset" buttonStep3: Return with goal accomplished
27
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL--comparison of optimal and perverse designs
Optimal Perverse
number of different methods 6 11
number of methods used 6 18
number of different steps 21 38
number of steps required 21 66
28
George Mason UniversityHuman Factors & Applied Cognitive Program
Cost-of-Knowledge Characteristic Function
Card, S. K., Pirolli, P., & Mackinlay, J. D. (1994).
The cost-of-knowledge characteristic function:
Display evaluation for direct-walk dynamic
information visualizations. In B. Adelson, S.
Dumais, & J. Olson (Eds.), ACM CHI'94 Conference
on Human Factors in Computing Systems (Vol. 1,
pp. 238-244). New York: ACM Press.
CENTURY-METHOD =GOAL: DO-TASK
GOAL: GET-DATETURN-TO [MANUSCRIPT]GET-DATE
GOAL: ACCESS-DAY-CALENDARGET-YEAR . . . if necessaryGOAL: SELECT-CENTURY (1700’s)
POINT-TO (Century=1700-1790s))=> Century-display
GET-YEAR . . . if necessaryGOAL: SELECT-DECADE (1710’s)
POINT-TO (1710-1719))=> Decade display
GET-YEAR . . . if necessaryGOAL: SELECT-YEAR: (1719)
POINT-TO (1719))=> Year-display
GET-MONTH . . . if necessaryGOAL: SELECT-MONTH: (November)
POINT-TO (November))=> Month-display
GET-DAY . . . if necessaryGOAL: SELECT-WEEK: [??]
POINT-TO [23]=> Week-display
GET-DAY . . . if necessaryGOAL: SELECT-DAY: [23]
POINT-TO (23))=> Day-display
33
George Mason UniversityHuman Factors & Applied Cognitive Program
But,
Card, Pirolli, & MacKinley did not do a complete
GOMS analysis
Only did enough of an analysis to determine that the
“volume part” of the task was the repeated accesses
of menus for each component (century, decade, year,
month, week, day)
34
George Mason UniversityHuman Factors & Applied Cognitive Program
Cost-of-Knowledge Function
Small amounts of knowledge can be accessed
quickly with access costs increasing as amount of
knowledge accessed increases ≈6.9 s for 1 day
≈10.4 s for 7 days
≈14 s for 30 days
≈17.5 s for 365 days
≈21 s for 3,562 days
≈24.6 s for 36,525 days
A = Spiral Calendar as isA = Spiral Calendar as isB = SC w/ 2 s cycle time (was 3.5)B = SC w/ 2 s cycle time (was 3.5)C = Eliminate need to select “week” can do C = Eliminate need to select “week” can do that when select day from the month calendarthat when select day from the month calendar
D = Eliminate “week” & improve cycle timeD = Eliminate “week” & improve cycle time
36
George Mason UniversityHuman Factors & Applied Cognitive Program
Design Improvements?
Speed up system response time (SRT)
Time to access = 3.3 + 3.5 * Ncycles
Imply in text that it takes 1 s to point to item and 2.5 s for system to bring
up item
Eliminate some of the cycles -- seem redundant
Get rid of the week cycle
Have users bring up month and directly select day from the month
calendar rather than first selecting week and then month
Issue --
Implicit estimate of 2.5 s srt seems tediously long
Suggest need for a full NGOMSL analysis
37
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL Methods for Spiral Calendar
stmt time op t/f
op time tot time assumptions
Method for goal: Set-Date 0.1 0.10Step 1 Accomplish goal: get-date 0.1 0.10Step 2 Accomplish goal: Access-day-calendar 0.1 0.10Step 3 Return with goal accomplished 0.1 0.10
Total Time 0.40
Method for goal: get-date 0.1 0.10
Step 1 Decide: IF page not turned, then turn page 0.1 R T 0.88 0.98
source: Barnes, 1963, reach = 410 msec, grasp = 070 msec, + 400 est for flip. ** could analysis this further e.g. CPM-GOMS
Step 2 Read Date 0.1 M 1.2 1.30use M for estimate of time to read and put into memory
Step 3 Home hand to keyboard 0.1 H 0.4 0.50Step 4 Return with goal accomplished 0.1 0.10
Total Time 2.98
Top method and one subordinateTop method and one subordinate
38
George Mason UniversityHuman Factors & Applied Cognitive Program
NGOMSL Methods (cont’)
Method for goal: access-day-calendar 0.1 0.10Step 1 Decide: IF forgot date, then Read Date 0.1 M F 1.20 0.10Step 2 Accomplish Goal: Set-<timeUnit> 0.1 0.10
Step 3Decide: If current_day N.E. target_day, then GO TO Step 1 0.1 0.10
Step 4 Verify date 0.1 M 1.20 1.30Step 5 Return with goal accomplished 0.1 0.10
Total Time 1.80
Method for goal: Set-<timeUnit> 0.1 0.10Step 1 Point to <timeUnit> 0.1 P 1.10 1.20Step 2 ClickOn <timeUnit> 0.1 BB 0.20 0.30Step 3 Wait for System Response 0.1 W 1.00 1.10Step 4 Return with goal accomplished 0.1 0.10
Total Time 2.80
0.10 Method for goal: Set-Date0.10 Step 1 Accomplish goal: get-date0.10 Method for goal: get-date0.98 Step 1 Decide: IF page not turned, then turn page1.30 Step 2 Read Date0.50 Step 3 Home hand to keyboard0.10 Step 4 Return with goal accomplished0.10 Step 2 Accomplish goal: Access-day-calendar0.10 Method for goal: access-day-calendar0.10 Step 1 Decide: IF forgot date, then Read Date0.10 Step 2 Accomplish Goal: Set-<timeUnit>0.10 Method for goal: Set-<timeUnit>1.20 Step 1 Point to <timeUnit>0.30 Step 2 ClickOn <timeUnit>1.10 Step 3 Wait for System Response0.10 Step 4 Return with goal accomplished0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 10.10 Step 1 Decide: IF forgot date, then Read Date0.10 Step 2 Accomplish Goal: Set-<timeUnit>0.10 Method for goal: Set-<timeUnit>1.20 Step 1 Point to <timeUnit>0.30 Step 2 ClickOn <timeUnit>1.10 Step 3 Wait for System Response0.10 Step 4 Return with goal accomplished0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 10.10 Step 1 Decide: IF forgot date, then Read Date0.10 Step 2 Accomplish Goal: Set-<timeUnit>0.10 Method for goal: Set-<timeUnit>1.20 Step 1 Point to <timeUnit>0.30 Step 2 ClickOn <timeUnit>1.10 Step 3 Wait for System Response0.10 Step 4 Return with goal accomplished0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 11.30 Step 4 Verify date0.10 Step 5 Return with goal accomplished0.10 Step 3 Return with goal accomplished
40
George Mason UniversityHuman Factors & Applied Cognitive Program
How well does NGOMSL fit the data?
Pretty well by rPretty well by r22
41
George Mason UniversityHuman Factors & Applied Cognitive Program
How well does NGOMSL fit the data?
0
5
10
15
20
25
30
1 2 3 4 5 6
Number of Cycles
Time in Seconds
empirical data card model NGOMSL
Pretty well compared to the equation they Pretty well compared to the equation they derived from the dataderived from the data
42
George Mason UniversityHuman Factors & Applied Cognitive Program
How well does NGOMSL fit the data?
0
5
10
15
20
25
30
35
1 2 3 4 5 6
Number of Cycles
Time in Seconds
empirical data card model NGOMSL NGOMSL w/2.5 s srt
But not too well when I assume a 2.5 sec But not too well when I assume a 2.5 sec system response timesystem response time
43
George Mason UniversityHuman Factors & Applied Cognitive Program
What Happened?
Did the modeler cheat?
Let’s examine the assumptions of the model
Critical assumptions concern the time per cycle and
the elements that compose the cycle
Card et al., empirically derived 3.5 s per cycle
The NGOMSL model, which assumed a 1 s SRT,
predicted 3.1 s per cycle
0.10 Method for goal: Set-Date0.10 Step 1 Accomplish goal: get-date0.10 Method for goal: get-date0.98 Step 1 Decide: IF page not turned, then turn page1.30 Step 2 Read Date0.50 Step 3 Home hand to keyboard0.10 Step 4 Return with goal accomplished0.10 Step 2 Accomplish goal: Access-day-calendar0.10 Method for goal: access-day-calendar0.10 Step 1 Decide: IF forgot date, then Read Date0.10 Step 2 Accomplish Goal: Set-<timeUnit>0.10 Method for goal: Set-<timeUnit>1.20 Step 1 Point to <timeUnit>0.30 Step 2 ClickOn <timeUnit>1.10 Step 3 Wait for System Response0.10 Step 4 Return with goal accomplished0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 10.10 Step 1 Decide: IF forgot date, then Read Date0.10 Step 2 Accomplish Goal: Set-<timeUnit>0.10 Method for goal: Set-<timeUnit>1.20 Step 1 Point to <timeUnit>0.30 Step 2 ClickOn <timeUnit>1.10 Step 3 Wait for System Response0.10 Step 4 Return with goal accomplished0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 10.10 Step 1 Decide: IF forgot date, then Read Date0.10 Step 2 Accomplish Goal: Set-<timeUnit>0.10 Method for goal: Set-<timeUnit>1.20 Step 1 Point to <timeUnit>0.30 Step 2 ClickOn <timeUnit>1.10 Step 3 Wait for System Response0.10 Step 4 Return with goal accomplished0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 11.30 Step 4 Verify date0.10 Step 5 Return with goal accomplished0.10 Step 3 Return with goal accomplished
3.1 s3.1 s
3.1 s3.1 s
3.1 s3.1 s
45
George Mason UniversityHuman Factors & Applied Cognitive Program
Where is the problem?
For NGOMSL to accept 2.5 s SRT means either accepting a very bad fit, or
changing the model
But, what can be changed?
My guess is that they never measured the SRT of
their system
In any event it is incredible that they assumed that all
of the cycle time not devoted to pointing was
attributable to SRT
46
George Mason UniversityHuman Factors & Applied Cognitive Program
Two Morals to this Story?
First Even good researchers can be careless when they do not check
their assumptions against a model
(& yes, Card, Pirolli, and Mackinlay are VERY good researchers)
Second Before investing lots of effort into jazzing up system response
time, do a profile analysis with a complete GOMS model
47
George Mason UniversityHuman Factors & Applied Cognitive Program
Next Week -- TUESDAY!!
NGOMSL#1 Produce NGOMSL methods from the trace of the VCR (my trace not yours)
Are no pops in NGOMSL. Verifies may be suitable replacements. Yours to decide.
NGOMSL#2 Create NGOMSL methods for the ManTel task
Should be one set of methods that can be applied to both interfaces and to 1 or 2
queries
Obviously, not all methods need to be used in each analysis
Produce an NGOMSL trace for Dialog-Box 1 query and PopUp 2 queries
Readings Paul Green’s 1999 Human Factors Ergonomics Society conference article