![Page 1: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/1.jpg)
The User In Experimental Computer Systems Research
Peter A. DindaGokhan Memik, Robert Dick
Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff
Department of Electrical Engineering and Computer ScienceNorthwestern University
http://presciencelab.org
![Page 2: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/2.jpg)
2
Experimental Computer Systems Researchers Should…
• Incorporate user studies into the evaluation of systems
• Incorporate direct user feedback into the design of systems
![Page 3: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/3.jpg)
3
Experimental Computer Systems Researchers Should…
• Incorporate user studies into the evaluation of systems– No such thing as the typical user– Really measure user satisfaction
• Incorporate direct user feedback into the design of systems– No such thing as the typical user– Measure and leverage user variation
![Page 4: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/4.jpg)
4
Outline• Prescription• Experiences with user studies and direct user
feedback– User comfort with resource borrowing– User-driven scheduling of interactive VMs– User satisfaction with CPU frequency– User-driven frequency scaling– User-driven control of distributed virtualized envs.– Prospects for speculative remote display
• Principles for client/server context• General advice
![Page 5: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/5.jpg)
5
Experiences in Detail• Concepts : ExpCS 2007 @ FCRC• Specific Projects
– User comfort with resource borrowing• HPDC 2004, NWU-CS-04-28
– User-driven scheduling of interactive VMs• Grid 2004, SC 2005, VTDC 2006, NWU-EECS-06-07
– User satisfaction with CPU frequency• CAL 2006, SIGMETRICS 2007, NWU-EECS-06-11
– User-driven frequency scaling (/process-driven voltage scaling)• CAL 2006, SIGMETRICS 2007, NWU-EECS-06-11
– User-driven control of distributed virtualized envs.• Portion of Bin Lin’s thesis, see also ICAC 2007
– Prospects for speculative remote display• NWU-EECS-06-08
![Page 6: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/6.jpg)
6
User Comfort With Resource Borrowing
• Systems that use “spare” resources on desktops for other computation– *@Home, Condor on desktops, etc.
• How much can they borrow before discomforting user?– Inverse: How much must desktop
replacement system give?
![Page 7: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/7.jpg)
7
User Comfort With Resource Borrowing
• Developed system for controlled resource borrowing given a profile– CPU contention, disk BW contention, physical
memory pages– User presses “irritation button” to stop
• User study – 38 participants– Four apps
• Word, Powerpoint, Web browsing, Game
– Ramp, Step, Placebo profiles– Double blinded
![Page 8: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/8.jpg)
8
Example Result
Massive Variation inUser Response
![Page 9: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/9.jpg)
9
User-driven Scheduling of Interactive VMs
• Virtual machine-based desktop replacement model– VM runs on backend server– User connects with remote display– VM is scheduled according to periodic real-
time model• Allows straightforward mixing of batch and
interactive VMs + isolation properties
• What should interactive VM’s schedule be?
![Page 10: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/10.jpg)
10
User-driven Scheduling of Interactive VMs
• VSched scheduler on server
• User interface on client
Non-centering joystick allows user to set schedule
$10 interface in studyCheaper interfaces possible
Onscreen display indicates price of current schedule
Also indicates whenschedule cannot be admitted
![Page 11: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/11.jpg)
11
User-driven Scheduling of Interactive VMs• User study
– 18 participants– 4 applications
• Word, Powerpoint, Internet browsing, Game
– Survey response + measurement• Deception scheme to control bias in survey response
• Results– Almost all could find a setting that was comfortable– Almost all could find a setting that was comfortable and
believed to be of lowest cost– Lowest cost highly variable, as expected given previous
results– <1 minute convergence typical
• Interface captures individual user tradeoffs– Fewer cycles for tolerant users– More cycles for others
![Page 12: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/12.jpg)
12
User Satisfaction With CPU frequency
• Modern processors can lower frequency to reduce power consumption– Software control: DVFS - conservative
• How satisfied are users of different applications at different clock frequencies?
• User Study– 8 users– 3 frequencies + Windows DVFS– 3 apps
• Presentation, Animation, Game– Rate comfort on 1 to 10 scale– Double-blinded
![Page 13: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/13.jpg)
13
Example Results
• Dramatic variation in user satisfaction for fixed frequencies– And for DVFS
Presentation
Game
![Page 14: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/14.jpg)
14
User-driven Frequency Scaling
• Developed system to dynamically customize frequency to user– User presses “irritation button” as input– 2 very simple learning algorithms
• User study – 20 participants– Three apps
• Powerpoint, Animation, Game
– Comparison with Windows DVFS
– Double blinded
![Page 15: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/15.jpg)
15
Example Results (Measured System Power)
Users
% g
ain
over
Win
dow
s D
VF
S
Users % g
ain
over
Win
dow
s D
VF
S
Powerpoint
Game
![Page 16: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/16.jpg)
16
Outline• Prescription• Experiences with user studies and direct user
feedback– User comfort with resource borrowing– User-driven scheduling of interactive VMs– User satisfaction with CPU frequency– User-driven frequency scaling– User-driven control of distributed virtualized envs.– Prospects for speculative remote display
• Principles for client/server context• General advice
![Page 17: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/17.jpg)
17
Principles for the Client/Server Context
• User variation– Considerable variation in user satisfaction with any
given operating point– No such thing as a typical user
• User-specified performance– Have user tell system software how satisfied he is– No decoupling of user response from user and
OS-level measurements• Think global feedback
• Thin, simple user-system interface– One bit is a lot of information compared to zero
• Learning to decrease interaction rate– Model the individual user
![Page 18: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/18.jpg)
18
Outline• Prescription• Experiences with user studies and direct user
feedback– User comfort with resource borrowing– User-driven scheduling of interactive VMs– User satisfaction with CPU frequency– User-driven frequency scaling– User-driven control of distributed virtualized envs.– Prospects for speculative remote display
• Principles for client/server context• General advice
![Page 19: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/19.jpg)
19
General Advice for Evaluating Systems with User Studies
• Consult an HCI or psychology expert– User studies are different but not impossible– At least consult the literature
• Engage your IRB early– These are “social science”-based studies– Easier the second time around
• Accept small study size– Parameter sweeps, hundreds of traces impossible– Internet volunteerism not especially effective – Use non-user studies to augment if possible– Robust statistics
![Page 20: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/20.jpg)
20
General Advice for Evaluating Systems with User Studies
• Accept that random sample unlikely– Selection bias estimation, if possible– Report all your data, not just summaries
• Histogram instead of curve fit
• Measure the noise floor / placebo effect– Vital to determine how much of user satisfaction is
actually under your control
• Double-blind to greatest extent possible– Investigator bias and subject bias
![Page 21: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/21.jpg)
21
General Advice for Evaluating Systems with User Studies
• Correlate system-level measurements with user responses to validate the latter– Consider deception when this is impossible
• Eliminate user-visible extraneous information during any study– What the user knows can hurt you
• Example: disk light
![Page 22: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/22.jpg)
22
General Advice for Incorporating Direct User Feedback
• Out-of-band devices work best– Avoid cognitive context switch
• Use as little input as possible– One bit is much more information than zero– Utility of input may not be clear to user
• Output as little information as possible• Minimize input rate through learning• Bridge explicit feedback to implicit feedback
when possible
![Page 23: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/23.jpg)
23
Experimental Computer Systems Researchers Should…
• Incorporate user studies into the evaluation of systems– No such thing as the typical user– Really measure user satisfaction
• Incorporate direct user feedback into the design of systems– No such thing as the typical user– Measure and leverage user variation
![Page 24: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/24.jpg)
24
For MoreInformation
• Peter Dinda– http://pdinda.org
• Prescience Lab– http://presciencelab.org
![Page 25: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/25.jpg)
25
User-driven Control of Distributed Virtual Environments
• Area of current exploration (part of Lin’s thesis)
• Idea: Can we frame these problems as games that naïve or expert users/admins can solve?
• Initial results interesting, but still too early too tell– Scaling– Dimensionality– Categorical dimensions– …
![Page 26: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/26.jpg)
26
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• Systems software’s decisions have dramatic effect on user experience
• But how does systems software know how well it is doing?
Systems Software
Application(s)
Satisfaction with System/AppCombination
IndividualUser
Interface Considerations
Resource Managementand SchedulingConsiderations
Core API
![Page 27: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/27.jpg)
27
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• One option: let the application tell it!
• But how does the application know?
Systems Software
Application(s)
Satisfaction with System/AppCombination
IndividualUser
Interface Considerations
Resource Managementand SchedulingConsiderations
Core API Policy API
![Page 28: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/28.jpg)
28
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• One option: let the application tell it!
• Assume typical user and apply general rules derived from him/her– And figure out how to
translate to the policy APISystems Software
Application(s)
Satisfaction with System/AppCombination
TypicalUser
Interface Considerations
Resource Managementand SchedulingConsiderations
Core API Policy API
<500 ms latencyand <100 ms jitter
![Page 29: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/29.jpg)
29
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• One option: let the application tell it!
• Or formalize tradeoffs– And figure out how to
translate to the policy API
Systems Software
Application(s)
Satisfaction with System/AppCombination
TypicalUser
Interface Considerations
Resource Managementand SchedulingConsiderations
Core API Policy API
Utility Function LatencySat
isfa
ctio
n
![Page 30: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/30.jpg)
30
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• Another option: generalize over applications and infer user experience
Systems Software
Application(s)
Satisfaction with System/AppCombination
TypicalUser
Resource Managementand SchedulingConsiderations
Core API
LatencySat
isfa
ctio
nInterface Considerations
Inferred Latency
Good/Bad?
![Page 31: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/31.jpg)
31
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• Another option: Get the utility function right from the individual user– Assuming he/she
knows it…
Systems Software
Application(s)
Satisfaction with System/AppCombination
IndividualUser
Resource Managementand SchedulingConsiderations
Core API
Interface Considerations
Systems Software
Application(s)
Resource Managementand SchedulingConsiderations
PolicyInterface
“What’s a utility function?”
“What is your utility function?” or“Which of these profiles are you most like?”
![Page 32: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/32.jpg)
32
Typical Design Models• Optimize User
Satisfaction Subject to Constraints
• Another option: Expose the system software to the user in its glory details– Works great for us!
Systems Software
Application(s)
Satisfaction with System/AppCombination
IndividualUser
Resource Managementand SchedulingConsiderations
Core API
Interface Considerations
Systems Software
Application(s)
Resource Managementand SchedulingConsiderations
PolicyInterface
“What the…”
![Page 33: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/33.jpg)
33
Typical Evaluation Approaches
• Workloads– User workload model/generator
• How to account for user variation?• How to evaluate as closed system?• How to validate?
– User traces• Context dependent• How to evaluate as closed system?
![Page 34: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/34.jpg)
34
Typical Evaluation Approaches
• Metrics– Can system meet performance objectives
given through policy interface?• What should the objectives be?
– Can system optimize over some combination of utility functions?
• What should the utility functions be?
![Page 35: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/35.jpg)
35
New Model for Characterization and Evaluation
• User studies to characterize user response– Examine the range of user satisfaction for
some perceivable quantity or combination of quantities
– Capture the variation, not only the mean– Variation = opportunity
• User studies for evaluating systems– Directly measure user satisfaction with
your system
![Page 36: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/36.jpg)
36
New Model: Direct User Feedback
• Optimize User Satisfaction Subject to Constraints
• User conveys satisfaction (or dissatisfaction) through a simple user interface
Systems Software
Application(s)
Satisfaction with System/AppCombination
IndividualUser
Resource Managementand SchedulingConsiderations
Core API
Interface Considerations
Systems Software
Application(s)
Resource Managementand SchedulingConsiderations
SatisfactionFeedback
![Page 37: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/37.jpg)
37
New Model: Direct User Feedback
• Optimize User Satisfaction Subject to Constraints
• User has some direct control over systems-level decision making through a simple interface
Systems Software
Application(s)
Satisfaction with System/AppCombination
IndividualUser
Resource Managementand SchedulingConsiderations
Core API
Interface Considerations
Systems Software
Application(s)
Resource Managementand SchedulingConsiderations
Some ControlOver DecisionMaking
![Page 38: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/38.jpg)
38
User-driven Control of Distributed Virtual Environments
• Virtuoso project (see virtuoso.cs.northwestern.edu)– User “rents” collection of virtual machines
• Virtuoso front-end looks like computer vendor– Providers stand up resources on which VMs can run
or communicate– Virtuoso provides adaptation mechanisms
• VM migration• Overlay topology and routing (VNet)• CPU reservations (VSched)• Network reservations (optical with VReserve)• Transparent network services (VTL)
– Virtuoso provides inference mechanisms• Application traffic and topology (VTTIF)• Network bandwidth and latency (Wren)
![Page 39: The User In Experimental Computer Systems Research Peter A. Dinda Gokhan Memik, Robert Dick Bin Lin, Arindam Mallik, Ashish Gupta, Sam Rossoff Department](https://reader038.vdocuments.site/reader038/viewer/2022103022/56649d375503460f94a0f277/html5/thumbnails/39.jpg)
39
User-driven Control of Distributed Virtual Environments
• Optimization problem: Given the inferred demands and supply, choose a configuration made possible by the adaptation mechanisms that maximizes a measure of application performance within constraints– Formalizations– NP-Hard problem in general– Approximation bound is not great either– Heuristic solutions
• Can the user or a system administrator solve these problems given the right interface?– Can a naïve human do it?