user studies ii

User Studies II

With your instructor, Jeremy Hyland

Plan for Today

• Discuss the reading:– Why Johnny Can’t Encrypt– Johnny 2: Judgment Day

• Do a little testing of our own…

Why Johnny Can’t Encrypt

• Who’s Johnny and why can’t he encrypt?

Posner saysWhat’s Johnny trying to hide?


• Whitten and Tygar, 1999– http://www.usenix.org/publications/library/proceedings/sec99/full_papers/whitten/whitten_html/index.

html

• A Usability Evaluation of PGP 5.0

http://www.usenix.org/publications/library/proceedings/sec99/full_papers/whitten/whitten_html/index.html

http://www.usenix.org/publications/library/proceedings/sec99/full_papers/whitten/whitten_html/index.html

Why Johnny Can’t Encrypt• “Security mechanisms are only effective when used correctly”

So:

If Usable then

else


• Defining Usable Security Software– Whitten and Tygar:

Security software is usable if the people who are expected to use it:

1. are reliably made aware of the security tasks they need to perform.

2. are able to figure out how to successfully perform those tasks

3. don't make dangerous errors

4. are sufficiently comfortable with the interface to continue using it.


• Why is usable security hard?

McNealy saysYou have no usable security, get over

it.


• Why is usable security hard?– Five reasons:

1. The unmotivated users• “Security is usually a secondary goal”

2. Policy Abstraction• Programmers understand the representation but

normal users have no background knowledge.

Why Johnny Can’t Encrypt• Why is usable security hard?

– Five reasons:

3. The lack of feedback• We can’t predict every situation.

4. The proverbial “barn door”• Need to focus on error prevention.

5. The weakest link• Attacker only needs to find one vulnerability


• Usability Evaluation– PGP 5.0

• Pretty Good Privacy• Software for encrypting and signing data• Plug-in provides “easy” use with email clients• Modern GUI, well designed by most standards


• Usability Evaluation– Whitten and Tygar focus their evaluation

on a question based off their definition of usable secure software:

If an average user of email feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all?

Loaded question?


• Usability Evaluation– Cognitive walk through

• Mentally step through the software as if we were a new user. Attempt to identify the usability pitfalls.

• Focus on interface learnablity.


• Usability Evaluation– Cognitive walk through results:

Visual metaphors– Public vs. Private keys– Signatures and verification



Different key types– Compatibility increases complexity– Keys listed as users

Why Johnny Can’t EncryptKeys listed as users



Key server– Hidden?– What is it doing?– Revocation not automatic

Would that help?



Key management policy– Unneeded confusion– What’s the difference between trust and validity?



Irreversible actions– Need to prevent costly errors

Consistency– “Encoding”?!?

Too much information– More unneeded confusion– Show the basic information, make more advanced information

available only when needed.


• Usability Evaluation– User Test

• PGP 5.0 with Eudora

• 12 participants all with at least some college and none with advanced knowledge of encryption

• Participants were given a scenario with tasks to complete within 90 min

• Tasks built on each other

• Participants could ask some questions through email

Why Johnny Can’t Encrypt• Usability Evaluation

– User Test Results:• 3 users accidentally sent the message in clear text

• 7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problem

• Only 2 users were able to decrypt without problems

• Only 1 user figured out how to deal with RSA keys correctly.

• A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted emails.

• One user was not able to encrypt at all


• ConclusionIf an average user of email feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all?

→Nope

– Is this a failure in the design of the PGP 5.0 interface or is it a function of the problem of traditional usable design vs. design for usable secure systems?

• Security as the primary function vs. a secondary function

Johnny 2

• Garfinkel and Miller, 2005– http://www.simson.net/clips/academic/2005.SOUPS.johnny2.pdf

• Follow-up to Why “Johnny Can’t encrypt”• Test of new encryption technology

– Key Continuity Management– S/MIME certificates

• Better interface– Simple buttons

http://www.simson.net/clips/academic/2005.SOUPS.johnny2.pdf

Johnny 2

• Garfinkel and Miller:– Johnny couldn’t encrypt because of the key

architecture behind PGP.“….the fundamental usability barriers

that Whitten identified could be overcome by replacing the underlying third-party certification model with Key Continuity Management.”

Johnny 2

• User Test– Tried to stay as close to the Johnny

experiment as practical• Same methods of user solicitation/selection• Same basic scenario• Similar user tasks

– Added attackers

Johnny 2

• User Test– Attacks:

• new key attack• new identity attack• unsigned message attack

– How well does the interface enable users to respond to these attacks?

Johnny 2

• User Test– Test application: CoPilot

• “Wizard of Oz” prototype• S/MIME certificate handling:

– First time = Yellow– Trusted certificate = Green– Changed certificate = Red– Unsigned message = White– Unsigned message from a sender that normal sends

signed messages = Gray• Better tools allow for a more automated and

scientific test

Johnny 2

• User Test– 43 test subjects– Three groups:

No KCMColorColor+Briefing

Johnny 2• User Test

– Results:• Users generally understood the basics

• Little understanding of signature integrity guarantees

• Verifying attack message authenticity was difficult for most users

• No group resisted attacks 100% of the time

• Color and Color+Briefing resisted new key attack and the unsigned message attack better then No KCM

• The interface did not help against new identity attacks

Johnny 2

• User Test: Conclusions– A few surface interface issues

– Do not trust button

– Misconceptions about the security of sealed messages

– Generally, the new interface simplifies email encryption

– Still problems with determining certificate trust, however some of these problems may be unavoidable.

So Now What?

• Now its time to do your own test!

User Test

• 3 groups:– Cell Phone– CD player– Calculator

• Take a few minutes to create a simple user test

• One member of each group switches to be a tester

User Test

• Guidance:– Decide whose going to do what!– Create a Use Case Scenario– Define user tasks for completion of the

scenario– Set up metrics for results evaluation

– What qualifies as success vs. failure?

User Test

• Results!?

user studies ii

Documents