authentication melee a usability analysis of seven web authentication systems scott ruoti, brent...

AUTHENTICATION MELEEA Usability Analysis of Seven Web Authentication Systems

Scott Ruoti, Brent Roberts, Kent Seamons

Internet Security Research Lab

Brigham Young University

http://isrl.byu.edu

World Wide Web Conference 2015, Florence, Italy

Acknowledge co-authors

Scott Ruoti Brent Roberts

Passwords Rule the Web• Deployed everywhere• Well-known security problems• Many proposed systems to replace them• Passwords have combination of usability, deployability, and security that is

hard to beat• Bonneau et al. The Quest to Replace Passwords, IEEE Security & Privacy 2012

May 20, 2015 WWW 2015, Florence, Italy 3

Our Research Agenda• Develop Single Sign-on Protocols using Secure Remote Password (SRP)• Analyze security and usability of our system• We wanted to leverage experience from prior work in usable authentication

• What are the most usable authentication systems?• How to measure the usability of an authentication system?


Where Are We?

• Looked at systems proposed in research literature

• No clear best system• Lack of empirical analysis


Where Are We?

• Looked at systems proposed in research literature

• No clear best system• Lack of empirical analysis• Limitations

• Proposals are not evaluated using standard usability metrics

• Proposals are not compared against each other


Security vs. Usability

Big Dog vs. little dog


Where Do We Want to Be?• Elevate usability on an equal footing with security

• Truly secure systems must be both secure and usable

• Determine which proposals have the best overall usability• Use a standard metric• Head-to-head comparison of proposals• Identify best in class systems• Establish a basis for evaluating new proposals

• New proposals should not receive serious attention until they demonstrate acceptable usability• Security researchers can be poor predictors of usability


Authentication Melee• Conducted empirical analysis of seven web authentication systems

• Federated single sign-on: Google OAuth 2.0, Facebook Connect, Mozilla Persona• Email-based: SAW, Hatchet• QR Code-based: WebTicket, Snap2Pass

• Used the System Usability Scale (SUS) as a standard usability metric• Organized systems into head-to-head competitions


Tournament Structure• Difficult to do a full combinatorial study

• If each participant tests two systems, it requires a large number of participants• If each participant tests all systems, it can lead to study fatigue

• Instead we structured our study into a tournament• First round based on type of authentication system


Federated Single Sign-on• Authentication is centralized into a single identifying party• The website relies on the identifying party to verify the identity of users• Systems

• Google OAuth 2.0• Widespread

• Facebook Connect• Widespread

• Mozilla Persona• Identifying party only handles authentication


Email-based• Single sign-on where all email providers are identity providers• Users verify their identity by demonstrating their ability to send or receive

email• Systems

• SAW• Click on a link sent in an email message

• Hatchet• Enter a code sent in an email message


QR Code-based

• Encodes authentication credentials into a QR code• Two recent systems

• WebTicket• Snap2Pass


QR Code-based

• Encodes authentication credentials into a QR code

• Two recent systems• WebTicket

• Credentials encoded into a token that is printed out

• Token is shown to the website to authenticate the user


QR Code-based

• Encodes authentication credentials into a QR code

• Two recent systems• WebTicket

• Credentials encoded into a token that is printed out

• Token is shown to the website to authenticate the user

• Snap2Pass• The user’s phone acts as the identity

provider• The website sends information to the

phone through QR codes


Methodology• Four studies in total

• Federated single sign-on: 24 participants• Email-based: 18 participants• QR code-based: 25 participants• Championship round: 30 participants

• Participants were from BYU• Most were undergraduates• Most were between 18 – 24 years old• On average rated themselves as having intermediate technical skills


Study design• Built two websites

• Forum website• Bank website

• Implemented the seven authentication systems• Existing implementations unavailable• Consistent look and functionality

• Six tasks• 2 registration tasks• 4 authentication tasks• Repeated same tasks for each system tested

• Questionnaire• After each system• After study as a whole


System Usability Scale• Single numeric score between 0 and 100 (higher is better)• Calculated based on user responses to 10 Likert-scale questions• Individual participants’ SUS scores are averaged to give the overall SUS score


What Does the SUS Score Mean?• If a system has a SUS score of 75, what does that mean?• Bangor et al. examined SUS in over 200 usability studies and developed an

adjective-oriented interpretation of a SUS score


Results: Federated Single Sign-on• Winner: three way tie

• SUS scores between 71 and 72• Good

AcceptableC grade

• Chose Google OAuth 2.0 to advance

• Mozilla Persona took longer to authenticate• Difference was not mentioned in participants

qualitative responses

• Trust issues with Google OAuth 2.0 and Facebook Connect


System SUS Score

Google 72.0

Facebook 71.4

Mozilla 71.8

Results: Email-based• Winner: SAW • Both systems performed poorly• SAW

• OKLow-marginal acceptabilityD grade

• Participants disliked checking their email

• Hatchet• OK

Low-marginal acceptabilityF grade

• Users don’t want to leave their browser


System SUS Score

SAW 61.0

Hatchet 53.5

Results: QR Code-based• Winner: Snap2Pass • WebTicket

• OKLow-marginal acceptabilityD grade

• Snap2Pass• Good

AcceptableB grade


System SUS Score

WebTicket 57.9

Snap2Pass 75.7

Championship Round• Participants: Google OAuth 2.0, SAW,

Snap2Pass• Google OAuth 2.0 and Snap2Pass tie• SUS scores consistent with earlier scores• Overall winners:

• Federated single sign-on• Snap2Pass


System SUS Score

Google 75.0

SAW 53.2

Snap2Pass 68.4

Championship Round• Participants were asked how the systems

compared to each other and to passwords


System Usability Scale• Repeatable results - consistent SUS scores between studies• Good predictor of overall preference

• More accurate than mean time to authenticate

• Recommendation: All new proposals be evaluated using SUS• New system proposals should not be seriously considered until they receive a score of at

least 70


Qualitative Feedback• Users provided feedback via open-ended survey questions and in-person

interviews• The results provide interesting user perspectives on authentication


Transparency• In usable security, transparency refers to hiding security details• Transparency increases usability• Tested this by modifying SAW to automate token retrieval

• Used participants from second usability study (email-based)• Increased SUS score by 12.1 points• Statistically significant difference (p=0.01)


Transparency• Transparency can result in a lack of trust

• Similar phenomenon in our secure email research• Confused Johnny: When Automatic Encryption Leads to Confusion and Mistakes

Ruoti et al., SOUPS 2013

“I would like to understand more about how it works up-front. It doesn't feel secure.”

“I think it was very straightforward to use. Once again like with the other system, perhaps an explanation of how it protected information would give me more confidence in using it.”


Single Sign-on Protocols• Users liked the speed and convenience• Users recognized the risk of putting all their eggs in one basket

• Suggested augmenting SSO with low-entropy passwords at the website• Adds security if identity provider account is compromised


Single Sign-on Protocols• Reputation of the identity provider is important

• Desire dedicated identity providers

“I would be worried about security. I've heard that Facebook is ‘relatively’ easy to hack. I would want to be sure that it was all secure before I started using it.”

“I trust Google with my passwords.”

“I would make an account separate from my social network and mail specifically for functions like banking etc.”


The Coolness Factor• Participants were most willing to adopt systems that they described as “cool”

“Man was that cool!”

“Also, the feel of it made me enjoy doing it. I felt technologically literate and the app felt futuristic as a whole, which I enjoyed.”


Biometrics• We did not test or mention biometrics in our study• Users consistently mentioned them as being a “cool” way to authenticate

• Indication that users may be accepting of viable biometric solutions

“retinal scanner so i just sit in front of my computer and it scans my eye. dope.”

“The ideal system would scan some part of my body - either eye or thumb - because these are literally ALWAYS with me.”


Conclusion• We tested seven web authentication systems

• Found federated single sign-on and Snap2Pass to be the most usable• First empirical analysis of a heterogeneous collection of authentication proposals

• System Usability Scale• SUS is a good measure of usability for authentication proposals• Repeatable results that allow for comparing heterogeneous systems• Recommend it be used for all new authentication proposals• Minimum score of 70 for serious consideration

• Future work• Exploring the tradeoffs of transparency in authentication• Low-entropy passwords with single sign-on• Biometric-based web authentication


Questions?


Contact: [email protected]

authentication melee a usability analysis of seven web authentication systems scott ruoti, brent...

Documents