The problem withthe real worldSecuring user passwords fromweb developer's point of view

Michal Špaček

Real stories of a web developer and his colleagues improving security, not necessarily in chronological order. With added speaker notes.

The problem with1.Passwords2.Users3.The real world

I've borrowed the name from two Passwords13 Bergen talks about problem with passwords and problem with users and made the talk the last part of this unofficial trilogy. I'll talk about what we've changed, what worked flawlessly and what the users had some troubles with. Might sound like complaining towards users at times but let's call it a reality check instead.

I'll talk about Slevomat.cz, the biggest deal-of-the-day site in the Czech Republic with 1B CZK (50M USD) in revenues in 2013, 150 employees, 1M customers, 1.8M user accounts, powered by PHP and MySQL (Percona Server). Customers have credits on their accounts and “stored” credit cards, although credit card info is stored at the payment gateway. I joined back in September 2013, not as a security guy but as a web developer and have seen a lot of security issues since then so I've started fixing them. I left the company in June 2014.

8b8f05049b2114ad3d330db391e78549670d6b4b

One of the issues the site had was using unsalted SHA-1 for hashing user passwords. Although a great chance for a trip to users' minds, we couldn't justify attacking our users and didn't crack the passwords. Cracking passwords is job for people like Team hashcat :)

2000×lower(firstname)

400×firstname

Eventually, I've run some simple SQL queries, with unsalted SHA-1 it was just a matter of WHERE password = SHA1(…). Out of 1.8M, some 400 users had their password set to their firstname and some 2000 users had their password set to their firstname, lowercased.

500×lower(lastname)

100×lastname

For lastname, the original/lowercased ratio was the same but the total number of users with password set to their lastname, both original and lowercased, was four times less than the total number of users with password set to their firstname.

1400×ddmmyyyy

3×dd.mm.yyyy

Dot-separated dd.mm.yyyy format is a national date format used in Czech Republic and very few people had password set to their birthday in such format. 1400 people used ddmmyyyy format and only 30 people used yyyymmdd format of their birthday for their password.

foo@bar.org

3800× foo

10× bar.org

440× bar

For user e-mail addresses in local@domain.tld format, 3800 users out of 1.8M total had their password set to the local part of their e-mail, just 10 to domain.tld and 440 to domain part of their e-mail address.

bcrypt(sha1(…))

bcrypt(…)

This is what we've done with the password hashes. We've hashed all SHA-1 hashes with bcrypt at once (took 7 days) so that there are no weak SHA-1 hashes in the database even for the users who have not signed in for a long time. Once the user logs in or changes or resets their password we "upgrade" the hash to plain bcrypt with no intermediate SHA-1 hash. At that moment the user's password is known in the application so it's possible.

aes(bcrypt(sha1(…)))

aes(bcrypt(…))

Database breach or leak is quite likely nowadays so we've added AES-256-CBC encryption on top of bcrypt (or bcrypt+sha1 in case of older passwords) hashes so that people cracking bcrypt (hi Katja) would need to obtain the encryption key first and then still crack the hashes.

cHJvZDE=;GwEuCSSgBCLx0BHHAAb8IA==;kvmDV/DEJ3H+how7bErLa+/xapV2282MmISXqVo8xREfnBKcNmAixIaDDnYkODCu9KW3+6sbM04W7Tm3w1S+NQ==

The site uses different encryption key for live and dev environments. Support for key rotation has been introduced too and is also used for re-encryption of live employee credentials in dev environment. Displayed above is my current password as stored encrypted in live database with semicolon-separated Base64-encoded key id, initialization vector and encrypted hash.

bcrypt(token)

The site also hashes tokens for semi-permanent logins and tokens for password resets. These are as good as passwords for hijacking user accounts.

$dibi→select('name')→where("id = $_GET[id]");

Why database breaches or leaks are likely nowadays? Because people write code like this. One day I opened a random file and stumbled upon a source code similar to this. Someone worked around his lack of knowledge of SQL and introduced an SQL Injection vulnerability.

The other day we've found a directory on one of our servers full of database backups of other projects the admin was managing and were wondering where our dumps might end up at. The site also had some other serious vulnerabilities back in the day, a friend of mine not connected to the company told me. That's why the extra steps at securing passwords.

Support agents

Next big thing we've removed and I don't mean the agents. Customer support can no longer set user passwords in their tool. People forgetting passwords were calling in and asking the agents to reset their password manually. Such people are almost impossible to verify so we've removed the option completely.

Password reset links

Now, customer support agent can only send a password reset link to user's registered e-mail address. Removing the option to set the password manually has wreaked havoc. Some people were missing their e-mails with reset links. With some insights from Seznam.cz, the biggest mailbox provider in Czech Republic we've realized that the people were blocking our From sender because they simply couldn't or didn't know how to unsubscribe our newsletters.

Mark as spam?

The problem was not marking messages as spam, thanks to a thing called feedback loop, the problem was actual blacklisting. We've used same sender address for both mass-mailing and password resets, the idea behind one sender was that people would not block the mass-mailing sender address in order to receive transactional emails too. We were wrong. People thought they were blocking newsletters and blocked password resets too.

Unique senderper task

The issue was fixed easily by changing sender address for password resets and some other emails. Make sure you use different senders for different things.

Password reset links?Manual setting

One day users of Centrum.cz have stopped receiving all of our e-mails. Centrum.cz is the second-biggest mailbox provider in Czech Republic (after Seznam.cz) and they have just outsourced their antivirus and spam filtering technology and their constantly overloaded systems were delivering e-mails with up to 30 hours delay. Our password reset links were expiring in 15 minutes so we had to put fully logged and audited manual password setting back for Centrum.cz users. Everytime a password is changed the user is notified.

33% tokensexpired

Initially, the expiration time for password reset tokens was set to 15 minutes. We saw that a third of all tokens were coming back expired. No change after extending the expiration time to 30 minutes, still a third of the tokens coming back expired. People either set their password immediately when they request the reset link or do it often as late as next day.

The site www.slevomat.cz is now accessible only via HTTPS with Perfect Forward Secrecy, AES-GCM, HTTP Strict Transport Security header and disabled SSLv3 and older. All passwords and session ids are now transferred securely. The switch to HTTPS took a while, had to switch servers to nginx instead of Apache for better TLS optimizations.

cpacpa

This was my CEO's password. Not exactly this but the pattern was very similar. How do I know? I was looking at log tables in the database the other day and found plaintext passwords of all employees. These ended up logged due to a strange and not that obvious behavior in the logging code. Look closely and you'll find a lot of security WTF-moments in the code.

Password×

Token

We've reset passwords for all employees and they had to use the password reset feature to set their new password. We didn't delete permanent login tokens so that people could use the site to do their things. I got a lot of e-mails saying that the old password still worked for signing in but people were not signing in using their password but using the permanent login cookie. Some users don't really understand the concept behind permanent login and difference between cookie-based login and password-based login.

The problems1.External parties2.Users3.Companies

There are more problems with real world than just one. When implementing security best practices one depends on third parties and they sometimes fail. Some users don't understand several key concepts of web applications nowadays. Stuff's not really secure and companies couldn't care less. That all is old news I guess. Still worth repeating, though.

Michal Špačekwww.michalspacek.cz

@spazef0rze