DON’T FEAR FAILUREA TALK ABOUT SOME OF MY MANY FAILURES

SOME THINGS ABOUT ME

• Most people call me Stabby (@StabbyCutyou)• I’ve never started my own company• I’ve never written any popular or even recognizable software• I am definitely not good at “Computer Science”• I almost dropped a live database once

KEY TAKEAWAYS FOR THIS TALK

• I am a huge failure• Just have a plan• Own it• It’s OK to fail

FAILURE“LACK OF SUCCESS”

Set out to do a thing

Did it work?YES NO

FAILURE LEVEL:

EXPERT

OVER 12 YEARS OF MY LIFE

• .NET, Javascript, Ruby, SQL, Go• Love what I do• Screw up tons of stuff in all the time• Manage to get enough right to get by

LESSONS TO LEARN

FROM FAILING

LESSON ONE:

ITS OK IF YOU’RE NOT A SUCCESS STORY

OUR CAREERS ARE DEFINED BY “SUCCESS”

• Large valuations• Perks• Bonuses• Exits• Renown

Most of won’t have that kind of success

I THOUGHT MY FIRST JOB WAS GOING TO BE “IT”

• Guess how that went• One day got an email my company sponsored gym

membership was cancelled• Next day, company was closing down• Had to tell a recent hire who just started two weeks ago that

no, this was not a joke.

I LEARNED A LOT ON THE WAY DOWN

• Built a solid set of SQL skills• Learned a lot of ways you probably shouldn’t build a desktop

app• Exposed to running a dreadful meeting every week• First encounter with bizarre interoffice politics / relationships

LESSON TWO:

NO MATTER HOW HARD YOU WORKYOU CAN’T ALWAYS STOP FAILURE

FOR EXAMPLE… AWS

• AWS is great• Network can be unreliable• Resiliency / Durability in your stack only goes so far• Partitions happen

ANOTHER JOB

• Built the worlds worst Foodler clone• Amongst the worst times in my life• Barely slept, constant rewrites, no guidance or design• It had no chance in the market

There was no way to save it

It was simply destined to fail

I beat myself up over it for a long time

LESSON THREE:

ASK FOR HELP

IT’S OK TO NOT KNOW WHAT YOU’RE DOING

• No one really does at first• Many of us still don’t• Find someone who does

FOR EXAMPLE… RIAK

• Built around the idea of replication• Writes get copied to 3 nodes (by default)• Highly Available - data always accessible• Conflict resolution: Siblings, Vector Clocks

WE WERE CONVERTING TO A NEW JOB SYSTEM

• It kept hanging• I would wake up every few hours to check it• Worked with other engineers to solve the issue• Figured out how to predict it, put in mitigations• Realized no-one cared so long as it got fixed

LESSON FOUR:

OWN YOUR MISTAKES

FOR EXAMPLE… GO

• Writing a lot of Go• Lots of gripes• Fell in love with error handling in Go• Responsibility strictly delegated upward until you “care”

WE TRIED TO BUILD OUR OWN QUEUE

• Sure was fun• But did not succeed like we hoped• Learned a lot of great information• But ultimately wasted a lot of time and effort

LESSON FIVE (a):

IT’S OK TO FAIL…

LESSON FIVE (b):

…SO LONG AS YOU HAVE A PLAN

FOR EXAMPLE… TCP

• Built around the idea of missing data• User land VS Using your brain• Defends against problems in-depth

I MOVED A BUNCH OF FINANCIAL DATA

• This was a big one• We moved some finance data into a more scalable system• During the migration some data was initially lost• We didn’t catch it• We paid out accounts based on incorrect data

WELL

I’M DEFINITELY FIRED

BACK TO MY MAJOR SCREWUP…

• Multiple copies of the missing data• We built scripts to compare datasets and find missing records• We repopulated the missing data• We worked with finance to fix the numbers• Did not get fired

CULMINATION OF ALL LESSONS LEARNED

• We didn’t succeed like we hoped• We had multiple layers to verify, but not enough• We had a plan incase this happened• Asked for help from the right folks• Stepped up, personally accepted responsibility immediately

Set out to do a thing

Did it work?YES NO Did you learn?

YES

NOTry Again

IN THREE DAYS

I START A NEW JOB

LESSONS

• Not everyone is a success story• Failure is inevitable• Ask for help• Own your failures• It’s ok to fail, just have a plan for it

THANKS

@StabbyCutyou