forget methodology. focus on what matters

forget methodologyfocus on what matters

Jack HumphreyKeep Austin AgileMarch 21, 2014

JACKHUMPHREY

About Me

● I’ve been…● a software developer for 19 years,● a manager for 10 of those years,● at companies including:

○ Trilogy (3 years)○ Coremetrics (3 years)○ Google (5 years)○ Indeed (5 years and counting)

About Me

● I’ve worked on products like:○ a web content management system,○ enterprise software development frameworks,○ a large online data collection platform,○ features for online advertisers, and○ a few large-scale consumer web applications

I help people get jobs.

About IndeedMore job searches worldwide than any other employment website● Over 100 million unique users ● Over 3 billion searches/month● Over 24 million jobs● 50+ countries, 28 languages

Jun-07 Jun-08 Jun-09 Jun-10 Jun-11 Jun-12 Jun-13

Unique Visitors / Month

focus on what matters…

focus on what matters…SUCCESS

successfor software products

build a solution

build a solutionto a real problem


that people want to use


that (enough) people want to use

● Planning (what to do)● Process (how you do it)

Build a solution to a real problem…

● Timing● Pricing● Competition● Promotion● Quality● Random chance● etc.

… that enough people want to use.

We identified the problem, now where do we start?

Build a solution to a real problem…

STEP 1define success

STEP 1build a great team

talent+

passion

talent+

passion+

opportunity

What is the overall mission?

Does everyone understand it and agree?


Does everyone understand it and agree?

Does project success line up with the mission?If not, then a) it’s not success, or b) wrong mission!


● Not a subjective determination

Success should be quantifiable


● Shipping is not success



● Shipping is not success

● Prove success with data


I help people get jobs.

Job Search helps people get jobs

...by helping them find relevant job postings.

Success for Indeed Job Search

Success: Unique Visitors?

Success: Job Searches?

success?

● Demonstrate success with data● Evaluate data objectively (unique visitors?)

Avoid Subjectivity

● You may need short-term approximations.

When Success is Hard To Measure

● Best metric for “help people get jobs”: hires

Success for Job Search, take 2


● What’s better than unique visitors & searches?repeat visitors, bounce rate, click-through rate



● What’s better than unique visitors & searches?repeat visitors, bounce rate, click-through rate

● What’s even better?applies (IndeedApply)


● Measurement should be repeatable

● Analysis should be:○ accurate: based on correct/complete data○ consistent: metrics don’t change meaning○ fast: seconds/minutes, not hours/days/weeks

Measure and Learn

Google Analytics

In-House Analysis Tools

● We use Google Analytics

● We build our own frameworks and tools○ logrepo: capturing what happens○ ramses, imhotep: detailed analysis in seconds○ Demo of these tools @IndeedEng on April 30th

http://engineering.indeed.com/talks

Measurement & Analysis Tools



Use whatever tools work for you, but make sure:● measurement is repeatable● analysis is accurate, consistent, fast

Measurement & Analysis Tools

STEP 1build a great team

STEP 3turn the crank

process can never replacetalent and passion

process won’t deliver asuccessful product

but the right processcan help

Repeatable and predictable mechanisms for:

How process can help

Repeatable and predictable mechanisms for:● Communication: what, who, how, when


Repeatable and predictable mechanisms for:● Communication: what, who, how, when● Priority: do things in the right order


Repeatable and predictable mechanisms for:● Communication: what, who, how, when● Priority: do things in the right order● Memory: what happened, what next


Repeatable and predictable mechanisms for:● Communication: what, who, how, when● Priority: do things in the right order● Memory: what happened, what next● Delivery: get it done as efficiently as possible


Repeatable and predictable mechanisms for:● Communication: what, who, how, when● Priority: do things in the right order● Memory: what happened, what next● Delivery: get it done as efficiently as possible● Quality: it should work intended


Repeatable and predictable mechanisms for:● Communication: what, who, how, when● Priority: do things in the right order● Memory: what happened, what next● Delivery: get it done as efficiently as possible● Quality: it should work intended● Risk: minimize the chance of bad things


“forget about methodology” is an exaggeration

How methodologies can help

Can be templates for improving:● communication● priority● delivery


Can be templates for improving:● communication● priority● delivery

...helping your team iterate on process


Can help you think about● What is working well for us?● What could be better?


Can help you think about● What is working well for us?● What could be better?

… and offer prescribed process changes


Recently, an experienced engineering manager described a process change that he loved:

An interesting anecdote

Recently, an experienced engineering manager described a process change that he loved:

“we stopped tracking bugs”

An interesting anecdote

Ask yourself, what techniques can help your team:

Focus on what matters.

Ask yourself, what techniques can help your team:● communicate


Ask yourself, what techniques can help your team:● communicate● prioritize


Ask yourself, what techniques can help your team:● communicate● prioritize● remember what has been done


Ask yourself, what techniques can help your team:● communicate● prioritize● remember what has been done● know what needs to be done


Ask yourself, what techniques can help your team:● communicate● prioritize● remember what has been done● know what needs to be done● build solutions that work, efficiently


Ask yourself, what techniques can help your team:● communicate● prioritize● remember what has been done● know what needs to be done● build solutions that work, efficiently● manage risk associated with changes


Does your process (or lack thereof)enable success or just get in the way?


Start small and iteratively build process.

...or start big and iteratively reduce/modify process.


the evolution of processat Indeed

Process can be summarized simply:GET STUFF DONE

Started Lean

Started Lean

Deliver

MeasureLearn

Started Lean

Quickly!

Deliver

MeasureLearn

Idea to implementation, as quickly as possible!● Software Architecture (Service-Oriented)● Tools, like automated translation● Risk Management with Rapid Iteration

Engineering Velocity

@IndeedEng Talk:http://go.indeed.com/velocity

http://go.indeed.com/velocity

http://go.indeed.com/velocity

Rapid Iteration… to production (multiple times a week)

Velocity and Delivery


● Don’t wait to fail: means no big failures



● Don’t wait to fail: means no big failures● Work that’s ready doesn’t have to wait



● Don’t wait to fail: means no big failures● Work that’s ready doesn’t have to wait● More satisfying for developers



● Don’t wait to fail: means no big failures● Work that’s ready doesn’t have to wait● More satisfying for developers● Less deadline pressure



● Don’t wait to fail: means no big failures● Work that’s ready doesn’t have to wait● More satisfying for developers● Less deadline pressure● Developers own quality


Velocity and Quality

● Encourages modular thinking


● Encourages modular thinking● Fewer integration issues


● Encourages modular thinking● Fewer integration issues● Fewer changes → faster, comprehensive QA

○ Release multiple times a week for 12 weeks, vs.○ Release once in 12 weeks


● Encourages modular thinking● Fewer integration issues● Fewer changes → faster, comprehensive QA

○ Release multiple times a week for 12 weeks, vs.○ Release once in 12 weeks

● Encourages automated testing


● Team was small, talented, passionate, focused.● They iterated quickly.● Job search was successful.

Success...

Process was extremely ad-hoc● not very consistent (e.g. bug tracking)● not very repeatable or predictable● wasn’t creating a memory for the future

...and Mess

Process was extremely ad-hoc● not very consistent (e.g. bug tracking)● not very repeatable or predictable● wasn’t creating a memory for the future

… and that was fine. For a while.

...and Mess

...and develop predictable, repeatable ways to:● Coordinate effort● Improve communication and transparency● Create a memory for the future● Manage risk● Ensure quality

Time to reflect on process...

Infrastructure for Process

Infrastructure for Process

...for everything!

Type: PRODUCT

Product Management in JIRA

Issue Type:Initiative

requirements

Project Management in JIRA

Type: PROJECT

Issue Type:New Feature

Issue Type:Bug

Issue Type:Improvement

Issue Type:Task

developer work

deployablecomponent

“Incorporates”

Service A

Service B

Feature

Improvement

ProductInitiative

Webapp XFeature

incorporates

“Depends Upon”

Service A

Service B

Feature

Improvement

ProductInitiative

Webapp XFeature depends

upon

Links are bidirectional

Service BImprovement

ProductInitiative

Webapp XFeature

is depended on by

is incorporated by

When will the feature be ready?

Webapp X

Feature

Fix Version: 2014-03-20

target release

When will the initiative complete?

Product

Initiative

Fix Version: March 2014

completion timeframe

Self-Documenting Workflow

● Workflow states and actions should suggest a clear meaning

● Custom workflows for different types of issues

Example: Project Issue Workflow

Pending Triage

Pending Acceptance

Accepted

Pending Verification

Pending Closure

Closed

Create

Triage

Accept

Resolve

Verify

Close

● Fix Version deployed to production→ issues are closed→ future code changes require new issues

● Every code commit references issue key

Closing the Memory Loop

Closing the Memory Loop

JASX-4643Type: New FeatureFix Version: 2011-04-21

is incorporated by

UPDOG-76Type: InitiativeFix Version: June 2011

iterate on process.it’s never done.

translation processproduction deploys

a/b testing

● Planning rapid international growth

Iteration: Translation Process


● Need a localization strategy that wouldn’t slow us down




● Must coordinate many translators




● Must coordinate many translators

● We modeled it in JIRA.


Custom Translation Issues

Project

Explosion

Translation(French)

Translation(Spanish)

Translation(Japanese)

incorporates

Translation(*)

“Explosion” Issue Workflow

Pending Triage

PendingQA Instructions

Pending Review

PendingIssue Explosion

Pending Translations


Create

Triage

Submit

Reviewed

Explode

Complete

Verify

…

“Explosion” Issue Workflow

Pending Triage

PendingQA Instructions

Pending Review

PendingIssue Explosion

Pending Translations


Create

Triage

Submit

Reviewed

Explode

Complete

Verify

…

?

Translation Issue Workflow

Pending Acceptance

Accepted


Pending Closure

Closed

Accept

Resolve

Verify

Close

1 issue per language

● Custom issue types and workflows○ clear communication○ smooth transitions

How JIRA helped Translation

● Custom issue types and workflows

● Automation → 1 issue per language


● Custom issue types and workflows

● Automation → 1 issue per language

● Subsequent iterations:○ machine translation○ translation service integration


Iteration: Production Deploys


50/week

● Require a lot of communication and coordination between roles:○ Create the release candidate deployable○ Get it into a testing environment○ Find the bugs○ Fix the bugs○ Declare it ready for production deploy○ Deploy it to production


● Require a lot of communication and coordination between roles

● Many people care about current deploy status


Deploy Issue Workflow

Pre-Deploy

QA Ready

In QA

QA Complete

Deploy Ready

In Production

Ready

In QA

QA Complete

Ready

Deploy

Create

● Clear communication, smooth hand-offs

● Transparency for all stakeholders

● Ability to look at trends in deploys

How JIRA helped Deploys

● Proctor: data-driven framework for A/B testing○ No full code deploy required for test updates○ Lighter-weight, but still requires a process for

verification and deployment

Iteration: A/B Testing

● Proctor: data-driven framework for A/B testing○ No full code deploy required for test updates○ Lighter-weight, but still requires a process for

verification and deployment


Iteration: A/B Testing

Managing Data-Driven A/B Tests

Experiments and “Pro(ctor)Tests”

Service AProTest

Product

Experiment

Webapp XProTest

incorporates

ProTest Issue Workflow

Pending Requirements

Pending Triage

Pending Acceptance

AcceptedSubmit

Triage

Accept

Resolve

Create

…

QA Ready


QA Ready

In QA

In QA

ProTest Issue Workflow (cont.)

QA Ready

In QA

Production Ready

In Production

Pending Closure

Closed

Deploy

Verify

Deploy

Verify

Close

Resolve

…

● Clear communication● Tool integration saves time● Manages risk through verification process

How JIRA helped A/B Testing

Indeed’s approach: Iterate. Iterate. Iterate.

Summary

Find the best process for your team’s success.Iterate on process as necessary.

Stay focused on success.

Focus on Success

good luck, and thanks!

Q & Afor more about Indeed Engineering:

engineering.indeed.com

for more about me:indeed.com/me/JackHumphrey

http://engineering.indeed.com

http://engineering.indeed.com

http://www.indeed.com/me/JackHumphrey

http://www.indeed.com/me/JackHumphrey

forget methodology. focus on what matters

Technology

success success

data success

rate success

indeedapply success

help people

right process

project success line

repeatable analysis