Making More Happen – How to Measure Revenue in Milliseconds

Cliff Crocker@cliffcrockerSOASTAVP Product, mPulse

Performance is a Business Problem

Funnel Optimization is Everything

• Merchandizing• SEO• Multivariate testing• Pricing• Ad Campaigns & targeting• User Experience

Performance IS User Experience!!

Yahoo! - 2008Slower pages cause increase in abandonment

Increase of 400ms causes 5-9% increase in user abandonment

http://www.slideshare.net/stubbornella/designing-fast-websites-presentation

Shopzilla - 2009Faster pages increase site conversion rate

A reduction in Page Load time of 5s increased site conversion 7-12%!

http://assets.en.oreilly.com/1/event/29/Shopzilla%27s%20Site%20Redo%20-%20You%20Get%20What%20You%20Measure%20Presentation.ppt

Walmart - 2012Impact of slow pages on conversion rate

http://minus.com/msM8y8nyh#1e

SF WebPerf – 2012Up to 2% conversion increase for every second

SIMULATION

Unfortunately, it’s not always that straightforward

Not All Pages are Created EqualTop of funnel impact (Browse Pages)

For a typical eCommerce site, conversion rate drops by up to

50% when “browse” pages increase from 1

to 6 seconds

Not All Pages are Created EqualBottom of funnel impact (Checkout pages)

However, there is much less

impact to conversion

when “checkout”

pages degrade

Getting StartedThe challenge of change

• Organizational challenges• Process challenges• Visibility• Expertise• Focus & prioritization

Steve SkroceStaplesSr. Manager, Site Operationssteven.skroce@staples.com

How Did I Get Here

Developer SEO Performance

SiteSpeed & StaplesProcess change

• A top goal for Staples eCommerce• It’s about customer experience

• Speed is essential to UX• Because Staples is highly matrixed

• Visibility into the organization is key• Approach as a discipline and institutionalize the process

First Things FirstEstablishing a framework for SiteSpeed

• You can’t manage what you can’t measure• Earliest work involved selecting the proper tools

• Business and Engineering led team• Already had synthetic transaction monitoring tool• Backend tools were also available• RUM tool was missing from the set

• Needed RUM info if we were going to affect our customers’ experiences

First Things FirstStakeholders

• Assembling a cross-functional team• Business

• Decides which features and functionality lives on the site• Engineering

• How something gets built can make or break page load time• Marketing

• Asset optimization is important• Merchandising

• On-site content• Analytics

• Analytics team ties everything together• And others…

Why Is Speed Important to Staples?Customer experience

Bounce Rate increases as load time increases

Why Is Speed Important to Staples?Sales

Conversion rate decreases as load time increases

The First PassIdentify business functionality

• Governance meetings to evaluate functionality• Cataloged all site features• Weekly meeting to discuss one or two features in detail

• Feature owner comes prepared to review all aspects

• Full Page Load vs. Time to Interactive• How Staples defines them• FPL ensured wide exposure / awareness through the organization

• Everyone was involved from the beginning• Set the expectation of accountability

The First PassOptimizing for speed

• Onsite promotional banners were too “heavy”• Inconsistent image optimization

• Depended on the individual, not the process• 25k and 250k

• Put a new process in place

The First PassA/B Testing

• Testing practices were inefficient• Two vehicles for A/B testing• Eliminate one

• Process improvement opportunity for ending campaigns• Eliminate lingering artifacts

The First PassFocus on third-parties

• Needed “sanity” around third party calls• Categorization of calls

• Ads• Marketing• Analytics• Recommendations• Social Buttons• Etc.

• Removed some calls, optimized others• Further work needs to be done

The First PassLooking at the back-end

• Database calls were inefficient• Some pages went from 200 calls to 10

EvolutionWhat we’ve learned

• Go bolder, sooner• Sprint model allowed us to deliver something every

release• Something, not something substantial• Not enough time for big, bold changes

• We needed to step-back and reevaluate

How fast are we, really?Getting situational awareness

• Examples of pagespeed for Staples• Median vs. percentiles

• Make it simple for a wide audience to understand the metrics• Key page performance – target specific pages to focus on

• Home, Product, Search, Class, Login, Cart

Make Faster Page Load Times HappenHow do we get there?

• mPulse• WebPagetest.org• Dev Tools• NetPanel

• “Motivational” emails from the top

Make ‘Faster Page Load Times’ HappenHow do we get there?

• Confirmation of what we already knew:• Front-end vs. back-end time

X

JS refactoring Reduced k size by 48% Reduced lines of code by 52%

CSS refactoring Reduced k size by 83% Reduced lines of code by 88%

Early ResultsHard work pays off

Early ResultsHard work pays off

• Shaved 1 full second off of Home page median• Shaved 6 seconds off of the 98th%• Conversion improved by roughly 10%