Handling an 1,800 Percent Traffic Spike During Super Bowl

XLVIJim HouskaEnterprise Architectjhouska@cars.com

David BeranPerformance Engineerdberan@cars.com

AgendaBusiness DriversLegacy APM ChallengesWhy dynaTrace?Our Secret to APM SuccessSuper Bowl XLVI2 Clicks to FixUEMWhat’s Next?

Confidence Comes Standard. Leading online automotive

destination Insight into all aspects of

automotive ownershipCar buyers are in controlTools, research and listings

The Cars.com mission: To make car buying and selling a great experience.

Cars.com - Fast FactsJVMs 600Dealer Customers 17,000Vehicle Listings 3,250,000Unique Visitors / Month 18,500,000Unique Images 40,000,000Page Views / Month 200,000,000Search Engine Queries / Month 785,000,000Consumer Activities / Month 5,400,000,000Images Served / Month 7,200,000,000 Kilobytes of storage 536,000,000,000

IT World / Yottaa Prediction…

Our Journey to APM Enlightenment

Business DriversDemanding more from ITAgile-centric transformationRapid GrowthExploding complexityDesire for insightImproved time-to-marketPoor performance = Lost $ or Job?

Application architectures are rapidly evolving.Is your APM solution keeping pace?

“Now that you have an overview of the system, we’re ready for a little more detail.”

http://En.wikipedia.org/wiki/File:ArchitectureCartoon.png

Legacy APM Challenges Lagging YEARS behindOverhead and InstabilityNeed to know “what to look for” Infrastructure component-centric Intensive care and feedingRigid configurationsSpecialized skill set

http://cheezburger.com/2689476608

Why Did We Choose dynaTrace? PurePath provides DEEP diagnostics Robust data visualization Business transaction-centric Multi-dimensional drill down Extremely low overhead

<1% overhead in production, even during an 1,800% traffic spike

Automatic application discovery & mapping Quick time-to-value Collaboration across lifecycle Powerful UEM

Our Secret to APM SuccessAPMThink differently

Insight, not informationThe end-user is King!From browser to backendAdvocate and empowerBecome predictiveBest of breed is OK

http://www.drawinghowtodraw.com/stepbystepdrawinglessons/wp-content/uploads/2011/01/400x400-ninja.png

dynaTrace Super Bowl Deployment

Cars.com Super Bowl XLVI Commercial

13

Woooo!

Case Study 1 – Super Bowl XLVI

ProblemStatement

BusinessImpact

•Ongoing intermittent performance issues •Expecting huge onslaught of traffic

•Millions $ marketing spend •111 Million Viewers•Need we say more?

Pre-Super Bowl Performance Issues

During periods of daily peaktraffic…

We were experiencing

nightly response time spikes

And discovered web container

thread saturation

Pre-Super Bowl Performance IssuesDaily ResponseTime Spikes

WebContainerSaturation

ExpensiveGarbageCollection

JDBC PoolSaturation

Search Application PurePath Example

http://apublicdefender.com/wp-content/uploads/2008/08/burglar.jpg

New Dual-Data Center Configuration

18

10 Search Accelerators

Database

VIPs

C1Search

FrontendServices

WS8WS10

WAN

Prod Web

ProdAccelerators

Stage Web

StageAccelerators

8 Search Accelerators

Database

C2Search

FrontendServices

C3Search

FrontendServices

C4Search

FrontendServices

WS7WS9

WS2WS4

WS1WS3

WS7WS9

WS8WS10

C1Search

FrontendServices

C2Search

FrontendServices

Staging Data Center Production Data Center

Cars.com Super Bowl War Room

David Beran “Photographer”Pizza Wings

Super BowlXLVI

dynaTrace Dashboards

Beer

Jim Houska

Performance During Super BowlData Center 1 Data Center 2

Two Data Centers on One Dashboard!

Super Bowl-Sized TestSe

ries1

Page

s pe

r sec

ond

1,800%Traffic Spike

Stable Performance

Commercial Airs

Case Study 2 – Lead Generation

ProblemStatement

BusinessImpact

•Agile release triggered a loss of email leads •No application errors were generated•Developers pulling their hair out

•SIY Consumers and 17,000 dealers •Potential revenue loss of $200,000

Production LeadPath Issue

Case Study 3 –POC of UEM

ProblemStatement

BusinessImpact

•Visibility into 3rd party calls or CDNs•Visibility into client-side performance•Lack of network time or bandwidth metrics

•Tolerating or Frustrated consumers may abandon Cars.com•Page views = $$$ This is our Bottom Line

POC of dynaTrace UEM

What’s Next?1. Make Fewer HTTP Requests2. Reduce the Number of DOM Elements3. Reduce DNS Lookups4. Add Expire Headers5. Configure Entity Tags (Etags)6. Put JavaScript at the Bottom7. Compress Components with GZIP8. Avoid URL Redirects9. Minify JavaScript and CSS10. Effective use of a CDN

Top 10 Targeted Client-Side Improvements for Cars.com

http://cheezburger.com/2689591040

What Else?DynaTrace 4.2 Upgrade Smart Monitoring, Workflow, & Detection Self Learning; Intelligent Auto-Baselining Zero-Configuration System Health Monitoring Faster Production Analytics – 100X Native Mobile UEM

Client / Browser-side Optimization POC of Google Page-Speed Top 10 Client-Side Improvements

IT World / Yottaa Prediction… “D’oh!”IT World & Yottaa said that Cars.com actually “…fared really well during the Super Bowl.”

“ ”

http://www.smtexas.net/faculty/jackson/CAPPS61011/CAPPS6T2I/Reddy/Webpage/D'oh.html

Transaction Flow Topology

Dynamic Measure Matrix

Dashboard Examples

“Fast / Good / Slow” Dashboard

Questions?

?