good enough performance

How Fast is Fast Enough?How Fast is Fast Enough?

© Copyright Performize-IT LTD.

Haim Yadid Performance expert, Performize-IT

Performize ITPerformize IT

© Copyright Performize-IT LTD.How fast is fast enough?

Services

Consulting for R&D organizations• Problem analysis• product analysis

• Training• Java Performance Optimization


Good Performance


Why

?What ?

How?

How fast is fast enough?

““Fast” is More than a FeatureFast” is More than a Feature


Speed is

© Copyright Performize-IT LTD

First and foremost, we believe that speed is more than a feature. Speed is the most

important feature. If your application is slow, people won’t use it.

Fred Wilson: Top tech VC from NYC with investments in companies such as Twitter, del.icio.us, Etsy, and FeedBurner.


Revenue Drop

Bing – A page that was 2 seconds slower resulted in a 4.3% drop in revenue/user.

• Google – A 400 millisecond delay caused a 0.59% drop in searches/user.

• Yahoo! – A 400 milliseconds slowdown resulted in a 5-9% drop in full-page traffic.

© Copyright Performize-IT LTD

Taken from Steve Sounders blog


Effect of improving performance

• Shopzilla – Speeding up their site by 5 seconds increased the conversion rate 7-12%, doubled the number of sessions from search engine marketing, and cut the number of required servers in half.

• Mozilla – Shaving 2.2 seconds off their landing pages increased download conversions by 15.4%, which they estimate will result in 60 million more Firefox downloads per year.

• Netflix – Adopting a single optimization, gzip compression, resulted in a 13-25% speedup and cut their outbound network traffic by 50%.

© Copyright Performize-IT LTDHow fast is fast enough?

Want to Ruin Your Reputation ?


So……

WhatIs

GoodPerformance?


What is Bad Performance ?

When user experience is bad• When customers are complaining• When application cannot scale• When applications crash• When applications become non responsive


What is good performance

When customers are happy about performance • Customers are complaining on other things• Application is responsive• Application is stable• Application scales as needed


Good Enough Performance

Performance behave according to requirements• Not the best performance Good enough

© Copyright Performize-IT LTDHow fast is fast enoughHow fast is fast enough?

Performance AssurancePerformance AssuranceActivities Activities


Analysis Design Development StabilizationHow fast is fast enough?

Example: Collaboration App.

A Google sites like collaboration application• Registered user will have access to a document repository (documents, presentations, spreadsheets)

• CRUD on all documents• Editing content online from the browser• Conversion from and to Microsoft office documents


Capacity planning

The important dimensions for your application• Expected value for each dimension.• Design capacity for all system and for a single computing unit.


AnalysisHow fast is fast enough?

Example: Capacity planning


AreaArea Sub areaSub area Capacity Capacity

Users Number of users 1M

Logged in users 60K

Documents Max #docs per user 100

Total number of docs 10M

Avg doc size 500KB

Max doc size 10MB

# import & exports/ min 50

Workload Requests / sec 2000


Define Major Scenarios

What are the application most common scenarios?• What is the correct distribution of user scenarios• Based on real statistics • In not possible… your best guestimate• Refine your scenarios along product versions



Example: Major Scenarios

Scenarios for a regular user

Scenario nameScenario name DistributionDistribution DescriptionDescription

Login 10%

Create new doc 15%

Update document 50%

Delete document 5%

Share /Publish document

10%

Rename document

10%



Performance KPIs

Your performance contract• What is the maximum your user can tolerate for operations

• Usually (can vary between organizations) • 1 second for web interactive operations• 200ms for ajax operations • 5 secs for async operations • Several minutes for heavy data mining operations



Tolerance to regression

When you are developing new version• Adding new functionality• Changing architecture• Define what you expect to be the regression/ improvement from last version



Sizing

The amount of resources required to support the capacity planning

• Number of machines (Web servers, DB etc…)• Memory for machine• Amount of Disk space• Expected DB growth per day


DesignHow fast is fast enough?

Example: Sizing

Machine typeMachine type ParameterParameter ValueValue

Web container CPU Cores 8

Memory 4GB

Disk 40GB

DB server CPU Cores 16

Memory 16GB

Disk 4TB raid 5



System Safety Valves

• AKA Fuses• Protect application from crossing the capacity limit• Crucial to

• Keep system up and running• prevent customer misuse• prevent DoS attacks

• Example:• Do not enable to create than 10M more users• Do not let the 60001 user to login• Do not permit to create more then 10M document



UI related Safety Valves

Give bounds to certain UI behaviors• Crossing the bounds may have no sense or damage user experience

• Number of documents per folder may not exceed 100

• Number of chars per line of document may not exceed 2000



Risk Assessment

For new feature you must assess performance risk• High risk features require careful analysis• High risk features should be handled with care

• POC• Extensive testing

• How does a new feature affects the capacity planning



Example: Risk Assessment

FeatureFeature RiskRisk Measures to takeMeasures to take

Add tooltip to UI toolbar Low

Add indexing for all documents

HighPOC at early

stages



POC

For features with high performance risk • Test performance effect as soon as possible• Even before all functionality is done• Capture the risk nature with minimal implementation


Design DevelopmentHow fast is fast enough?

Automation

Development phase is the time for automation• Not later• Ability to populate the system• Reliable simulators• Load generation scripts• Automation should be • Fast • Reliable• Easy to use


DevelopmentHow fast is fast enough?

Code review

Delicate code areas should be reviewed • Concurrency should be handled with care• Premature optimisation should be avoided


DevelopmentHow fast is fast enough?

Component Tests

Test only one scenario/ component• Easier to isolate• May behave differently than the system test• Covering all components one by one may take long time


Development StabilizationHow fast is fast enough?

System Tests

Tests the system as whole• Configured according to capacity• Hard to isolate problem


StabilizationHow fast is fast enough?

The Performance Lab

A system which simulates real application• Simulate data feeds and virtual users



Populate your System

• Use automation modules to load your system with data • Create users• Create documents• Create document sharing

• Populate the system to its full capacity planning with a reasonable time



Challenge Your Backend

Mechanisms which do not require user interaction• Develop simulators • Run simulators according to capacity planning



Virtual Users

Running a performance test on real users is not practical

• Use a tool to • Record• Parameterize • Replay with large numbers of users



Load Tests

Running your application with accordance to your capacity planning

• This test when passes will give a good indication that your product will hold in production

• (Assuming that your capacity planning is accurate)



Stress Tests

Run your system under a massive load• 2 times or 3 times the capacity • May reveal concurrency problems such as deadlocks

• May show how system behaves under DoS attacks• Do not expect that system will behave correctly under those circumstances



Endurance Tests

Run the system under load for a long time• Measure application stability• Unveil resource leaks (memory threads etc.)• Deadlocks and other concurrency problems



Single User Test

Measure end to end performance • Include rendering time and network time• Single Users test should run under load



ApproachApproach


Performance Testing Dilemma

• Should start as soon as possible May have huge impact on code and design

• Cannot start too earlyConducting performance problem on a non

feature complete system is a waste of time


Tip: Give High Priority

Performance problem should get a high priority from the R&D

• They tend to be complex• They tend to be nasty• They tend to insert uncertainty • It is always simpler to fix the color of the button!!!


The Button will wait


Allocate the best resources

Your best engineers• Your best hardware


Questions ?


good enough performance

Technology