testing strategies to deliver consistent app performance

www.neotys.com© 2015 Neotys. All Rights Reserved.

30th of June

Henrik Rexed Manjunath PremkumarTechnical Evangelist

NeotysPerformance Test Architect Symphony Teleca- HARMAN

NO MORE GAMBLING WITH

APPLICATION PERFORMANCE DURING GRAND NATIONAL

& SEASONAL SPIKES


Agenda

Case study

Recommended testing approach

Betting industry has to use latest technology to ensure good UX

Performance Matters

Know your Customer

Major Betting Operators


Meet the most connected customer

Retail Shops

FitBit

iWatch

Narrative Camera

Samsung Gear

Smart TV

Phones and Tablets

Gaming Console

…and customers will be as connected in the next 2 years


What they expect from Betting Operators

“I WANT TO PLACE MY BETS ON ANY GAME/SPORTS FROM ANYWHERE,

ANYTIME”

“I WANT ACCESS TO FUNDS”“I WANT TO GET THE SAME GREAT

GAMING/SPORTS EXPERIENCE ACROSS ALL MY DEVICES”

“I WANT YOU TO GIVE ME SUGGESTIONS AND OFFERS THAT ARE RELEVANT TO ME”

“I WANT YOU TO SEND ME REAL-TIME ALERTS”

“I WANT YOU TO ENSURE SECURITY OF MY DATA”


Agenda

Case study



Performance Matters

Know your Customer



Performance to End User

What is PERFORMANCE to End User?“Hassle free application with rich contents and quick response time at finger tip for user to perform actions like Login, Signup, Bet placement, check wallet balances, funds transfer etc.”

Why PERFORMANCE Matters?User experience is an important factor , even minimal differences in the response time can have significant impact on the business

Amazon+100 ms1% drop in sales

Yahoo+400 ms5-9% drop in requests

Google+500 ms20% drop in requests

Bing+2000 ms4.3% drop in revenue/user

Shopzilla-5000 ms25% increase in requests7-12% increase in revenue50% reduction in hardware

Mozilla-2200 ms15.4% increase in downloads

KEY MESSAGES


• Websites are becoming richer in design and content

• Good user experience is most desirable attribute

• Web User turnaround starts from more than 3 seconds of response time

• Better interface demands performance optimization at all application levels mainly client side

“80-90% of the end-user response time is spent on the client side.”

Steve Souders, Google’s Head Performance Engineer

80-90% Client Side 10-20% Server Side

97%

www.aol.com

95%

www.ebay.com

95%

www.facebook.com

98%

www.msn.com

98%

www.myspace.com

94%

en.wikipedia.org/wiki

97%

www.yahoo.com

98%

www.youtube.com

TIME SPENT ON THE CLIENT SIDE

Performance Matters Continued


How Brain Perceives Performance


Agenda

Case study



Performance Matters

Know your Customer



We forecast more customers in betting and Gaming domain based on our research across competing operators, this in turn requires Betting Operators to have strong focus on enterprise scaling for Seasonal Sports and Festivals like Grand National to maximize its

customer acquisitions.

• Interim Q1 -2014- Mobile net revenue accounted for 50% of the online revenue

• Record new customer acquisition of 37.2%

Source: Ladbrokes Reports

• Interim Q1 -2014- Mobile revenue up 162% and the channel contributed 70% of the Sportsbook revenues

• 65% increase in the number of active customers in sustainable markets.

Source: BetFair Reports

• Interim Q1 -2014- Mobile net revenue accounted for 52% of the online revenue, with 73% of active Sportsbook customers

• Record new customer acquisition of 795,000 (up 35%)

Source: Paddy power Reports

• 1.5 million downloads from app store

• Mobile to generate 40% gaming revenue by mid – 2015

Source: WH Interim Reports

The Competetion IS ‘ON’


Lad-brokes

William Hill

Paddy Power

Bet365 Coral Betfred Skybet Betfair TitanBet0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

6440

3974

9354

1337

6561

5337

6841

8672

44654526

1514

3916

835

3240

19492150

4929

2302

Desktop Channel

Document Complete - First View (milliseconds)

Document Complete - Repeat View (millisec-onds)

WPT Report : Dated 20th-May -2015

Doc complete – user can perform action on the page. All DOM elements are downloaded.

Competitor Desktop APP Performance Analysis-Homepage

• Bet365 appears to be having quickest response for first time visitor with an average of 1.3 Seconds and repeat user at 900 milliseconds.

• Paddy Power seems to have taken high response time for DOM elements to be downloaded


Key performance testing challenges

Sustaining of Spike loads for short duration

Simulating realistic user behaviors across different customer facing channels

Simulating real time sports updates (Bet in Play and Non BIP sports) to client apps

Simulating realistic high volume transactional load patterns - Bets placed, signups, Deposits, Withdrawals for festive spirits like Grand National

Simulating tests on different clients apps-Desktop & Mobile /bandwidth simulation – 3G, 4G, WIFI/ Regions –UK and across the world

Understanding the underlying complex architecture and identifying the key touch points (shared architecture) in enterprise architecture

Maintaining high UX by introducing rich contents like Personalization, stylesheets, Capturing Analytics , Complex JavaScript's and Interactive chat services.

Unavailability of performance environment like to like with production environment


Agenda

Case study



Performance Matters

Know your Customer



Architecture are becoming very complex


Web pages are bigger and more complex

2.3 objects per pageAverage size 14k

75 objects per pageAverage size 498k

83 objectsAverage size 684k

1995 2010 2012


Latest Technologies used by Betting Operators

FRONT END

BACKEND

MIDDLETIER

BACKEND

CACHING STREAMING/PUSH

HTML 5, CSS3,Sass,

Jquery, Flash, Node.js,

Angular.js, Flex, AjaxJava, .Net,

Flash, Javascript,

Ruby, Python, PHP, Coffee

script

Mongo DB, Coherence, Memcache, Couchbase,

CDN

HDS, HLS, Smooth Streaming and

RTMPWebsockets/Push- Kaazing, Comet,

Redis, HTML5 Server Sent events ,

Lightstreamer, ICEfaces

Tibco, IBM Websphere,

Talend, UtlraESB,

Apache Camel

Informix, Oracle, DB2, IMS, MySQL


Agenda

Case study



Performance Matters

Know your Customer



Early Involvement

Any early involvement would be beneficial• Even if only asking a few key questions• Don't wait until everything gets in place

Unfortunately, you often get involved in the project at a later stage• Next sections are still fully applicable


Let ’s take the time to understand the applicationPerformance is driven by:• The end users• Third party system• …etc.

We need to ask the right questions:• How the users works on the system• What are their habits• When ? and how often• Are we going to expand in different geos?• Is the marketing plan to market/promote • the application? If yes what is the type

audience?• …etc.

Using the right methodology

Performance testing an existing application• Ask the support of functional

architect, HR..etc• Understand the behavior of the

application with the help of the logs

• Migration project : Don’t under estimate the history of the application

Performance testing a new application/service• Involve the project leader ,

functional architect• Try to understand the purpose

and relation with the business plan

Application is like a castle• You want to enter the castle to

load what’s inside • User pause would be the key

User’s habits are different• Include relevant behavior in your

tests to properly stress your components

• Involve the business to identify the ratio of users finalizing each user journey


Agenda

Case study



Performance Matters

Know your Customer



Grand national Performance Testing For a Betting Operator

Business Context

One of UK’s leading Betting and Gaming Company wanted to enhance the customer experience on web and mobile applications with 99.9% server uptime, quick server response and browser rendering time

Year’s highest turnover is expected on Grand National day for the Bookmaking Industry. Each Bookmaking Industry wants to make the best of this event by making a good profit but on this day most of Bookmaking Industry’s servers go down. How it was done ?

CREATE THE

SCHEDULES

FIRM UP NFRs

DESIGN THE

TESTS

EXECUTE TESTSREPORT & RECOMMEND

0 – FESTIVAL PREPARATION KICKOFF

43

21


Google AnalyticsWeb Server Log

Analysis to captureRealistic User

Journey

Volumetric NFRs on

Transactional Data like No. of Bet placement,

Deposits, Withdrawals etc.

Test scripts were designed with bespoke

frameworks/tools to mimic end user’s action and sports updates

Customized load profile to mimic

the load distribution pattern

as realistic as observed on GN

day

Amazon cloud to simulate the high Enterprise user’s

loads and distribute the users from

different geographies

Enterprise level system monitoring,

coordination across

development teams,

Infrastructure team and egaming

support teams

HOW DID WE GO ABOUT REALISTIC TESTS ?


TOOLSET FOR ENTERPRISE LOAD TESTS

WPT

• Client side tuning tool• User experience on

browsers• Private instance to

test in controlled environments

Correlsense

• Trace bottlenecks• Capture real time

metrics• Enables quicker turn

around in fixing the bottlenecks

Confluence

• NFRs• Execution reports• Documentation.

Neoload

• Mobile, Web, API, websocket, bandwidth simulations, customized reports etc.

• Identify bottlenecks in the early stages

Amazon Cloud

• To generate higher enterprise loads

• Mimic different geo• Cost effective• Cloud credit utilization model

Jira

• Defect management• Issue resolution tracking

Wireshark

• Network monitoring tool

• Capture network-related metrics

• Network Bottlenecks


KEY ENTERPRISE PERFORMANCE BOTTLENECKS FIXES & DECISIONS

SL NO. STREAM BOTTLENECK FIX

1 Mobile High response to download the static contentsCode changes to implement progressive download and using

application cache and moving few static contents to CDN

2 Desktop\MobileNot able to scale to high volume of login and bet

placementsDoubled the APIs capacity of Backend App servers to

support the high volume loads

3 MobileWebservers reaching capacity and causing high

failuresAdditional 7 no. of webservers were added to support high

volume loads

4 Network

Bandwidth utilization was reaching its capacity (95Mbps out of 100 Mbps link) causing resulting in

packet drops and high response time for transactions

Network traffic was split to MPLS(100Mbps) and other on VPN(100Mbps) to support high volume traffic

5 Desktop\MobileHigh response time for Login and Bet placement

observed at ESB layerAddition Instances for Login and Bet placement , turn off

logging payload and bypass EMS servers

SL NO. STREAM CASE DECISION OUTCOME

1 Desktop\MobileAuto –top up module not scalable to higher

volumes and impacting others functionality due to shared services

To Turn off Auto-top up functionality on GN day

To improvise user’s experience on the banking transactions without any

hassle on GN day

2 Desktop\Mobile

High volume of Bet history and Transaction History calls impacting server performances

and other modules hosted on the same servers due to shared services

To Turn off Bet History and Transaction History on peak hour of GN day

To reduce impact on user experience to customers on GN day

3 Desktop\MobileFraud rules check impacting the Login and Customer Deposits transactions at higher

volumes

To Turn off and Optimizing few Fraud rules causing problems and retaining the key

fraud rules on GN day

To reduce impact to the customers carrying out deposits on GN day


SOLUTION HIGHLIGHTSSpecific scenarios for AJAX calls, Push requests were simulated as per production website. Focus on improving Time to First Byte (TTFB), Page Rendering and Full Load for first view and repeat viewUser load distribution across multiple customer facing channels based on historical stats from Google Analytics, production web server logs. And real-time sports updates were pushed to desktop and mobile user interface to simulate realistic behavior

SOLUTION SHOWCASEPeak bets : 7000 bets/min. 1200 Deposits/Min, 700 signups/min and Peak user load of 60,000+ concurrent users on mobile and 25,000+ concurrent users on Desktop. Tools used: Neoload (cloud + on premise), Webpage test & bespoke simulation toolMaximised customer retention and acquisition

300% Response time Improvement in

application response time

Solution Showcase and Highlights


Key Factors for successful GN 2015 Readiness

Key DecisionsDriving Force – IT and Business

Real time monitoring

Risks/issues and mitigation planDaily MeetingsPerformance

StrategyCollaboration

GN 2015


Next Steps

Email [email protected] and [email protected]

Visit Neotys Community at answers.neotys.com

Download NeoLoad Free Edition from www.neotys.com

Evaluation support at www.neotys.com/support/contact.html

Questions?

Visit www.neotys.com and www.symphonyteleca.com

mailto:[email protected]

mailto:[email protected]

http://www.neotys.com/

http://www.neotys.com/support/contact.html

http://www.neotys.com/

http://www.symphonyteleca.com/


40GLOBAL OFFICES

8600EMPLOYEES

23COUNTRIES

Top 10Mobile Companies

Top 5Outsourced Product Development Companies

2012 Partner of the year Windows Azure, Finalist

Excellence AwardTechnology Agency of the

Year 2015

testing strategies to deliver consistent app performance

Technology

application performance

performance optimization

good ux performance

head performance engineer

content good user experience

quick response time

enduser response time

latest technology