shiftleft:continuousperformance! …...what is continuous performance testing? 10/29/2018 4 7 8...

   DT3  Integrating  Testing  into  DevOps  Thursday,  November  8th,  2018  10:00  AM              

Shift  Left:  Continuous  Performance  Testing  in  the  CI/CD  Pipeline  

 Presented  by:    

Gajan    Pathmanathan  Quicken  Loans  

‘      

Brought  to  you  by:        

350  Corporate  Way,  Suite  400,  Orange  Park,  FL  32073    888-­‐-­‐-­‐268-­‐-­‐-­‐8770  ·∙·∙  904-­‐-­‐-­‐278-­‐-­‐-­‐0524  -­‐  info@techwell.com  -­‐  http://www.starwest.techwell.com/      

Gajan    Pathmanathan      Gajan  Pathmanathan  is  a  quality  evangelist,  DevOps  enthusiast,  and  engineering  leader  with  fifteen  years  of  experience  in  designing,  developing,  and  testing  software.  He  currently  leads  the  Enterprise  Quality  Architecture  team  at  Quicken  Loans,  where  he  is  responsible  for  implementing  solutions  and  strategies  for  continuous  quality  and  quality-­‐driven  development  across  the  enterprise.  Before  joining  Quicken  Loans,  Gajan  worked  at  Intel  Security/McAfee  as  a  lead  engineer  and  led  the  team  that  was  responsible  for  defining  quality  processes  and  test  automation  processes  across  the  organization.  Gajan  holds  a  master's  degree  in  computer  engineering  from  the  University  of  Waterloo,  Canada,  and  he  is  a  certified  SOA  Architect  and  ScrumMaster.  Gajan  has  given  guest  lectures  at  technical  institutes  and  spoken  regularly  at  technical  conferences  across  Canada  and  the  United  States.      

10/29/2018

1

Shift Left: Continuous Performance Testing in the CI/CD Pipeline

https://www.linkedin.com/in/gajanp https://twitter.com/gajanp

Gajan PathmanathanEngineering Leader @ Quicken Loans

Award-Winning Culture

10/29/2018

2

More Than Just Mortgages…FINTECH PRIVATE EQUITY REAL ESTATE

VENTURE CAPTIAL

FOC INVESTEMENTS

SPORTS/MEDIA/ENTERTAINMENT/

GAMING/HOTELS

On The Agenda

10/29/2018

3

01 03

0204

05

What is Continuous Performance Testing?01

The Why Behind The What02

Recommended Process03

Tooling04

Demo05

What is Continuous Performance Testing?

10/29/2018

4

7

8

Continuous Performance TestingA process of executing automated performance tests as part of

delivery pipelines

10/29/2018

5

9

❑ Performance Smoke/Regression Testing

❑ End-to-End Performance Testing

Types

The Why Behind The What

10/29/2018

6

11

“You can fix it now on

the drafting board with an

eraser, or you can fix it later on

the construction site with a

sledgehammer.”

12

https://reqtest.com/general/a-bug-goes-skateboarding-on-boehms-curve/

10/29/2018

7

13

14

10/29/2018

8

10/29/2018

9

17

10/29/2018

10

Performance & User Behavior

https://raygun.com/blog/cost-of-software-errors/

Process

10/29/2018

11

21

Maturity Levels

Continuous Performance

ManagementRelease Gate

N/A Manual Automated (manual run)

22

10/29/2018

12

23

24

10/29/2018

13

25

26

10/29/2018

14

Tooling

10/29/2018

15

30

10/29/2018

16

31

❑ Open source desktop application written in Java

❑ Plugin-based architecture

❑ 20+ years in existence

32

❑ Open source tool written in Python

❑ 5+ years in existence

❑ Code-driven framework

10/29/2018

17

33

❑ Open source automation framework written in Python

❑ Taurus – Test Automation Running Smoothly

❑ Abstracts other testing tools

Feature JMeter Locust

Open Source Yes Yes

Operating System Any Any

GUIYes

(non-GUI mode available)No

Recording Functionality Yes No

Test Recorder HTTP No

Test Language XML Python

Extension Language Java, Beanshell, Javascript, Jexl Python

Load ReportsCSV, XML, Embedded Tables,

Graphs, PluginsHTML

Feature comparison

10/29/2018

18

Feature JMeter Locust

Protocol Support

(in-built)

HTTP, FTP, JDBC, SOAP

LDAP, TCP, JMS, SMTP

POP3, IMAP

HTTP

Execution Monitoring

Console, File, Graphs

Desktop client

Custom plugins

Console

Web

Integrated Host monitoringYes

(PerfMon plugin)No

Support of “Test as Code” Weak (Java) Strong (Python)

Feature comparison

Feature JMeter Locust

Distributed Execution Yes Yes

Easy to use with VCS No Yes

Resources Consumption More resources required Less resources required

Number of Concurrent UsersThousands

(under restrictions)Thousands

Ramp-up Flexibility Yes No

Test Results Analyzing Yes Yes

Feature comparison

10/29/2018

19

Demo

38

❑ JMeter & Taurus Command Line Tool (bzt)

❑ Locust & Taurus Command Line Tool (bzt)

❑ Integration with Jenkins (Build step)

o Command (bat or sh)

o Performance plugin

Let’s Look at…

10/29/2018

20

39

❑ JMeter:http://jmeter.apache.org/

❑ Locust:https://locust.io/

❑ Taurus:http://gettaurus.org/

❑ Performance Plugin:https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

Questions? Come up and chat!

You can find me at:

@gajanp

Feedback is a GIFT!