performance engineeringsigist.org.il/_uploads/dbsattachedfiles/performance_eng.pdf · goals of...

Performance

Engineering

Salpeter Ariel

May 26 ,2009

Performance Engineering - PCoE

2

Introduction

Definitions

Performance Engineering

Project Flow - SDLC

PCOE

3

Introduction

Definitions


Project Flow - SDLC

PCOE

4

Copyright ©

What Is Performance?

Computational performance

How many instructions are required to execute a statement?

How much overhead does a particular virtual method call incur?

Should I use a quick-sort or a bubble-sort here?

5

Copyright ©



RAM foot-print

Amount of memory needed to run the application

Virtual memory

Good in dev, poor in deploy

Consider in design !

Co-hosting?

6

Copyright ©



RAM foot-print

Startup time

GUI application (BRRRR…..).

Web application.

7

Copyright ©



RAM foot-print

Startup time

Application scalability

A system that doesn't scale well

0

50

100

150

200

0 500 1000 1500 2000

Number of Users

Avera

ge R

esp

on

se

Tim

e

Series1

A system that scale well

0

5

10

15

20

25

0 500 1000 1500

Number of Users

Avera

ge R

esp

on

se

Tim

e

Series1

8

Copyright ©



RAM foot-print

Startup time

Application scalability

Perceived performance

Mouse cursor -> wait cursor

Background tasks

Responsive UI

9

Copyright ©

10

Copyright ©

Some Saying About Load & Performance

“There is no time now to create the optimize system, just develop a prototype and then…”

“The machines that we have are strong enough to hold the performance required for the system, no need to think about design aspects for load…”

“Why we need to invest on load, we’re delivering functionality?”

“There is no need to study exhaustively…”

11

Introduction

Definitions


Project Flow - SDLC

PCOE

12

Load & Performance – Wikipedia

• Is the process of creating demand on a system or device and measuring its response.

Load testing

• Is testing that is performed, from one perspective, to determine how fast some aspect of a system performs under a particular workload.

Performance testing

• Performance engineering within software engineering, encompasses set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the system.


Goals of Performance Engineering

Improve product quality

By producing reliable and tuned products that enhance our industry leadership position and are deliverable and serviceable

promote Company competitive advantage

By reduce COGS through benchmarks processes

high Performance products, that successfully Address Our customers’ business needs .

14

Software Performance Engineering

Build Performance into the Product rather then fixing it later

Lower the cost of handling performance issues

Requirements Design Implement Integration Test Maintain

$1 $2.50 $6.50 $16 $40 $110

“Clearly, it pays to invest effort in finding requirements errors early and correcting them in, say, 1

man-hour rather than waiting to find the error during operations and having to spend 100 man-

hours correcting it.” [Boehm]

Relative values

Software Performance EngineeringLower Costs

15

Performance EngineerBring Focus on Performance in Product

• Performance Testing

• Performance Tuning

• Analyzing Performance impact during Design

• Performance Benchmarking

• Capacity Planning and Sizing

• Defining Performance Requirements

• Enable Performance focus during Implementation

• Problem Diagnostics

• Collaborating with CS for Customer Performance Issues

16

Performance Engineer Skills

Networking knowledge

Load tools/simulators

Protocols knowledge

Operation System knowledge

monitoring

Analysis

Scripting knowledge

programming

Hardware knowledge

Database knowledge

17

Performance Engineer interfaces

18

SYS EngRequirements1. Definition of product performance 2. Database population pattern 3. external processes to simulate 4. Usage Model5. Density of events 6. Transaction response time criteria7. Band width calculation and

limitation

Performance Engineer interfaces

19

Introduction

Definitions


Project Flow - SDLC

PCOE

Performance Project Flow Diagram

P.20

Development Flow

PCoE Activity

PCoE Deliverable

Push Performance Upstream in SDLC

Component sizing, resource budget metrics, configuration tuning Build models, prototypes, Validate/POC

BenchmarkingTesting identified scenariosIdentify key performance

scenarios

ProductionQADesign ImplementationRequirements

Define performance

requirements


Software Development LifeCycle

22

SDLC Phase: Requirements

Pre-requisites: Product requirements exists

Goals Identify high level risks during requirements/design phases, mitigate or

manage them

Practices Analyze product requirements

Assess the performance risk of each requirement

Come up with recommended mitigation

After assessing all requirements, performance requirements are formed based

on the risky areas


Performance Risks Analysis

Overlay on Software Life Cycle

23

SDLC Phase: RequirementsPre-requisites Performance Risks Analysis Performance Requirements

Goals Define measurable performance objectives that are aligned with customer

needs and agreed upon all stakeholders

Practices Analyze customer environments (Workload Characterization) Determine the right workload for the testing environment.

Workload consists of: number of users, concurrent active users, data volumes, key scenarios and transaction volumes

Define objectives for: Response Time, Throughput, Resource Utilization Communicate Performance Objectives Get commitment for Performance objectives


Performance Objectives


http://wiki.mercury.com/confluence/display/itg/Workload+Characterization+and+Definition

24

SDLC Phase: Design

Pre-requisites:Performance Objectives

Goals

Ensure application key design choices takes performance principles under

consideration

Practices Learn design tradeoffs for performance and scalability Apply a principle-based approach on design Identify and use a performance and scalability framework Learn design considerations for scaling up and scaling out Minimize communication and data transformation overhead Improve application concurrency Manage resources efficiently Cache application data effectively Manage application state efficiently Design an efficient presentation layer, business layer and data access layer


Performance Design Guidelines


25

SDLC Phase: Design, Develop, TestPre-requisites: Performance Objectives High Level Design

Goals Characterize and understand the system’s behavior so that effective decisions may be

made.

Practices Identify key scenarios Identify workload Identify performance objectives Identify budget Identify processing steps Allocate budget Evaluate: run models, prototype Validate: measure in load tests


Performance Modeling


26

SDLC Phase: Design, Develop, Test, Deploy, Maintain

Goals Identify how the application performance stands in relation to defined performance goals Helps to identify the bottlenecks that affect application performance

Practices Measure System and Platform metrics Measure Network Profiling Analyze log files Instrument the application


Performance Measuring



SDLC Phase: Test

Goals Identify how well application performs in relation to performance objectives

Practices Identify key scenarios. Identify application scenarios that are critical for performance. Identify workload. Distribute the total application load among the key scenarios

identified in step 1. Identify metrics. Identify the metrics that you want to collect about the application

when running the test. Create test cases. Create the test cases where you define steps for executing a

single test along with the expected results. Simulate load. Use test tools to simulate load according to the test cases and to

capture the result metrics. Analyze the results. Analyze the metric data captured during the test.

27


Performance Testing

28

SDLC Phase: Develop, Test, Deploy, Maintain

Goals identification and systematic elimination of bottlenecks to improve performance

Categories for Tuning

Application. Includes the application configuration parameters.

Platform. Includes the host operating system, together with Web and database servers

System. Includes the hardware resources on a server. These resources include CPU,

memory, disk I/O, and network I/O.

Practices Assess the problem and establish numeric values that categorize acceptable behavior. Measure the performance of the system before modification. Identify the part of the system that is critical for improving the performance. This is called

the bottleneck. Modify that part of the system to remove the bottleneck. Measure the performance of the system after modification


Performance Tuning


29

SDLC Phase: Test, Deploy

Goals Assess the relative performance of an object, normally by running a number of standard

tests and trials against it Equip field personnel with valuable information to facilitate product sales

Details

Benchmarks provide a method of comparing the performance of various subsystems

across different chip/system architectures. Benchmarking is helpful in understanding how

the database manager responds under varying conditions. You can create scenarios that

test deadlock handling, utility performance, different methods of loading data, transaction

rate characteristics as more users are added, and even the effect on the application of

using a new release of the product.


Performance Benchmarking


30

SDLC Phase: Test, Deploy, Maintain

Goals Planning for growth and forecasting peak usage periods in order to meet system and

application capacity requirements

Practices Transaction cost analysis

Compile a user profile Execute discrete tests Measure the cost of each operation Calculate the cost of an average user profile Calculate site capacity Verify site capacity

Predictive analysis Collect performance data Query the existing historical data Analyze the historical performance data Predict the future requirements


Capacity Planning


31

Introduction

Performance Eng

Project Flow - SDLC

Interfaces

PCOE

Executive summary

P.32

Solution :

performance Center of excellence-

central scalable performance group that produces

significant quality improvement and synergy according to

Comverse business strategy

Lack of synergy between Performance teams ,and

Performance engineers abandonment leads to

unbalanced Performance level in the various products

33

What is a CoE- Center of Excellence?

TECHNOLOGY

METHODOLOGY

EXPERTS

Focus & Expertise

DriveOptimization

ProveValue

ConsolidateKnowledge

DisseminateExpertise

Maximize Business Value

Goals of the Performance Group

Improve product quality

By producing reliable and tuned products that enhance our industry leadership position and are deliverable and serviceable

promote Comverse competitive advantage

By reduce COGS through benchmarks processes

Increase efficiency

Enable synergetic leverage of the company performance teams and experts

Reduce performance engineers abandonment

By creating technical career path for performance engineers34

high quality products, that successfully Address Our customers’ business

needs and enhance our competitive edge

Platform Group Gathering – Q1 2008

Comverse PCoE – Products

• MMS

• SMS

• Unified Messaging

• Mobile IM

• Network address book

• Mobile Email

• Mobile Gateway

Advantages

Expertise in Performance Area Solid performance testing methodology

Monitoring best practices

Expertise in load Testing, diagnostics and profiling tools

Scalable Performance Team Quick ability to recover in case of leave

Shares human resources upon need to handle peak times

Experience in recruiting engineers with appropriate talent/potential

Technical Career Path for Performance Experts Performance engineering oriented management

Long term vision

Performance experts home

36

Advantages

Bring Focus on Performance in Product Development

Analyzing performance impact during design

Defining performance requirements

Capacity planning and sizing

Drive performance agenda across organization

Centralized Performance specific knowledge base

Performance is our profession

Collaboration and Visibility PCoE representative attends product status meeting and provide performance

status

work on daily basis with developers

Team manager works closely with QA manager on priorities and realigns efforts

per demand

Delivering Scalable & Reliable Product Development awareness & attention to performance aspects

Ability to handle customers’ escalations

37

Requirements

phase

Design

phase

Implement

phase

Execution

phase

38

performance Eng in Project life CycleKnowledge Base/

Group Experts

TL review PCEO reviewTL review

Performance

Eng

SUT Testing

PU&PCOE review

PCOE life cycle

The End

40

Scalability Testing

• Complementary to load testing

• Workload Simulated: Normal & Peak workloads expected for a given system configuration

• Test Duration: typically few hours

• Examined Aspect

− Resource Utilization Levels

− Others: only if the load testing didn’t reveal relevant issues (e.g. response times)

• Can be used as a basis for Capacity Planning models

• Scale up or Scale Out tests might be considered in case system configuration is not sufficient for examined workload

Quantify Performance Under Increased Levels of Workloads

Scalability Testing


• Load Testing

• Stress Testing

• Scalability Testing

• Other activities outside testing zone

− Capacity planning

− Performance Optimization

− Performance Tuning

Performance Tests Types


Load Testing

• Workload Simulated: 120% of Normal & Peak Workload

• Test Duration: varies from few hours to few days

• Examined Aspects

− Response times

− Throughput rates

− Resource utilization levels

− Breaking Points (in case it happens below the peak workload conditions)

Test Behavior Under Normal & Peak Workload

Load Testing


Stress Testing

• Workload Simulated: Extreme workload. Might reach 500% of Normal & Peak workload

• Duration: Typically less then an hour

• Goal: unearth application bugs that surface only under high load conditions

• Examined Aspects

− Synchronization issues

− Race conditions

− Memory leaks

Evaluate Behavior Beyond Normal & Peak Workloads

Stress Testing

performance engineeringsigist.org.il/_uploads/dbsattachedfiles/performance_eng.pdf · goals of...

Documents