1

Perforce Load Testing - AgendaPerforce Load Testing - Agenda

• Speaker introduction• Presentation scope• Corporation background• Load Test initiating: motivation, goals• Load Test planning: software, hardware, licenses, clients• Load Test execution: user emulation limits, escalation• Load Test control: monitor clients and metrics• Load Test closing: charts, analysis, lessons learned• Questions?

2

Perforce Load Test – Speaker IntroductionPerforce Load Test – Speaker Introduction

• Dan Healy – daniel.f.healy@bankofamerica.com– Current SCM admin based in London, UK– Former SCM admin for BofA in San Francisco & Concord, CA– Former index, currency and hedge fund application developer at

Wells Fargo Investment Advisors and Barclays Global Investors inSan Francisco

– Former Defence Dept. developer at White Sands, NM– Academic background in physics, astronomy and computer science– Solar eclipse chaser and mountaineer

3

Perforce Load Test – Presentation ScopePerforce Load Test – Presentation Scope

• Primary focus on why and how the load test was conducted• Sterling test results should not be a surprise to anyone• Provide a list of items to consider when planning,

configuring and executing a Perforce load test• Project Management Institute (PMI) methodology

4

Perforce Load Test – Corporation BackgroundPerforce Load Test – Corporation Background

• Bank of America– First/second largest financial institution in the world– 200,000+ employees, 10,000 developers– 21,000,000 online banking customers– 443,000,000 ATM transactions per month– 3.9 billion bankofamerica.com hits per month– Pioneered consumer loans and branch banking– Historical technology innovator

• First credit card – Bankamericard now known as Visa• Magnetic ink numbering on checks

– Current technology innovations• Distributed Computing• Continuous Integration• Nanosecond f/x transactions

5

Perforce Load Test - InitiatingPerforce Load Test - Initiating

• Application development highly dependant on globally distributed teams– ClearCase insufficient– 3 separate rogue development teams began using Perforce

• Architecture Review Board (ARB) certification required– Management skeptical regarding Perforce

• Perforce load test added to certification submission– Determine any hardware or performance limitations on BofA

hardware running extraordinary usage– Attempt to exceed 10x peak ClearCase usage, 24/7

6

Perforce Load Test - PlanningPerforce Load Test - Planning

• 2000 temp user licenses from Perforce• Software

– Latest version of Perforce – 2006.1– Perforce support load test engine

• Created a client workspace template called “bulk”• Runs sync, flush, info & files, “x” iterations, “y” users, unique prefix for user IDs

– Server performance monitoring – RRDTool– Control Panel Administrative Tools – Performance– Excel spreadsheet

• Test servers– similar setup to production environment

• Primary server in UK – Linux 4 CPU i686, 2.6.9 kernel• Proxy servers in Chicago & Charlotte, Solaris 5.8, Sun Ultra-250• 3 155MB T1’s between US & UK

• Test clients– Cooperation from several colleagues to use their PCs– Invaluable use of training room PCs in Chicago & London

7

Perforce Load Test - ExecutingPerforce Load Test - Executing

• How many users can each PC emulate?– PC disk space limitations vs. ample workspaces– Settled on a 300MB depot and workspace– 5-60 users per PC

• Used Control Panel Scheduled Tasks– Typical activity: 20 users/50 iterations every 10 minutes– Annoying pop up window

• Escalated usage to 800 users• Occasional manual remove & submit

– 969 files, 8 folders, 56.2MB• Focused on average background and peak activity

8

Perforce Load Test – Monitoring & ControlPerforce Load Test – Monitoring & Control

• PC performance tool and graphics ensured clients were not limiting activity

• Every morning, made sure all clients were active using Perforce GUI

• Use RRDTool before and during major activity– Number of connections– Number of processes– % CPU– Load average– Memory usage– Manually recorded of number of users

• Determined production ClearCase and Perforce usage• Determined server peak levels

– Memory– % CPU– Load average

9

Perforce Load Test – RRDTool – Number of ProcessesPerforce Load Test – RRDTool – Number of Processes

10

Perforce Load Test – RRDTool – Active ConnectionsPerforce Load Test – RRDTool – Active Connections

11

Perforce Load Test – RRDTool – Production Active ConnectionsPerforce Load Test – RRDTool – Production Active Connections

12

Perforce Load Test – RRDTool – Production ClearCase ServerPerforce Load Test – RRDTool – Production ClearCase Server

13

Perforce Load Test – RRDTool – Memory UpgradePerforce Load Test – RRDTool – Memory Upgrade

14

Perforce Load Test – Closing – Background ActivityPerforce Load Test – Closing – Background Activity

15

Perforce Load Test – Closing – Peak ActivityPerforce Load Test – Closing – Peak Activity

16

Perforce Load Test - ClosingPerforce Load Test - Closing

• Analysis– BofA hardware remains effective up to about 500

simultaneous intensive users– Most vulnerable metric is memory

• Extrapolated peak at about 1000 users

• Lessons learned– More user participation and/or training rooms– Employ UNIX/Linux clients– Capture of PC performance graphics– Automate peak activity– Run Scheduled Tasks as alternative user