How to Design and How to Design and Implement Load Testing Implement Load Testing

for Sakaifor Sakai

The Michigan Experience

8 Dec 2006 lmp@umich.edu 2

Why Load Test?

– Measure end-user response time• How long does a specified operation take?• Currently a minor part of our testing

– Define optimal hardware configuration• Enables pre-tuning of Oracle• Testing various JVM settings

– Check reliability• How long or hard can system work without

producing errors (failures)?

8 Dec 2006 lmp@umich.edu 3

Why Load Test....– Check hardware & software upgrades

• Does new host or software perform as well as previously tested versions?

– Evaluate new products• Used to evaluate hardware for database replacement

– Measure system capacity• How much can the system handle before performance

degradation occurs?

– Identify bottlenecks• What part of the environment causes performance

degradation?

8 Dec 2006 lmp@umich.edu 4

Test Environment

• Performance Testing Tool

• Infrastructure

• Courses & Users

• Data

8 Dec 2006 lmp@umich.edu 5

Performance Testing Tool

• Mercury Interactive LoadRunner– VuGen (virtual user scripting)– Controller (scenario and load generation)– Analyzer (analysis and reports)

• Approach appropriate to any tool

• Presentation will not focus on tool

8 Dec 2006 lmp@umich.edu 6

Infrastructure• As close to Production as possible

– Hardware– Software

• Same configuration unless testing something specific

– Authentication• Use application rather than CoSign

› UM-developed, open-source single web login service

– UMIAC• Warehouse for needed administrative data

› UM-developed, PERL scripts accessing Peoplesoft system

– Database servers not always identical• Where less “powerful,” larger host will likely perform better than

observed in tests

– AFS resource storage

8 Dec 2006 lmp@umich.edu 7

Courses & Users

• UMIACload provides course names, instructor owners, and rosters of students– 11,000 courses– 11,000 “instructors”– 43,000 “students”

• Why so many?– To permit high level of activity without

excessive concurrency

8 Dec 2006 lmp@umich.edu 8

Data

• Testing requires that all courses be pre-populated with materials to be used by students

• Both AFS and Oracle data used

• Saved versions of “base” data maintained to enable rollback to “clean” data

• Always start with data refresh when adding new tools to test suite

8 Dec 2006 lmp@umich.edu 9

Testing Process

• Scripts

• Scenarios

• Analysis

• Results

8 Dec 2006 lmp@umich.edu 10

Script Setup• Analyze tool to be scripted• Understand tool usage

– Value of user profiling

• Design test plan– Goals– Single or multiple scripts?– Student or Instructor activity?– Are course materials required?

• Design data files– Used by Virtual Users (VU) to change input parameters

8 Dec 2006 lmp@umich.edu 11

Script Setup...

• Create course materials

• Populate courses with materials– Must be done sequentially to ensure all

courses get materials– Multiple resources can be added to same

course with same script– Need more efficient method

• Record activity to be tested

8 Dec 2006 lmp@umich.edu 12

Script Parameterization• Scripts recorded as single user• Load iterations need to change users each

iteration– Data files used to manage user & input changes

• All object-oriented elements in Sakai receive associated ID numbers that must change whenever a course or user changes– Recording rules used to manage ID number changes– Generic parameter identifiers changed to make

analysis easier

8 Dec 2006 lmp@umich.edu 13

Script Pseudocode - Chat• Login as “u10001”

– Moves to “My Workspace”

• Select course tab– Use “LOADTEST 102 001”

• Select “Chat” tool from left-hand navigation column– Moves to Chat tool

• Enter message into Chat text entry box• Return to “My Workspace”• Logout

Chat(){

web_set_max_html_param_len("1024");

web_url("xlogin", "URL=https://ctoolsload.ds.itd.umich.edu/portal/xlogin", "Resource=0", "RecContentType=text/html", "Referer=", "Snapshot=t1.inf", "Mode=HTML", EXTRARES, "Url=../ctlib/skin/ctools/images/logo_inst.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/tool_base.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topleft-p-title.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/help.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topright-p-title.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/reload_h.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/chatpresicon.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, LAST);

Script Walkthrough - Chat

// [WCSPARAM WCSParam_UserID 36 b2de7f47-7c74-40d5-0024-98d191450a5d] Parameter {WCSParam_UserID} created by Correlation Studioweb_reg_save_param( "WCSParam_UserID", "LB=site/", "RB=\"", "Ord=2", "Search=Body", "RelFrameId=1.1", LAST );web_submit_form("xlogin_2",

"Snapshot=t2.inf", ITEMDATA, "Name=eid", "Value={CTlogin}", ENDITEM, "Name=pw", "Value={CTpwd}", ENDITEM, "Name=submit", "Value=Login", ENDITEM, EXTRARES, "Url=../ctlib/skin/ctools/images/topright-tab.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/portal.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topright-curtab.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/portal.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topright-tab_hov.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/portal.css", ENDITEM, LAST);

web_link("LOADTEST {Course2} 001 W06", "Text=LOADTEST {Course2} 001 W06", "Snapshot=t3.inf", LAST);

lr_think_time(5);

// [WCSPARAM WCSParam_ChatID 36 194d7683-e9fc-41d3-0012-f0ac6fea0d60] Parameter {WCSParam_ChatID} created by Correlation Studioweb_reg_save_param( "WCSParam_ChatID", "LB=title/", "RB=?", "Ord=1", "Search=Body", "RelFrameId=1.3", LAST );web_link("Chat Room",

"Text=Chat Room", "Snapshot=t4.inf", EXTRARES, "Url=/courier/194d7683-e9fc-41d3-0012-f0ac6fea0d60/1145294730027-0.2702248719455739?auto=true", "Referer=", ENDITEM, "Url=/courier/194d7683-e9fc-41d3-0012-f0ac6fea0d60/1145294740215-0.4220853133253527?auto=true", "Referer=", ENDITEM, "Url=/courier/194d7683-e9fc-41d3-0012-f0ac6fea0d60/1145294750293-0.2283065980084248?auto=true", "Referer=", ENDITEM, "Url=/courier/{WCSParam_UserID}-presence/1145294759434-0.23544053982281854?auto=true", "Referer=", ENDITEM, LAST);

web_url("{WCSParam_UserID}", "URL=https://ctoolsload.ds.itd.umich.edu/portal/presence/{WCSParam_UserID}", "Resource=0", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/presence/{WCSParam_UserID}", "Snapshot=t5.inf", "Mode=HTML", EXTRARES, "Url=/courier/{WCSParam_ChatID}/1145294760356-0.31310668668084407?auto=true", "Referer=", ENDITEM, LAST);

web_submit_data("{WCSParam_ChatID}", "Action=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=Control", "Method=POST", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=Control", "Snapshot=t6.inf", "Mode=HTML", ITEMDATA, "Name=message", "Value=Who me? Wadya want, meathead? sheesh.....", ENDITEM, "Name=eventSubmit_doSend", "Value=x", ENDITEM, "Name=send", "Value=Add message", ENDITEM, EXTRARES, "Url=/courier/{WCSParam_ChatID}/1145294761621-0.647618403375133?auto=true", "Referer=", ENDITEM, LAST);

web_url("{WCSParam_ChatID}_2", "URL=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=List", "Resource=0", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=Main", "Snapshot=t7.inf", "Mode=HTML", LAST);

lr_think_time(6);

web_url("%7E{CTlogin}", "URL=https://ctoolsload.ds.itd.umich.edu/portal/site/%7E{CTlogin}", "Resource=0", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/site_tabs/{WCSParam_UserID}", "Snapshot=t8.inf", "Mode=HTML", LAST);

return 0;}

8 Dec 2006 lmp@umich.edu 20

Scenarios

• One or more scripts to be run in load test– Number of virtual users running script– Number of iterations per virtual user– Pacing (how much delay between iterations)– Think time (emulates user variability)– Browser emulation– Network bandwidth emulation– Results detail (step or full transaction level)

8 Dec 2006 lmp@umich.edu 21

13 “Core” Tests

• Instructor– CreateAnnounce– CreateAssign– ReviseResource– ScheduleEvent– UploadLgFile– UploadSmFile

• Student– Chat– CreateProject– DownloadSmFile– DownloadLgFile– NewDiscussion– ReadBrowse– SubmitAssign

8 Dec 2006 lmp@umich.edu 22

“New” Tool Tests

• Instructor

– AddSoMod– AddVzMod– EditSyllabus– EditWiki3L– SubmitGrades

• Student– ReadTxtMod– ReadSoMod– ReadVzMod– ReadSyllabus– ReadWiki3L– ViewGrade

8 Dec 2006 lmp@umich.edu 23

Scenario Preparation

• Single script runs– 1 VU– 100 iterations– No wait time between iterations– No think time– Showing response times for each step of

transaction

• Identify serious script problems• Take average response time for scenario

8 Dec 2006 lmp@umich.edu 24

Scenario Iteration Timing (125%)

Script name (VU, Wait)

CreateSite/Project 3VU, 0sec

UploadSmFile (85%) 4VU, 0sec

UploadLgFile (15%) 2VU, 7sec

DownloadSmFile (85%) 78VU, 0sec

DownloadLgFile (15%) 10VU, 0sec

CreateAnnounce 2VU, 0sec

ScheduleEvent 6VU, 0sec

NewDiscussion 1VU, 1sec

CreateAssign 1VU, 57sec

SubmitAssign 8VU, 0sec

ReviseResource 2VU, 4sec

Chat 10VU, 0sec

LoginLogout & ReadBrowse 76VU, 0sec

Script name (VU, Wait)

EditWiki3L 1VU, 136sec

ReadWiki3L 1VU, 1sec

EditSyllabus 1VU, 157sec

ReadSyllabus 1VU, 29sec

SubmitGrades 2VU, 67sec

ViewGrades 1VU, 12sec

AddVzMod 1VU, 140sec

AddSoMod 1VU, 141sec

ReadTxtMod 1VU, 36sec

ReadVzMod 1VU, 23sec

ReadSoMod 1VU, 44sec

8 Dec 2006 lmp@umich.edu 25

Analysis & Results• Scenario runs for 1 hour plus ramp up and down

time• Upon completion of run, use LoadRunner

Analyzer to crunch data• Compare resulting graphs with previous runs• Developers, DBA, SysAdmins look for

performance bottlenecks• Fixes proposed and test rerun

8 Dec 2006 lmp@umich.edu 26

CTload Scenario Results:2.0_r3636w8app (100%)

8 Dec 2006 lmp@umich.edu 27

CTload Scenario Results:2.2.1_r14159w6app (125%)

8 Dec 2006 lmp@umich.edu 28

Sharing the Burden?• Multiple campuses participate

– Ideally a few dedicated partners

• Localized hardware• Localized test tool choice• Centralized resources

– Test design– Pseudocode for scripts– Course data files– Results– Support and discussion

8 Dec 2006 lmp@umich.edu 29

Community Testing?

• Share common core tests – Baseline for comparison

• Focus on new tools of interest to campus– Contribute resources back to central

repository– Contribute results to community

• Greater stress on performance in development