lotus domino 6€¦ · setting up the system under test..... 23 setting up the drivers ... setting...

Lotus NotesBench Guide 6.0

Lotus Domino 6.5

Lotus software

Disclaimer

THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS PROVIDED FOR INFORMATIONALPURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS ANDACCURACY OF THE INFORMATION CONTAINED IN THIS DOCUMENTATION, IT IS PROVIDED “ASIS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THISINFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARESUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANYDAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THISDOCUMENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THISDOCUMENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANYWARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), ORALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENTGOVERNING THE USE OF IBM SOFTWARE.

Licensed Materials - Property of IBM

©Copyright IBM Corporation 1985, 2003All rights reserved.

US Government Users Restricted Rights - Use, duplication or disclosure restricted by GS ADP ScheduleContract with IBM Corp.

Lotus SoftwareIBM Software GroupOne Rogers StreetCambridge, MA 02142

List of Trademarks

IBM, the IBM logo, 1-2-3, AIX, AS/400, DB2, Domino, Domino Designer, Freelance Graphics, iNotes,iSeries, LearningSpace, Lotus, Lotus Discovery Server, Lotus Enterprise Integrator, Lotus Notes, LotusOrganizer, MQSeries, Netfinity, Notes, OfficeVision, OS/2, OS/390, OS/400, S/390, Tivoli, QuickPlace,Sametime, SmartSuite, WebSphere, and Word Pro are trademarks or registered trademarks ofInternational Business Machines Corporation in the United States, other countries, or both. Pentium is atrademark of Intel Corporation in the United States, other countries, or both. XEON is a registeredtrademark of Intel Corporation in the United States, other countries, or both. Microsoft, Windows, andWindows NT are registered trademarks of Microsoft Corporation in the United States, other countries, orboth. UNIX is a registered trademark of The Open Group in the United States and other countries. Javaand all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems,Inc. in the United States, other countries, or both.Other company, product, or service names may betrademarks or service marks of others.

Chapter 1 Introducing NotesBench 1.........................................What is NotesBench? 1................................................................................

What is a Benchmark? 1........................................................................NotesBench Terminology 3..........................................................................

Workloads 3...........................................................................................Price 3....................................................................................................Price/Performance 3..............................................................................Performance 3........................................................................................Scalability 3............................................................................................Steady State 3........................................................................................What’s in NotesBench? 3.......................................................................NotesBench User Interface 3.................................................................NotesBench Scripts 3.............................................................................NotesBench Workloads 3.......................................................................The NotesBench Consortium 5..............................................................NotesBench Consortium Membership 5................................................NotesBench Consortium Web Site 5......................................................

NotesBench Requirements 6........................................................................Network Protocol Requirements 7.........................................................Memory Requirements 7........................................................................Hard Disk Requirements 7.....................................................................

NotesBench Environmental Variables 9.......................................................Auto Initialization 9.................................................................................Variables Fixed for Publishing Results 9................................................Variables Based on the Number of Users 9...........................................User-Settable Variables 9......................................................................

NotesBench executables, notebnch, notesnum, and probe 9......................notebnch 9.............................................................................................notesnum 9............................................................................................probe 11...................................................................................................Additional Details about Probe_Interval 11..............................................Configuring Network Protocols 11............................................................Installing Templates on Child Drivers 11..................................................

NotesBench Scripts 13...................................................................................Viewing the NotesBench Scripts 13.........................................................

Chapter 2 QuickStart 15.................................................................NotesBench QuickStart - Setting up NotesBench and RunningWorkloads 15..................................................................................................

Quick Start - Setting up the workload 15..................................................Quick Start - Running a Workload 15.......................................................

Typical NotesBench Configuration 17............................................................NotesBench Quick Start Table of Setup Instructions 17................................

Chapter 3 Preparing the Environment 21.....................................Planning the Duration of the NotesBench Tests 21........................................Installing Lotus Domino and Notes software for use with NotesBench 21.....

Installing NotesBench Software 21..........................................................

Installing NotesBench on a UNIX system 21...........................................Scaling NotesBench tests 21..........................................................................

Determining the Number of Users 23.......................................................Varying the Number of Users with Multiple Run (MultiRun) 23................

Setting Up NotesBench 23.............................................................................Setting up the System Under Test 23......................................................Setting up the Drivers 25..........................................................................Setting Up Location Documents for Parent and Child Drivers 25............Setting up the Parent Driver NOTES.INI File 25......................................Setting up Child Driver NOTES.INI Files 29.............................................The NotesBench Configuration File NBCONFIG.INI 31...........................

Setting up destination servers 31...................................................................Creating Mail Databases for NotesBench 31...........................................Using the "Create NotesBench Mail Persons Documents" Agent 33.......

Using the NotesBench Results database 33..................................................Setting Up the NOTES.INI File for NotesBench 35........................................

NOTES.INI settings used by NotesBench 37...........................................Sample NOTES.INI files for NotesBench tests 45..........................................

Child driver sample NOTES.INI file 45.....................................................Parent/Probe Driver sample NOTES.INI file 47.......................................R6Mail Test sample NOTES.INI file 47....................................................R6IMAP Test sample NOTES.INI file 49..................................................R6iNotes Test sample NOTES.INI file 49................................................R5Mail Test sample NOTES.INI file 49....................................................R5IMAP Test sample NOTES.INI file 51..................................................R5iNotes Test sample NOTES.INI file 51................................................ClusterMail Test sample NOTES.INI file 51.............................................WorkGroup test sample NOTES.INI file 53..............................................MakeReplicas Initialization Script sample NOTES.INI file 53..................

Chapter 4 Running the Tests 55....................................................General Instructions for Running all NotesBench Tests 55............................

Initial Check and Auto Check 55..............................................................Initial Check 55.........................................................................................Auto Check 55..........................................................................................Using NOTEBNCH to run the tests 55.....................................................Some NOTEBNCH examples: 55............................................................Error Handling During the Tests 55..........................................................Using NOTESNUM to obtain the results 8.............................................Monitoring the tests 8.............................................................................Monitoring the results on the System Under Test 3...............................Comparing Users on Drivers and the System Under Test 59..................Using Probe 59........................................................................................Probe Setup 59........................................................................................Running Probe 61....................................................................................Considerations for DEBUG_OUTFILE 61................................................

Running the NotesBench Tests 61.................................................................

NotesBench workloads 61.......................................................................NotesBench Initialization Tests 61...........................................................

R6Mail Test 63...............................................................................................Metrics 63.................................................................................................Requirements 63......................................................................................Test Initialization 63.................................................................................NOTES.INI Settings for the child driver for R6Mail 63.............................NOTES.INI settings for the R6Mail SUT 65.............................................Server document settings for the R6Mail test 65.....................................Running the Test 65.................................................................................

R6IMAP Test 67.............................................................................................System information 67.............................................................................Time Requirements 67.............................................................................Metrics 67.................................................................................................Requirements 67......................................................................................Test Initialization 67.................................................................................NOTES.INI settings for the child driver for R6IMAP test 69.....................NOTES.INI settings for the R6IMAP SUT 69...........................................Server document settings for the R6IMAP test 69...................................Special instructions for the R6IMAP Test 69............................................Running the test 71..................................................................................Client Setup 71.........................................................................................

R6iNotes Test 73............................................................................................Time Requirements 73.............................................................................Metrics 73.................................................................................................System Requirements 73.........................................................................Test Initialization 73.................................................................................Special Considerations for this workload 75............................................Server document settings for the R6iNotes test 75..................................NOTES.INI settings for the R6iNotes test SUT 75...................................Access rights 75.......................................................................................Authentication 75.....................................................................................Running the test 75..................................................................................

R5Mail Test 77...............................................................................................Metrics 77.................................................................................................Requirements 77......................................................................................Test Initialization 77.................................................................................NOTES.INI settings for the child driver for R5Mail 77..............................NOTES.INI settings for the SUT 77.........................................................Server documents settings 77..................................................................Running the test 79..................................................................................

R5iNotes Test 79............................................................................................Time Requirements 79.............................................................................Metrics 79.................................................................................................System Requirements 79.........................................................................Test Initialization 81.................................................................................Special considerations for this workload 81.............................................

NOTES.INI settings for the Notes client driver systems 81......................Setting up the Domino Server for the R5iNotes Workload 81..................Server document settings for the R5iNotes workload 83.........................NOTES.INI settings for the SUT 83.........................................................Access Rights settings 83........................................................................Running the test 83..................................................................................

R5IMAP Test 85.............................................................................................Time Requirements 85.............................................................................Metrics 85.................................................................................................Requirements 85......................................................................................Test Initialization 85.................................................................................NOTES.INI settings for the child driver for R5IMAP 85............................NOTES.INI settings for the R5IMAP SUT 87...........................................Server document settings for the R5IMAP test 87...................................Running the test 87..................................................................................

Cluster Mail Test 87........................................................................................Metrics Used 87.......................................................................................Test Initialization 89.................................................................................NOTES.INI settings on the child driver for the Cluster Mail test 89.........Running the test 89..................................................................................

WorkGroup Test 91........................................................................................Time Requirements 91.............................................................................Metrics 91.................................................................................................Requirements 91......................................................................................Test Initialization 91.................................................................................NOTES.INI settings for the child driver for the WorkGroup test 91..........Running the test 93..................................................................................

NotesBench Database Initialization Tests 93.................................................NRPC Mail Initialization Workload 95.............................................................

Running the Web Mail Initialization or Mail Initialization Test 95.............Prerequisites 95.......................................................................................Using the "Create NotesBench Mail Persons Documents" Agent 95.......Copying an Existing Mail Database 95....................................................

Cluster Mail Initialization Test 97....................................................................Prerequisites 97.......................................................................................Using the "Create NotesBench Mail Person Documents" Agent 97........NOTES.INI settings on the child client drivers 97....................................Running the test 97..................................................................................

R6IMAP Initialization Workload 99.................................................................Prerequisites 99.......................................................................................

iNotes Initialization workload 99.....................................................................Using the "Create NotesBench Mail Persons Documents" Agent 99.......Running the test 101..................................................................................

Chapter 5 Using the Test Results 101............................................NotesBench Test Results 101..........................................................................

Workload Metrics Summary 101................................................................

Viewing the Test Results 103.....................................................................Test Results Explanations 103...................................................................

Evaluating the Results 105...............................................................................Summary Performance Across Tests 105.................................................Error-Free Interval 105...............................................................................Response time 105....................................................................................Specific Results Metrics 105......................................................................Interpreting Web Statistics 107..................................................................

Evaluating System Cost 109............................................................................Price and Performance 109.......................................................................Pricing Assumptions 111............................................................................Calculating a System Price 111.................................................................For Driver Systems 111.............................................................................For the Interconnecting Network 111.........................................................For the System Under Test 111.................................................................For Destination Systems 113.....................................................................Quantity Pricing 113...................................................................................

Disclosing NotesBench Test Results 113.........................................................Disclosure Requirements 113....................................................................Disclosure Report Contents 113................................................................

Special Disclosure Report 115.........................................................................Table of Contents 115................................................................................Explanation of the Sample Disclosure Report 115.....................................Executive Summary 115............................................................................Objectives 115...........................................................................................Test Methodology 117................................................................................Data 117.....................................................................................................Analysis 117...............................................................................................Conclusions 117.........................................................................................Price/Performance 117..............................................................................Statement by Auditor 117...........................................................................Appendixes 119.........................................................................................

Auditing Organizations 119..............................................................................NotesBench Auditing Rules 119.......................................................................

General Rules - Full Disclosure Report 119...............................................General Rules - OnSite Audit 119..............................................................General Rules - OffSite Audit 121..............................................................General Rules - Partitioned Server Audits 121..........................................Auditing Rules 121.....................................................................................Cluster Mail Test 121.................................................................................R5IMAP Test and R6IMAP Test 121.........................................................NOTES.INI file settings on the SUT 123....................................................WorkGroup Test 123..................................................................................

Auditing NotesBench Results 125....................................................................Auditing Standards 125..............................................................................Auditors Checklist 127...............................................................................

Appendix A Benchmark Specification Language 129...................Commands used with NotesBench Specification Language 129.....................

ADD 129.....................................................................................................BEGINLOOP 129.......................................................................................CHANGETO 129........................................................................................CLOSE 129................................................................................................CREATEFILE 129......................................................................................DELETE 129..............................................................................................ENTRIES 131.............................................................................................FTENTRIES 131........................................................................................FTSEARCH 131.........................................................................................GETALL 131..............................................................................................NABUPDATE 131......................................................................................NAVIGATE 131..........................................................................................NEWREPLICATEDB 131...........................................................................OPEN 131..................................................................................................PAUSE 131................................................................................................POPULATE 133.........................................................................................REPLICATE 133........................................................................................REWIND 133..............................................................................................SENDMESSAGE 133................................................................................SESSIONSCLOSE 133..............................................................................SESSIONSOPEN 133................................................................................STAMP 133................................................................................................UNREAD 133.............................................................................................UPDATE 133..............................................................................................

Index 1...........................................................................................

Chapter 1 Introducing NotesBench

This chapter explains what NotesBench is and what it can do for you, as well as theNotesBench tests (workloads), the NotesBench Consortium, and the requirementsfor using NotesBench.

What is NotesBench?Lotus NotesBench is a collection of tests that simulate the behavior of Dominoworkstation-to-server or server-to-server operations. The tests return measurementsthat you can use for performance evaluation.

Third-party tools are available to accomplish other tasks: to capture the work onone server and replay it on another server; to simulate varying load and peak periodusage; and to test error states — such as, the effect on the Router when aconnection to the destination is not available.

What is a Benchmark?A benchmark is a test of the performance of a computer system. Benchmarks cantest the following:

w Two software applications running on the same hardwarew Different hardware platforms from the same vendor running the same softwarew Different releases of software on the same machinew Different systems running the same software

NotesBench provides benchmarks for the last category of this list, using Notessoftware.

1

NotesBench TerminologyBefore actively working with NotesBench, familiarize yourself with the morecommon NotesBench terminology defined in this topic.

WorkloadsWorkloads are a predetermined set of transactions to be executed by the hardwareand software under test. The software is executed by driver systems to put ameasurable load on the system under test (SUT).

PriceThe price of the SUT is an integral part of a benchmark, along with its performancemetric. The price metric used is the five-year cost-of-ownership of the system.NotesBench provides guidelines for calculating the system price.

Price/PerformancePrice/performance is a benchmark’s price and performance looked at together as aratio: price divided by performance. Price/performance lets you weigh the workdone by the software against the cost of ownership.

PerformanceThe purpose of a NotesBench benchmark is to obtain and record performancemeasurements to compare two or more systems. The systems’ performance ismeasured by a throughput metric - usually in units of work/second. For example,the performance of full text search software on each benchmarked system istypically search transactions completed per second.

ScalabilityNotesBench is designed for scalability; that is, the workloads are applicable to bothlarge and small computer systems. A scalable benchmark maintains a constantrelationship between the workload presented to a system and the capacity of thesystem. As the capacity increases, the benchmark’s workload increasesproportionally. Scalability allows the benchmark user to adjust the workload to thesize of the SUT.

3

Steady StateSteady state is the condition of the SUT that represents its true sustainableperformance with reproducible results. Any NotesBench test reaches steady statewhen the number of Notes users on the SUT is equal to the sum of the number ofclient users across all child drivers. Additionally, ten minutes should have elapsedsince all threads have become active and performed at least one successful iterationof the workload script.

What's in NotesBench?NotesBench provides these features:

w A command-line user interface for running the benchmarksw Utilitiesw Scripts to perform the benchmarksw Workloadsw Special variables for NOTES.INI filesw A checklist of disclosure informationw A checklist for auditorsw This Lotus NotesBench Guide

NotesBench User InterfaceThe NotesBench software provides a command line interface to invoke the setupand execution procedures. You invoke the NotesBench commands, withparameters, on the parent and child drivers.

NotesBench ScriptsThe NOTEBNCH command launches a script consisting of a simple procedurallanguage that presents transactions to the SUT. Each statement in the languagecorresponds to one or more Notes API functions.

For more information on the notebnch command, see the topic “NotesBenchexecutables, notebnch, notesnum, and probe” in this chapter.

NotesBench WorkloadsThe NotesBench software consists of a suite of benchmarks. Each benchmark mapsto a workload, or test, that simulates Notes workstation-to-server orserver-to-server operations. Workloads are a predetermined set of transactions tobe executed by the hardware and software under test. The software is executed bydriver systems to put a measurable load on the SUT.

3

The NotesBench ConsortiumThe NotesBench Consortium is an independent, nonprofit organization dedicatedto providing Domino performance information. The Consortium provides currentbenchmarks and performance information to ensure rapid, optimal deployment ofDomino. In addition, the Consortium serves as an industry conduit for specifyingfuture benchmarks.

NotesBench Consortium Membership Any company or corporate division except those that manufacture only softwareoperating systems is eligible for membership. Members have full access to allreports and are involved in the decision-making processes of the Consortium.

The NotesBench Consortium Web site is located at www.notesbench.org.

NotesBench Consortium Web SiteThe home page for the NotesBench Consortium is located at www.notesbench.org.There you will find information about the history of the Consortium, and the mostcurrent information, under these topics:

w Membership Applicationw Training w Member Cafe: a discussion arena w NotesBench Reports & Server.Planner Data setsw FAQsw Audit Dataw Member Directoryw Registrationw Reference Informationw Job Opportunities

5

NotesBench RequirementsDuring a typical NotesBench test, one parent driver controls one or more childdrivers that drive the SUT.

w The SUT is a Domino server. The parent driver and child drivers should haveNotes and NotesBench software installed.

w The SUT can be a partitioned server, where each partition can serve as an SUTto run the tests. The test results are reported in a separate Results directory.

w You can run Probe to the SUT on the parent driver.

Typical Driver System Configuration

For the SUT, NotesBench for Lotus Domino 6 supports all of the operatingsystems that the Domino server supports. The NotesBench software itself,however, runs on the following operating systems:

w AIX(TM)w Solaris/UNIXw Windows NT w Win2000 server on Intelw Windows XP

Additional requirements:

w Notes and Domino software must have the same release numberw Lotus Domino 6 or later

5

Network Protocol RequirementsYou may need to configure the software that implements network protocols. Thissoftware lets a parent or child driver communicate with the SUT and anydestination servers.

TCP/IP is required if you are planning to run the HTTP tests.

Memory RequirementsFor each driver, the guideline for a RAM requirement for NotesBench 6 tests isshown in the table below. The requirement is based on threads; each user is athread. The number of users per child is the number specified for that child in theparent driver’s NOTES.INI file. The number of users on the SUT is the totalnumber of users across all drivers.

5001MWorkgroup1000750KR5iNotes, R6iNotes1000750KR5Mail, R6Mail1500500KR5IMAP, R6IMAP

Maximum practicalnumber of users persystem

Memory requirements perthread

Workload

Configure the SUT so that it can run the test with the reported result continuouslyfor at least eight hours after all users are connected. Allow sufficient storage tohandle all test database and server log growth for eight hours.

7

Hard Disk RequirementsHard disk requirements apply to the SUT.

For tests that include mail routing, the steady state requires a constant sizeMAIL.BOX database.

15 MB when usingINOTES6.NTF

R6iNotes

9 MB when usingINOTES5.NTF

R5iNotes15.1 MB R6IMAP

14.5 MB when usingMAIL6.NTF

R6Mail

The growth rate perdatabase is a function ofthe ratio of the number ofusers sending mail and thenumber of recipientsreceiving mail. An increaseof 1MB per hour for theduration of the test. Thisfigure is not dependent onthe number of users. Anincrease of 100KB perhour as impacted by thevalue of the NthIterationvariable in the NOTES.INIfile.

Approximately 7.5 MB onthe SUT for each maildatabase (one database peruser).

R5MailR5IMAPWorkGroup

15.4 MB when usingMAIL6.NTF

Cluster Mail

Subsequent DiskRequirement

Initial Disk RequirementTest

For more information about the duration of the NotesBench tests and theirmeasurement periods, see the chapter “Preparing the Environment.”

7

NotesBench Environmental VariablesNotesBench uses environmental variables defined in the NOTES.INI file when itruns a script. A script presents a workload (test) to the SUT. Values for Notesenvironmental variables must be present in the NOTES.INI of each NotesBenchtest driver. Several of the variables are fixed to meet publishing standards, andsome need to be defined.

This section presents an overview of the types of variables and how the AutoInitialization function works with them.

For information about the variables needed for the parent and child drivers, and forinformation on the individual variables needed to run the individual tests, see thechapter “Preparing the Environment.”

Auto Initialization When initializing for a test, starting a run, or performing any other operation,NotesBench uses an Auto Initialization routine. Auto Initialization checks thechild’s required NOTES.INI variables. If you did not specify the variables, AutoInitialization sets them to their default values. If a variable’s value cannot bedetermined at runtime (such as the mailserver variable), NotesBench terminatesand reports the default values and the missing variables to you.

Variables Fixed for Publishing ResultsIf you want to publish your results, you must set certain variables—particularlythose that determine notes sizes and operation counts, to their fixed value. Thesevalues ensure that the test workload presents the same amount of work to allsystems. You may customize any of the fixed variables for your specific workload,although any results from such a workload are not publishable.

Variables Based on the Number of UsersThe intent of scaling rules is to maintain a proportional relationship between thecapacity of the SUT and the workload. The variables based on user numbersincrease or decrease with the value for the maximum number of users, and theworkloads in which they appear increase or decrease with the maximum number ofusers. They model the environment in a proportionally-sized organization.

The number of users determines how many threads run in the NotesBench testprocess.

User-Settable VariablesThe user-settable variables required in the NOTES.INI file depend on the workloadselected and the configuration under test. Some variables are applicable either tothe parent or child driver’s NOTES.INI file. Examples of user-settable variables

9

include defining the time child drivers run a test and specifying the path name ofthe shared directory that contains results and child control files.

In addition to setting variables in the NOTES.INI file, you need to define the Mailfile variable in the Location document.

NotesBench executables, notebnch, notesnum, and probeThis section describes briefly the tools provided with NotesBench:

w notebnchw notesnumw probe

Instructions for using the tools are included with the steps for running eachworkload.

For more information on how to use the NotesBench executables with individualtests, see the chapter “Running the NotesBench Tests.”

notebnchThe NOTEBNCH executable invokes both the workload test scripts, and theworkload initialization scripts. Invoke NOTEBNCH from the parent and childdrivers.

notesnumThe NOTESNUM executable invokes the utility that retrieves, formats, and storestest results. Run NOTESNUM from both the parent and child drivers to reset theresults file and to observe the results files.

For more information about notesnum and test results, see the chapter “ Using theTest Results” in this guide.

9

probeThe NotesBench probe option (in the NOTEBNCH command) displays andrecords the time it takes for one simulated user to open and close a discussiondatabase on the SUT. This utility is useful for observing the total perceived time tocomplete a typical Notes operation, such as opening the default view of a database,while the SUT is servicing a NotesBench workload.

For information about the other statistics that are reported, see the chapter “Usingthe Test Results.”

You can run the probe on the same system being used as the parent. By default, theprobe creates/accesses a test discussion database called TESTDISC.NSF on theSUT. The database name can be changed using the NOTES.INI setting,ProbeDatabase=. The default template used when this database is created isDISCSW50.NTF, but this, too, may be changed using the NOTES.INI settingProbeTemplate=. Every minute, the probe performs a set of actions on thedatabase—open the database, open the $all View, close the view, and close thedatabase. It also displays the total response time for these actions.

The NotesBench probe command uses two NOTES.INI environmental variableswith Domino Server.Planner:

w CPDatabase=<capacity planning database> is the name of the CapacityPlanning DatabaseNote CPDatabase must be in the form of CPdatabasename, wheredatabasename can be any text string that does not include blankspaces.

w SUTMachineID=<System Under Test Machine ID> is the Reference ID of thesystem under test. This ID is used when creating an entry in the capacityplanning database.Note SUTMachineID can be set to any text string that does notinclude blank spaces. Create the SUTMachineID variable in theMachine Id form of Server.Planner.

Both variables are used by NOTESNUM.EXE to create an entry in the CapacityPlanning Database for recording notesnum roll-up information and the probestatistics.

11

Additional Details about Probe_IntervalThe basic information about the Probe_Interval variable is located in the table inthe topic “Setting up Parent Driver NOTES.INI File” in the chapter “Preparing theEnvironment.”.

Probe_Interval is required in order to use the Multiple Run feature. The valuesrepresent the maximum number of users for each of the runs, distributed across allthe clients. Values must be nonzero integers, separated by commas. The differencesbetween the numbers should be significant. These numbers override the number ofusers on the child set by the NumClients variable.

For example, to ramp up by 100 users for each successive run, use:

w NumClients1=100w NumClients2=200w NumClients3=300w Probe_Interval=100,200,300,400,500,600w With the Parent running: notebnch maildb 3 -mrw With the Probe running: notebnch probe -mr

The result is a multi-run test with 6 iterations: 100, 200, 300, 400, 500, and 600.The distribution of users across a child driver is shown here:

3002001006006200200100500510020010040040200100300301001002002001001001Child_3 usersChild_2 usersChild_1 usersTotal UsersIteration

Configuring Network ProtocolsYou may need to configure the software that implements network protocols. Thissoftware allows a parent or child driver to communicate with the system under test(SUT).

Installing Templates on Child DriversSome tests require templates which are located on the Domino 6 server. Install theMAILn.NTF and DISCSWn.NTF templates on your child drivers. Use the templatethat corresponds to the release of Domino that you are running. To install thetemplates, copy them from the SUT and install them in the Domino data directoryof the child drivers.

11

NotesBench ScriptsA NotesBench script executed on a driver system presents a workload (test) to theSUT. You choose a script when you specify a test name in the notebnch commandon the parent driver. The scripts cannot be modified or edited — a feature thathelps ensure the consistency of the benchmarks from one SUT to another.

Each NotesBench script executes as a process of concurrently running threads.Each thread is equivalent to a NotesBench user, and each thread repeatedlyexecutes the entire script for the test. For example, if you assign 100 users to aNotesBench driver, there are 100 threads simultaneously executing that workloadscript.

NotesBench scripts consist of statements of a simple procedural language — theNotesBench benchmark specification language. Each statement corresponds to oneor more Notes API functions with a similar name.

Also, most benchmark specification language statements include Notesenvironmental variables. The statements use the values of the environmentalvariables as parameters. Scripts send Notes client transactions to the system(server) under test. Specify values for the variables in your NOTES.INI file.

Example

Here is an example of a benchmark specification language statement, found in thescript for the NotesBench R5Mail workload:

sendmessage [NormalMessageSize] [NumMessageRecipients][NthIteration]

The values for NormalMessageSize, NumMessageRecipients, and NthIterationmust be provided as NOTES.INI settings. For the R5Mail workload,NormalMessageSize, NumMessageRecipients, and NthIteration would be set to10000, 3, and 6 respectively.

For information about the procedural language used to build the scripts, see theappendix “Benchmark Specification Language” and see the chapter “Preparing theEnvironment.”

13

Viewing the NotesBench ScriptsThe text of the NotesBench scripts can be seen from the Server.Load application.To view the scripts, complete these steps.

Run the Server.Load utility. The Lotus Server.Load dialog box appears.Verify that “Built-in” is selected as the Test type.Scroll to the name of the workload for which you want to view a script.Highlight the script name and click View Script. Continue viewing scripts as necessary.

13

Chapter 2QuickStart

This chapter provides instructions on how to quickly set up a NotesBenchenvironment and how to run the NotesBench workloads.

NotesBench QuickStart - Setting up NotesBench and Running WorkloadsUse this QuickStart section to set up your NotesBench environment and quicklybegin running the NotesBench workloads.

Quick Start - Setting up the workloadMake sure that the same release of Notes Domino software and NotesBenchare installed.Set up the System Under Test (SUT).Populate the Domino Directory by Creating Person documents using the agent“Create NotesBench Mail Person Documents”.Create the mail databases on the SUT using the initialization tests and agentslisted below. The WorkGroup workload also requires a Test Discussiondatabase which is automatically created for you.w Create the mail databasesConfigure the parent driver.w You can use Probe (optional).Configure the child drivers.Test connectivity from the child driver to the SUT as follows:w Choose File - Preferences - User Preferences.w Choose Ports and then click Trace. The Trace Connections dialog box

appears.w From the “Trace options” drop-down menu, select one of the following:

“Full trace information” to display the full trace information, includingsearches through Connection documents.“Include drive messages” to display all the information from networkdrivers.

15

w (Optional) From the “Notes Log options” drop-down menu, select the levelof information for Notes to copy into the Notes Log database (LOG.NSF)while tracing the connection.

w Do one of the following:Click Trace to start the trace.Click Done to close the dialog box without tracing the connection.

Quick Start - Running a WorkloadSelect the NotesBench test to run.Set up the NOTES.INI file for the SUT and the child driver.

Note Each test contains some unique NOTES.INI settings. Refer to theNOTES.INI information for individual tests for details. The SUT uses thestandard Domino server NOTES.INI file.For more information on the NOTES.INI files for the individual tests, see Chapter3, “Preparing the Environment.”

15

Typical NotesBench ConfigurationThis configuration shows a typical NotesBench configuration. The graphic depictsthe relationship between the system under test (SUT), the child drivers, and theparent driver.

NotesBench Quick Start Table of Setup InstructionsUse the steps in this table to quickly set up NotesBench so that you can beginrunning the NotesBench tests. The general setup depicted in this table is as follows:

w This setup is for a 5000 user test, with 1000 users on each of the five childdrivers.

w Domino 6, or more recent, must be installed on the SUT, and the Notes Client6, or more recent, must be installed on the parent and child drivers.

w The parent driver and each child driver should set the results directory to thesame location, and each should have the path to the NotesBench executableincluded in their PATH statement. The NOTEBNCH command must beexecuted from the Notes directory.

17

w Each client must have a copy of the mail template in its Notes data directory.

Message Storage Format: MIME Mail System: POP3 or IMAP

R6IMAP Workload

Message Storage Format: No PreferenceMail System: Notes

R6iNotes Workload

Message Storage Format: NotesMail System: Notes

R6Mail WorkloadClusterMail Workload

From the Domino administrator, open thedatabase, NAMAGENT.NSF, and copy all of theagents into the Domino Directory (NAMES.NSF)on your SUT.From a Notes client, execute the ’CreateNotesBench Mail Person Documents’ agent tocreate directory entries. Workload-specificsettings are listed and described in the tableentries below.

All Workloads

Task: Directory Setup (Creating Users). Use theinformation in this column as appropriate for theindicated workload.

Workload NameStep 1: Create user documents in the Domino Directory.

Run the agent "Refresh all Documents".All workloadsTask: Directory Setup (Refresh All)Workload Name

MailTemplate=mail6.ntf nabentries=5000

R6Mail Workload

UseServerNAB=1NormalMessageSize=10000MessageLineSize=100NumMailNotesPerUser=100NumMessageRecipients=3 NthIteration=6Domain=yourdomainMailServer=CN=servername/O=yourdomainSUTMachineID=servername

All workloads

Task: Edit the file NBCONFIG.INI, located in theresults directory. Use the settings specified below.

Workload NameStep 2: Edit configuration files and add NotesBench settings.

17

MailTemplate=mail6.ntfnabentries=5000ClusterServer2=CN=FailoverServername/O=yourdomain

ClusterMail workload

MailTemplate=mail6.ntfR5IMAP_LOOP_N=1000 IMAIL_NO_APC=1 MailRecipientPercentUser=50MailRecipientPercentVolume=50MailRecipientBeginNumber=1MailRecipientEndNumber=5000SMTPHost=servername.yourdomain.comLDAPHost=servername.yourdomain.comIMAPHost=servername.yourdomain.com

R6IMAP Workload

MailTemplate=ServerName/DomainName!!iNotes5.ntfwebauthenticationoff=0 webpreferencesoff=0 HTTPHost=servername.yourdomain.comRecipientDomain=servername.yourdomain.comnabentries=5000

R6iNotes WorkloadStep 2: Edit configuration files and add NotesBench settings.

NodeName=clientnameDebug_Outfile=r:\resn (n=1,2,3, ..)ResultsDirectory=r:ClientHost=clientname.yourdomain.com

All Workloads

Task: Edit the file, NOTES.INI, on the clients.Use the settings listed in the table entries below.

Workload Name

NodeName=parentnameDebug_Outfile=r:\resparent ResultsDirectory=r:Runtime=810 ThreadStagger=1 ChildStagger=20 NumClients1=1000NumClients2=1000

All workloads

Task: Edit the file, NOTES.INI, on the Parentdriver. Use the settings listed in the table entriesbelow.

Workload Name

19

NumClients3=1000 NumClients4=1000 NumClients5=1000

Task: Edit the file, NOTES.INI, on the Parentdriver. Use the settings listed in the table entriesbelow.

Workload Name

Parent: ‘notebnch R6IMAPInit 5’R6IMAPInitParent: ‘notebnch Inotesinit 5’InotesinitParent: ‘notebnch Mailinit 5’R6MailInit

Child 1: ‘notebnch -c 1Child 2: ‘notebnch -c 2...

All workloadsTask: Create and populate mail files.Workload Name

Step 3: Create mail files by running NotesBench with an initialization workload.

Parent: ’notebnch Clumailinit 5’ClusterMailInit

After the mail files are created, run the agent:'Update All Mail DBs to include Owner(mail1, mail2, ...)'

Inotesinit

Task: Update the ACL of the Mail files toinclude the owner.

Workload Name

Parent: ’notebnch clumail 5’Cluster MailParent: ‘notebnch R6IMAP 5’R6IMAPParent: ‘notebnch R6iNotes 5’R6iNotesParent: ‘notebnch R6Mail 5’R6Mail

Child1: 'notebnch -c 1'Child2: 'notebnch -c 2'...

All WorkloadsTask: Run this Measurement WorkloadWorkload Name

Step 4: Run the Test.

19

Chapter 3 Preparing the Environment

This chapter contains the setup procedures that you use to set up the NotesBenchenvironment. It also contains sample NOTES.INI files.

Planning the Duration of the NotesBench TestsWhen planning for your NotesBench tests, keep in mind the following:

w Check the requirements for the test. You may need to allow time to run otherprograms before you can begin a NotesBench test.

w Note the minimum time required to run the test. This number appears in theinstructions for each test. The duration requirement ensures that all operationsare carried out before the measurement is complete. For example, deferredwrites to cached disk blocks must occur within the measurement interval.

For information on passing an audit, see the chapter “Using Test Results.”

These terms apply:

w Steady state represents the true sustainable performance of the system undertest (SUT) with reproducible results. Steady state occurs when all the users areup and running on the SUT. Specify the number of users for each child driverin the NOTES.INI file of the parent driver. The SUT must have an error-freerun.

w The measurement period can begin any time after a NotesBench test has metthe test duration requirements. The measurement period must produce asuccessful run on the SUT.

w A test has completely ramped up when all the users have completed at leastone iteration of the test script. There is no time limit, but you must include theramp-up time for each test in the Full Disclosure Report.

For a sample of the Full Disclosure Report, see the chapter “Using the TestResults.”

21

Installing Lotus Domino and Notes software for use with NotesBenchInstall the Domino server on the system under test (SUT).Install the standard Notes client on all drivers. Each driver may use the sameor different Notes IDs.

Installing NotesBench SoftwareOn all systems except UNIX

Install NOTEBNCH.EXE and NOTESNUM.EXE on the parent driver in thesame directory as the Notes client software (for example: c:\Notes).Install NOTEBNCH.EXE on each child driver in the same directory as theNotes client software (for example: c:\Notes).

Installing NotesBench on a UNIX system Install NOTEBNCH and NOTESNUM. For example:/opt/lotus/notes/latest/hppa/ Create symbolic links. For example: ln -s /opt/lotus/bin/tools/startup/opt/lotus/bin/notebnch

Scaling NotesBench testsScaling maintains a proportional relationship between the capacity of the SUT andthe workload, represented by the NotesBench tests. NotesBench workloads arescaled by the number of users simulated.

NotesBench runs its tests as threads. Assign a predetermined number of Notesusers to each child driver. Each user is represented by one thread. If you assign 100users to a NotesBench driver, 100 threads simultaneously run the script. Eachthread can run many iterations of the entire test script.

NotesBench runs its tests (workloads) as threads. The number of threads isdetermined by assigning a predetermined number of Notes users to each of thedriver systems. Each user is represented by one thread. If you assign 100 users to aNotesBench driver, 100 threads simultaneously execute the workload script. Eachthread can run many iterations of the entire test script.

21

Determining the Number of UsersThe maximum number of users for a child driver may vary. Use the estimatingtechnique described below to determine the maximum number of threads the drivercan simulate without a significant increase in response time.

You can run the Idle Usage test to establish the number of connections to theserver.

Estimating the maximum number of users for a child driverComplete this procedure for each unique child driver configuration and for eachtest, except the Idle Usage test.

Run each NotesBench test with a single child driver. In the parent driver NOTES.INI file, enter a number for the NumClientssetting, that is, an estimate of the maximum number of users for the childdriver. Increase the NumClients setting with each run, until you determine themaximum number of threads the driver can simulate without a significantincrease in response time.

If the child drivers have the same configuration (CPU, clock speed, cache, mainmemory, and operating system), perform this procedure on only one child driver. Ifthe child drivers are not identical, add child drivers, and determine the appropriateNumClients setting for each. The maximum number of users is the sum of all of theNumClients settings.

Varying the Number of Users with Multiple Run (MultiRun)Multiple Run allows you to test and collect results with an incrementing maximumnumber of users.

You must use the Probe function with Multiple Run. The probe looks for thespecified number of users (the maximum number of users for that test), and thencollects data for a specified period of time. At the end of that time, the probe issuesa stop, and the parent driver ramps down the child drivers. PROBE stores the dataand resets the databases. NotesBench then repeats the process with the next groupof users. To stop the test, press ESC on each child driver, the parent driver, and theprobe.

To use Multiple Run, determine the maximum total number of users, the number ofintervals, and the number of users to add at each interval. Use the Probe_Intervalsetting in the parent driver NOTES.INI file. For example, if you have a totalmaximum of 400 users across two child drivers, you could set up a multiple run totest at 100, 200, 300, and 400 user intervals. These user numbers override thosespecified in the NumClients setting in the parent driver NOTES.INI file.

In summary, the procedure for using Multiple Run is:

23

w Set up the environment as if testing for the total number of users, that is, thesum of all NumClients.

w Initialize for the test as you would for a single run of the test. w Run NOTEBNCH and NOTEBNCH Probe. Include -mr (indicating multiple

run) as the last parameter on the command line. The parent driver screen displays Multiple Run messages that report when theprobe is in Multi Collection mode, and when each stage is complete.

Note The ProbeRunTime interval defaults to two hours if you do not entera value.The difference between the number of users from one interval to the next should belarge enough to represent a significant load on the system.

Multiple Run does not work with partitioned servers.

Setting Up NotesBenchYou will be completing the following tasks when setting up your NotesBenchenvironment.

w Setting up the System Under Test w Setting up the Drivers w Setting Up Location Documents for Parent and Child Drivers w Setting up the Parent Driver NOTES.INI File w Setting up Child Driver NOTES.INI Files w The NotesBench Configuration File NBCONFIG.INI w Setting up destination servers w Creating Mail Databases for NotesBench

Setting up the System Under TestThe system under test (SUT) is the target for the NotesBench tests.

Check that the correct version of the Lotus Domino software is installed andoperational. For detailed information on installing Domino servers, see the Domino6 or Domino 6.5 Administrator online help or the printed documentation, InstallingDomino Servers.

For information on the individual tests, see the chapter “Running the Tests.”

23

Setting up the DriversDrivers run the NotesBench tests and generate the transactions on the SUT. Everyconfiguration must include at least one parent driver and one child driver.Additional child drivers are optional.

Be sure that you have installed all required Lotus Domino, Lotus Notes, andNotesBench software on the parent and child drivers.

NotesBench for Domino 6 supports a maximum of 256 drivers and a maximum of16,384 simulated users per driver. These numbers are valid assuming that yourconfiguration can support them. Generally speaking, 2 users can be supported forevery MB of RAM on the driver. For example, a 512MB Pentium system cansupport approximately 1000 mail users.

To set up the parent and child drivers, complete these steps:

Configure Location documents to the SUT.Configure the NOTES.INI file for the parent driver.Set up the file, NOTES.INI, for the child driver. To assign identical settings tomultiple child drivers, use NBCONFIG.INI.Set up the desired ID file on the driver. You can use the same ID on alldrivers.Configure the Network protocol for any drivers requiring it.Install the templates for the tests on the child drivers. The templates are on theDomino server.

For information on any of the “setup” procedures, see the chapter “Preparing theEnvironment.”

Setting Up Location Documents for Parent and Child DriversYou need to define mail server specifications in the Location document of both theparent and child drivers.

Note In this procedure, the SUT is called myservername/mydomainnameto provide a cohesive example.

To define the Location document:w From the Notes client, click File - Mobile - Locations and then click New -

Location. On the Basics tab, define these settings:w Location Type: Local Area Networkw Location Name: myservername/mydomainname

25

On the Servers tab, define these settings:w Home/Mail server: myservername/mydomainnamew Passthru server: myservername/mydomainnameOn the Mail tab, define these settings:w Mailfile Location:myservername/mydomainnamew Mailfile:mail database\mailfile.nsf (the path to the mail file)Save and close the document.Click on the Location document name in the lower right corner of your Notesapplication. Open the Location document that you just created by double clicking on thatLocation document name. The following NOTES.INI settings are updated: w MailServer=w Location= Close the document.

Setting up the Parent Driver NOTES.INI FileOpen the file, NOTES.INI, on the parent driver system and set the Notesenvironment variables as follows:

Directory for temporary files on the driver.Append the directory name to the pathname specified for the Directory variable inthe NOTES.INI file. Then, include the tmpvariable separately in the NOTES.INI file.

Tmp=

Used for an SUT that is a partitionedserver. Create a Results directory for eachpartition. Specify the path to each partition,using the full path (including node). Eachpartition has its own parent driver; includethis variable only in the NOTES.INI file ofthe one parent that runs NOTESNUM forall partitions. Example of an SUT with twopartitions:w PResultsDir1 = \\fileserver\p_resd1w PResultsDir2 = \\fileserver\p_resd2

PResultsDir<num>

Use the full path (including node) of theshared directory that contains the Resultsfile and child control files. This directorymust be accessible by the parent and allchild drivers. Example:ResultsDirectory=\\myserver\results

ResultsDirectory=Value to EnterNotes Environment Variable

25

(Optional, but recommended)Specifies, inChildStagger=

(Optional) Number of minutes to run thetest. Although you can set RunTime onboth the parent and the child driver, fortesting purposes it’s best to have the parentcontrol the start and end times for the testand to disable the RunTime setting on thechild.A value of 0 or the omission of this settingallows the test to run indefinitely. Multiple Run tests use ProbeRunTimeinstead of RunTime.

RunTime=

Time interval in minutes that the probeallows before checking for all users runningwhen using Multiple Run. The minimumtime interval is 5.

ProbeIntervalDelay=

The maximum number of users for eachrun, distributed across all the clients.Values must be nonzero integers, separatedby commas. The differences between thenumbers should be significant. Thesenumbers override the number of users onthe child set by the NumClients setting.For more information, see the topic"Additional Details about Probe_Interval"in the chapter "Introducing NotesBench" inthis guide.

Probe_Interval=

This represents the number of recipients foreach message.Example: NumMessageRecipients=3

NumMessageRecipients=q

This represents the initial number of mailmessages per user.Example: NumMailNotesPerUser=100

NumMailNotesPerUser=

The number of threads to start on the nthchild driver.Example: NumClients1=100

NumClientsN=

This represents the typical size, in bytes, ofeach message that is sent. Example: NormalMessageSize=10000

NormalMessageSize=

Node of the parent driver; used to identifythe output statistics. Example:NodeName=MYPARENT

Nodename=

Do not confuse this variable with thesystem temp variable.Example: Tmp = c:\Temp

Value to EnterNotes Environment Variable

27

(Optional) Identifier name of system undertest.

SUTMachineID=

Length of time a child driver waits after itis ready to ramp down, before it initiatesramp down.

ChildTerminationDelay

minutes, the delay between the start timesof each successive child.ChildStagger=n1, p1-q1; n2, p2-q2; ...w Where n1, n2 are the stagger intervals

in minutesw Where p1-q1 represent the children in

the ID rangesEvery child must be covered in thedefinition, and assigned only one staggertime. You can use a single value instead ofa range for the child ID.Example: ChildStagger=3w All child drivers are staggered 3

minutesExample: ChildStagger=3, 1-10; 5, 11-20;8, 21-30w Child drivers 2-10 are staggered every

3 minutes (Child 1 is never staggered)w Child drivers 11-20 are staggered every

5 minutesw Child drivers 21-30 are staggered every

8 minutesBefore starting the parent, be sure that allchild drivers are running and waiting forinstructions from the parent.

Value to EnterNotes Environment Variable

27

(Optional) Number of minutes to run theworkload for that interval when usingMultiple Run. If this variable is notspecified, probe runs for 2 hours.

ProbeRunTime=Value to EnterNotes Environment Variable

29

Setting up Child Driver NOTES.INI FilesOpen the NOTES.INI file on each child driver system and set the Notesenvironmental variables as follows:

(Optional) Number of minutes to run thetest. Although you can set RunTime onboth the parent and child driver, fortesting purposes it is best to have theparent control the start and end times forthe test, and disable the RunTime settingon the child. A value of 0 or omission ofthe setting runs the test indefinitely. Multiple Run tests use ProbeRunTimeinstead of RunTime.

RunTime=

The number of recipients for eachmessage.Example: NumMessageRecipients=3

NumMessageRecipients=

The initial number of mail messages peruser. Not required for the Idle Usage test.Example: NumMailNotesPerUser=100

NumMailNotesPerUser=

The typical size, in bytes, of eachmessage that is sent. Not required for theIdle Usage test.Example: NormalMessageSize=10000

NormalMessgeSize=

Node of the child driver to identify theoutput statistics.Example: NodeName=MYCHILD

NodeName=

Directory for temporary files on the childdriver. Append the directory to the pathspecified for the Directory setting in theNOTES.INI file. Then include the Tmpvariable separately in the NOTES.INI file.Example: Tmp = c:\Temp

Tmp=

ResultsDirectory\res number on eachchild driver, where res is the file for thechild and number is the child number.Example:Debug_Outfile=\\fileserver\results\res1

DEBUG_OUTFILE=

Full path (including node) of the shareddirectory that contains results and childcontrol files. This directory must beaccessible by the parent and all children.Example:ResultsDirectory=\\myserver\results

ResultsDirectory=Value to EnterNotes Environment Variable

29

(Optional) Specifies the delay, in seconds,between the launching of each thread on achild. If you set the value to 0 or omit thesetting, there is no delay when launchingthreads. Decimal values are allowed.Example: ThreadStagger=0.5 will launcha thread every half-second.

ThreadStagger=Value to EnterNotes Environment Variable

The NotesBench Configuration File NBCONFIG.INI Note You are not required to use the NotesBench Configuration file,NBCONFIG.INI. Use of this file is optional.Use the NotesBench configuration file (NBCONFIG.INI) to assign identicalsettings to multiple child drivers.

The NBCONFIG.INI file is located in the Results directory. NotesBench uses theNBCONFIG.INI file to override settings on all NotesBench tasks. At the beginningof a NotesBench run, each NotesBench task reads this file and overwrites its localNOTES.INI file with the settings in the NBCONFIG.INI file. For example, theNBCONFIG.INI file could contain the following settings:

w NumMessageRecipients = 5w NthIteration = 3w NormalMessageSize = 500w MailServer = MyServer/MyDomain

Every NotesBench driver would then incorporate the settings in its ownNOTES.INI file.

31

Setting up destination serversSet up the destination servers for those tests that require them.

Install Domino software on each server.Run the agent “Create NotesBench Person Documents.” Replicate the Domino Directory among all destination servers, the systemunder test (SUT), and the drivers.

Creating Mail Databases for NotesBenchUse the appropriate mail initialization workload to create, set up, and populate themail databases. The mail database owner is set to the mail file name — forexample, user mail2 is the owner of mail database mail2.nsf.

CluMailInitClusterMailIMAPInitR5IMAPR6IMAPInitR6IMAPiNotesInitR6iNotes, R5iNotesMailInitR6Mail, R5Mail, WorkGroup

Use this initialization workload to set upthe mail database

Workload

Be sure that the desired mail template is available on each child driver on theDomino data directory.Run the appropriate initialization workload to create the mail databases. Copy the agents in NAMAGENT.NSF to the SUT’s Domino Directory(NAMES.NSF).Run the agent “Create NotesBench Mail Person Documents” from the SUT’sDomino Directory to create the Person documents.

The Person documents use the MAIL*.NSF files on the SUT as the users’ mailfiles.

31

Using the "Create NotesBench Mail Persons Documents" AgentThe agent that you can use to create Person documents is in NAMAGENT.NSF.

Use the “Create NotesBench Mail Person Documents” NAMAGENT.NSF agenton any platform to create Person documents that use the iNotesInit’s mail*.nsf fileson the SUT as the users’ mail files. Copy NAMAGENT.NSF to the DominoDirectory and run it from there. See the Using this Database menu for moreinformation about the agents in NAMAGENT.NSF.

33

Using the NotesBench Results databaseThe NotesBench Results database is a capacity-planning tool that can help youachieve the best configurations for your environment.

Before you can use NotesBench Results, you need to generate NotesBench dataand submit it to the NotesBench Consortium Auditor.

w Use the NotesBench Results form DSPV.NTF to create the vendor database.Enter the database name in the NOTES.INI file (CPDatabase setting).

w Create an SUTMachineID name for each server configuration tested. Includethis name in the NOTES.INI file on the parent driver.

w Create a machine identifier form for each MachineID.w Run Probe with each test.w Use NOTESNUM to roll up the results into the vendor database. w Submit the NOTESNUM results, a Full Disclosure Report, and a Machine

Identifier form to the NotesBench Consortium Auditor. If the auditor acceptsthe database, the auditor applies a digital signature to the database, andNotesBench Results can then process the database.

To create the Machine Identifier form, follow these steps.

Note If you already ran NOTESNUM to create a Vendor Database, beginwith Step 3.

Download the vendor database template. Create a database using the template.Open the database. Choose Create - Machine Identifier. Create a name for the machine. On the Machine Identifier form, enter the appropriate information. Repeat steps 4 — 6 for any additional configurations whose data sets will beincluded in the database. Save and close the form.

The database may contain multiple vendor data forms for each Machine Identifierform.

33

The three NotesBench Results databases are attached to the NotesBenchdocumentation database. The files are:

w DSPA.NTF, the NotesBench Results Analystw DSPD.NTF, the NotesBench Results Decision Makerw DSPV.NTF, the NotesBench Results Vendor

35

Setting Up the NOTES.INI File for NotesBenchThe parent and child drivers use specific NOTES.INI settings for each NotesBenchtest. This section includes a table that describes all the NOTES.INI settings and asample NOTES.INI file for each NotesBench test.

In addition, you can update the local NOTES.INI for each child driver bymodifying the NBCONFIG.INI file in the Results directory.

Auto InitializationAuto Initialization checks the required NOTES.INI settings on the drivers and setsthem to their default values if no value is specified. If NotesBench cannotdetermine the value at runtime, NotesBench displays a message at the driverconsole, then stops.

Settings to use when publishing resultsIf you want to publish your NotesBench test results, you must use specific valueswith some NOTES.INI settings. These values ensure that the test presents the sameworkload to all systems. You can change the required defaults; however, youcannot publish the results from a test that does not use those defaults.

Variables that you setThe settings that you specify depend on the test and system under test (SUT). Somesettings apply to either the parent or the child driver’s NOTES.INI file.

For more information on setting up drivers, see the topic “Setting up the Drivers”in this chapter.

35

NOTES.INI settings used by NotesBench

R5IMAP, R6IMAPHost name of the LDAP serverLDAPHost

R5IMAPR6IMAP

Host name of the IMAP serverunder test.

IMAPHost

R5iNotes, R6iNotes,iNotesInit,WorkGroup

Name of the host system for HTTPrequests, for example,thisserver.acme.com.This is a required entry for all Webworkloads.

HTTPhost

WorkGroupTemplate used to create thediscussion database.

DiscTemplateWorkGroupName of the discussion database.DiscussionDB

Name of the Notes Data Directory.For example,Directory=C:\notes\data

Directory

Child driverNOTES.INI

ResultsDirectory/res num on eachchild driver, where res is the filefor the child and num is the childnumber.

Debug_Outfile

ProbeName of the Capacity Planning(output) database. CPDatabase isused with SUTMachineID byNOTESNUM.EXE to create anentry in the Capacity PlanningDatabase for recording notesnumrollup information and probestatistics.

CPDatabase

ClusterMailName of the server to be used formail failover testing underworkload.

ClusterServer2

R5IMAP, R6IMAPWorkGroup

Name of the client host system.ClientHost

R5IMAPR6IMAPWorkGroup

Length of time a child driver waitsafter it is ready to ramp down,before it initiates a ramp down.This variable is used to support anorderly ramp down of the childprocesses, thereby reducing thelikelihood of a server panic. Thisvalue, in milliseconds, is multipliedby the child ID. The default valueis 5000.

ChildTerminationDelay

Parent driverNOTES.INIProbe

Specifies the delay, in minutes,between the start times of eachsuccessive child launch.

ChildStaggerWhere usedDescriptionSetting

37

R5Mail, R6MailiNotesInit,WorkGroupClusterMail

Name of the server on which themail database is located.

MailServer

R5IMAPR6IMAP

Optional setting used to weigh thedistribution of mail amongrecipients. For example, if youwant 50% of the mail to be sent tothe first 10% of the users, enterthese variables: w

MailRecipientPercentVolume=50

w Mail RecipientPercentUser=10

MailRecipientPercentVolume

R5IMAPR6IMAP

Optional setting used to weigh thedistribution of mail amongrecipients. For example, if youwant 50% of the mail to be sent tothe first 10% of the users, enterthese variables:w MailRecipientPercent

Volume=50w MailRecipientPercent

User=10

MailRecipientPercentUser


Restricts the selection of mailrecipients. For example, to restricta test to the first 100 mailrecipients, enter:MailRecipientEndNumber=100

MailRecipientEndNumber

R5IMAPR6IMAPWorkload

Restricts the selection of mailrecipients. For example, to restricta test to the first 100 mailrecipients, beginning with the firstrecipient, enter: MailRecipientBeginNumber=1MailRecipientEndNumber=100

MailRecipientBeginNumber

Value based on the Locationdocument set up using a Notesclient on the driver system.

LocationWorkGroupunder test.Where usedDescriptionSetting

37

All workloadsThe NotesBench database directorywhere the mail files will reside onthe SUT. For example, settingNB_DBDIR = mail\ allows for thecreation of all mail files in the maildata directory relative to the Notesdata directory.Note NB_DbDir is an optionalsetting.

NB_DbDir

Name of the Domino Directory.For example, to specify the AcmeAddress Book, specify thefollowing: Names=Acme Address Book

Names


Represents the maximum numberof characters in each line of anSMTP message. You must use thevalue 100 if you plan to publish thetest results.

MessageLineSize

R5MailR6MailWorkGroup

The maximum number ofdocuments to be deleted from thecurrent database if NBTestResethas been defined. For example, todelete no more than 210 documentsfrom the current database, specifythe following: MaxDocToDelete=210

MaxDocToDelete

R5Mail, R6MailWorkGroup

Name of the mail template (forexample, MAIL6.NTF) used tocreate the mail database.

MailTemplateWhere usedDescriptionSetting

39

All workloadsControls the frequency with whicha particular command/action isexecuted.For example, mail is sent on thenth iteration (n times the executionof the workload).

NthIteration

R5IMAP, R6IMAPR5Mail, R6MailWorkGroup,ClusterMail

Represents the size of a typicalSMTP mail message. You must usethe value 1000 if you plan topublish the test results.

NormalMessageSize

For the child driver, NodeName isthe machine name displayed in thestatistics line of the child driver’sRES file.For the parent driver, NodeName isthe name of the driver system toidentify the output statistics.

NodeName

R5MailR6MailWorkGroup

Define this variable to deleteexisting documents and generatenew documents at the start of atest. Possible values are: w NBTestReset=1 — To delete

existing documents andgenerate new documents at thestart of a test

w NBTestReset=0 — To leavethe database “as is” at the startof a test

The number of newly-generateddocuments is defined by theNOTES.INI valueNumMailNotesPerUser.

NBTestResetWhere UsedDescriptionSetting

39

Parent driver’sNOTES.INIChild driver’sNOTES.INI

Full path of the shared directorythat contains the RESULTS filewhen testing a partitioned server.The num is the server number.Enter this setting in the parentdriver's NOTES.INI file.

PResultsDir<num>

WorkGroupThe number of shared notes(documents) to be populated into ashared database (TESTDISC.NSF).

NumSharedNotes

WorkGroupInitial minimum number of notes inthe driver’s local replica of theshared system’s database.

NumNotesLocalReplica

Parent driver’sNOTES.INIChild driver’sNOTES.INIR5IMAP, R6IMAPR5Mail, R6MailWorkGroup

Fixed variable that represents thenumber of recipients for eachSMTP message and invitation. Youmust use the value 3 if you plan topublish the test results.

NumMessageRecipients

All workloadsRepresents the initial number ofNotes mail messages per user. Youmust use the value 100 if you planto publish the test results.

NumMailNotesPerUser

Parent driverNOTES.INIProbe

Number of client subprocesses tocreate on the child driver. Eachthread represents one Notes userpresenting Notes transactions to theSUT.For example,w NumClients1=50w NumClients2=75w NumClients3=100The parent will launch child 1 with50 threads, child 2 with 75 threads,and child 3 with 100 threads.

NumClientsWhere usedDescriptionTable

41

ProbeRuns the ProbeTestIncrementCmdbefore and after Probe iteration.w ProbeTestIncrementCmd1=w ProbeTestIncrementCmd2=w ProbeTestIncrementCmd3=Sample uses include: w ProbeTestIncrementCmd1= sh

stat mailw ProbeTestIncrementCmd2= sh

stat semw ProbeTestIncrementCmd3= sh

stat Domino.Requests.*

ProbeTestIncrementCmd

ProbeThe name of the template to beused to create the ProbeDatabase.If a name is not specified, thefollowing values are used: w For non-Internet probe:

DISCW6NTFw For Internet probe:

MAIL6.NTF

ProbeTemplate

ProbeNumber of minutes to run probefor that interval when usingMultiple Run. If this variable is notspecified, probe runs for 2 hours.

ProbeRunTime

ProbeTime interval in minutes that probeallows before checking for all usersrunning when using Multiple Run.The minimum time interval is 5.

ProbeIntervalDelay

ProbeRepresents the number of users(threads) to launch across all childdrivers in a multiple-run (-mr) test.Enter this setting in the parentdriver's NOTES.INI file. Thesevalues override the number ofusers on the child set by theNumClients setting.

Where usedDescriptionTableProbe_Interval

41

Child driver’sNOTES.INI

Specifies the delay, in seconds,between the launching of eachsimulated user's thread on a child.A value of 0 or omission of thisvariable indicates no delay whenlaunching threads.

ThreadStagger

R5IMAP, R6IMAPIMAPInit,R6IMAPInitWorkGroup

Host name of the SMTP serverunder test.

SMTPHost


Specify the number of minutes torun the workload. For example, torun a workload for 100 minutes,specify the following: RunTime=100

RunTime


Location of the NotesBenchresults. Use the full path name,including the node name, of theshared directory that containsresults and child control files. Thisdirectory needs to be accessible bythe parent and all children. Enterthis variable in the NOTES.INI fileof the parent driver.

ResultsDirectory

R5IMAP, R6IMAPWorkGroup

Domain name of the SMTP mailmessage recipient.

RecipientDomain

ProbeThe number of minutes that theprobe task waits after the SUT hasreached the required number ofusers before it begins probing.

ProbeWaitTime

ProbeThe name of the view or folder thatthe probe task opens in thedatabase named in the“ProbeDatabase” setting in theNOTES.INI file. For example, thefollowing executes a probe on theInbox view of the databaseMAIL1.NSF.ProbeDatabase=MAIL1.NSFProbedView=$INBOX

ProbedViewWhere usedDescriptionSetting

43

Use this command to specifywhether authentication is active.Possible values are: w WebAuthenticationOff=1 —

To run without authenticatingWeb requests

w WebAuthenticationOff=0 —To authenticate Web requests

WebAuthenticationOff

Specifies whether to use theserver’s Domino Directory.Possible values are:w UseServerNAB=0 — Do not

use the server’s DominoDirectory

w UseServerNAB=1 — Use theserver’s Domino Directory

UseServerNAB

Parent driver’sNOTES.INIWorkGroup

Specify a directory for temporaryfiles on the child driver systems.Append the directory name to thepath name specified for theDirectory variable in NOTES.INI. This variable is not required for theIdle Usage Test or the Web tests. To specify a temporary directorycalled nb_temp, located on drive C,enter this value: Tmp=C:\nb_temp

TmpWhere usedDescriptionSetting

43

R6iNotes, R5iNotesUsed for Web workloads toprovide trace for debugging.Possible values are: w WebDebugOn=0 — No

tracingw WebDebugOn=1 — Minimal

tracing including errors,HTTP warnings, andtime-outs. This is therecommended setting.

w WebDebugOn=2 — Detailedtracing including memoryallocation/dealloc

w WebDebugOn=4 — Verbosetracing including HTTPsessions and connections

WebDebugOnWhere UsedDescriptionSetting

45

Sample NOTES.INI files for NotesBench testsEach NotesBench workload has a sample NOTES.INI file that you can use as aguideline when setting up your own NOTES.INI files.

w R6Mailw R6iNotesw R6IMAPw R5Mailw R5iNotesw R5IMAPw ClusterMailw WorkGroup

In addition to the sample NOTES.INI files for workloads, we also provide thefollowing other sample NOTES.INI files.

w Child driver sample NOTES.INI file w Parent/Probe Driver sample NOTES.INI file w MakeReplicas Initialization Script sample NOTES.INI file

Child driver sample NOTES.INI fileThe files shown in this section are examples only; they should not be used as youractual NOTES.INI files.

The following three variables should be set for all child drivers:

w NodeName w Debug_Outfilew ResultsDirectory

45

A child driver’s NOTES.INI file could contain the above three variables, as well asany others that apply, as shown here:

ResultsDirectory=r:\resultsDEBUG_OUTFILE=r:\results\res1NodeName=gehrigTmp=MailDBWebDebugOn=1httphost=MailTemplate=NthIteration=NumMessageRecipients=NormalMessageSize=WebAuthenticationOff=

Parent/Probe Driver sample NOTES.INI file;NotesBench Specifics SUTMachineID=CLONECPDatabase=CPplay.nsfNodeName=parentResultsDirectory=r:\resultsDEBUG_OUTFILE=r:\results\parentTmp=temphttphost=you_change_this.comProbe_Interval=50,100,150ProbeRunTime=60ProbeIntervalDelay=5ProbeWaitTime=5ChildStagger=5NumClients1=50NumClients2=100NuMClientsN=100ChildStagger=5NumSharedNotes=100;End of NotesBench Specifics

47

R6Mail Test sample NOTES.INI fileNormalMessageSize=10000 MailServer=server/domainMailTemplate=mail6.ntfNthIteration=6NumMessageRecipients=3NumMailNotesPeruser=100(optional)NBTestReset=1MaxDocToDelete=1000000

Note If a Mail Server is specified in the Location document, the servername in the Location document is used, and the setting in the NOTES.INIfile is disregarded. The Mail Server field is on the Server tab.

47

R6IMAP Test sample NOTES.INI file[Notes] MailTemplate=mail6.ntfIMAIL_No_APC=1 DOMAIN=IRISTSTmp=tmpThreadstagger=2 DEBUG_OUTFILE=z:\res1ResultsDirectory=z:\UseServerNAB=1NABEntries=5000NBTestReset=0NodeName=popsMailRecipientPercentUser=50MailRecipientPercentVolume=50MailRecipientBeginNumber=1MailRecipientEndNumber=5000R5IMAP_LOOP_N=1000 NthIteration=6ClientHost=pops.iris.comSMTPHost=grigsby2k.iris.comLDAPHost=grigsby2k.iris.comIMAPHost=grigsby2k.iris.comRecipientDomain=grigsby2k.iris.comMessageLineSize=100NormalMessageSize=50000NumMailNotesPerUser=100NumMessageRecipients=3childstagger=100Runtime=90000

49

R6iNotes Test sample NOTES.INI fileThis is an example of the NOTES.INI settings for this test:

HTTPHOST=yoursys.yourhost.comRecipientDomain=yourhost.yourdomain.comNthIteration=6 NormalMessageSize=10000 NumMessageRecipients=3 NumMailNotesPeruser=100 MailServer=server/domainMailTemplate=iNotes6.ntfThreadStagger=1NBiNotes=1 (optional) WebDebugOn=1 WebAuthenticationOff=1WebPreferencesOff=1

R5Mail Test sample NOTES.INI fileNormalMessageSize=10000 MailServer=server/domainMailTemplate=mail50.ntfNthIteration=6NumMessageRecipients=3NumMailNotesPeruser=100(optional)NBTestReset=1MaxDocToDelete=1,000,000

Note If a Mail Server is specified in the Location document, the servername in the Location document is used and the setting in the NOTES.INIfile is disregarded. The Mail Server field is on the Server tab.

49

R5IMAP Test sample NOTES.INI fileNormalMessageSize=10000 MailRecipientPercentUser=50MailRecipientPercentVolume=50MailRecipientBegin Number=1MailRecipientEndNumber=2000MailRecipientEndNumber (should be set to maximum entries in the DominoDirectory)R5IMAP_LOOP_N=1000;R5IMAP_LOOP_N (set sufficiently high to prevent the script from exiting —set to approximately 4 * the number of hours of run time)NthIteration=6ClientHost=hostname_of_the_clientSMTPHost=hostname_of_your_smtp_server_under_testLDAPHost=hostname_of_your_ldap_server_under_testIMAPHost=hostname_of_your_imap_server_under_testRecipientDomain=name_of_your_domainMessageLineSize=100NumMailNotesPerUser=100NumMessageRecipients=3

51

R5iNotes Test sample NOTES.INI fileThis is an example of the NOTES.INI settings for this test:

HTTPHOST=yoursys.yourhost.comRecipientDomain=yourhost.yourdomain.comNthIteration=6 NormalMessageSize=10000 NumMessageRecipients=3 NumMailNotesPeruser=100 MailServer=server/domainMailTemplate=iNotes6.ntfThreadStagger=1NBiNotes=1 (optional) WebDebugOn=1 WebAuthenticationOff=1WebPreferencesOff=1

ClusterMail Test sample NOTES.INI fileNormalMessageSize=1000 NumMailNotesPerUser=100AttachmentSize=500000LargeMessageSize=32000NumMessageRecipients=3AttachFileDirectory=[user defines this parameter]FilePattern=[user defines this parameter]ExtensionType=[user defines this parameter]NumSharedNotes=100(optional) NBTestReset=1

51

WorkGroup test sample NOTES.INI filetmp=tmp MailServer=server/domainMailTemplate=mail50.ntfNthIteration=6NumSharedNotes=100NormalMessageSize=10000NumMessageRecipients=3MessageLineSize=100DiscussionDB=testdisc.nsfDiscTemplate=discsw50.ntfNumNotesLocalReplica=100SMTPhost=hostname_of_your_smtp_server_under_testclienthost=myclient.mydomain.comrecipientdomain=mydomainLDAPhost=hostname_of_your_ldap_server_under_testHTTPhost=hostname_of_your_http_server_under_testSMTPDomain=yoursys.yourhost.comMaxDocToDelete=1,000,000NumMailNotesPerUser=NBTestReset=1

53

MakeReplicas Initialization Script sample NOTES.INI fileClusterSUT1=CN=SUT/O=companyClusterSUT2=CN=SUT/O=companyClusterSUT3=CN=SUT/O=company

53

Chapter 4 Running the Tests

This chapter explains how to run each of the NotesBench tests, or workloads, andexplains the benefits of each.

General Instructions for Running all NotesBench TestsThese general procedures apply to the tests. Detailed information follows.

w Initial Check and Auto Checkw Using NOTEBNCH to run the testsw Using NOTESNUM to obtain the resultsw Monitoring the testsw Using Probew Considerations for Debug_Outfile

Initial Check and Auto CheckThis topic explains the NotesBench initialization check routine and the automaticcheck routine.

Initial CheckEach test has an initialization routine that you invoke at the parent driver beforerunning the test itself. These routines perform various functions, depending on thetest. Instructions for using them are included with the run instructions for each test.

55

Auto Check When initializing for a test, starting a run, or performing any other operation,NotesBench uses an Auto Check routine to check the driver’s required NOTES.INIvariables. If they are not set, Auto Check sets them to their default values. If thevariable’s value cannot be determined at runtime, NotesBench terminates andreports the missing variables. The default values are reported as well.

If Auto Check finds no errors, it displays output status such as the output shown inthis example:

Auto Check Variable ResultsDirectory for R5maildbworkload....OK

Auto Check Variable NodeName for R5maildb workload....OK

Auto Check Variable Debug for R5maildb workload....OK

Using NOTEBNCH to run the tests Use the NOTEBNCH executable to launch the initialization programs and the testscripts. Instructions for running the individual tests appear with each test. Enter theNOTEBNCH command at the operating system prompt on the parent driver orchild driver.

On the parent driver, enter NOTEBNCH followed by the name of the test and thetotal number of child drivers. For example:

NOTEBNCH R5MAIL 6

On each child driver, enter NOTEBNCH -c, followed by the unique child number.The -c option is required and indicates that this is a child driver. Each child driverrequires a unique child number. For example:

NOTEBNCH -c 3 is the third child driver

To end the test, press ESC at the parent driver console. The child drivers rampdown, and the test stops.

Some NOTEBNCH examples: To start up the third child driver, enter:

NOTEBNCH -c 3

To start the R5Mail test on the parent driver, with 6 child drivers in use, enter:NOTEBNCH R5MAIL 6

Error Handling During the TestsServer errors may occur during the test run. They are usually caused byinterference from concurrent test users. Here are two:

55

w Nonfatal errors cause the script to pause for a few seconds before executingthe next script command. The operation that encountered the error is notretried.

w Fatal errors cause a thread script to terminate immediately. The one threadwith the error will terminate; the others continue processing.

Using NOTESNUM to obtain the resultsNOTESNUM is the utility that formats, stores, and retrieves test results. Run thecommand at the operating system prompt on the parent driver.

w Option 1: Enter NOTESNUM RESET. This clears the results of the previoustest.

w Option 2: Enter NOTESNUM followed by the name of the test you want torun, and the total number of child drivers. For a multiple run test, end thecommand with -mr. For example: NOTESNUM WEBMAIL 3 -mr

Monitoring the testsYou can monitor the progress of the tests from the parent driver and child drivers.This section presents some basic monitoring information.

Monitoring the results on each child driver The RESULTS file on the child driver displays the progress of a test. Here is anexample of some output from a RESULTS file, c:\results\res1.

A new summary line output similar to that shown in the following example isoutput to the screen on each child driver. The line is saved to the file specified inthe Debug_Outfile setting in the child’s NOTES.INI file. This output file istypically located on the parent driver.

04/04/00 12:15:01 PM (NotesBench for R5.0 - Notes Version:166 - Windows/32)

notebnch -C 1

04/04/900 12:16:01 PM (Notes Version: 166 - Windows/32)notebnch -s -nbr5mail -g 9999 -Q -t 50 -# 1

Begin statistics, Env=Windows/32, MultiFiles=FALSE,Processes=FALSE

Topaz 04/04/00 12:17:01 PM 50 Th 15 Agg 0 avgTh 0 minTh 8 maxTh 0 SemT/O 9234 NetTime(msec)


57




Sum of the time required (in milliseconds) to complete alltransactions across all threads on this driver for the one-minuteinterval.

14859 NetTime(msec)Semaphore timeout counters for the one-minute interval.0 SemT/O

Maximum number of transactions executed by a thread in theone-minute interval.

11 maxTh

Minimum number of transactions executed by any of thethreads in the one-minute interval.

0 minTh

Average number of transactions per thread in the one-minuteinterval. (158 Agg divided by 50TH.)

3 avgTh

Total number of transactions by all threads in the one-minuteinterval. See maxTh.

158 Agg

Number of threads (users) that are running on this driver duringthe one-minute interval.

50ThThe one-minute interval that ends at the date and time.04/04/00 12:28:01Name of this child driver.Topaz

Monitoring the results on the System Under TestOn the SUT you can generate a display that complements the Results file. Thedisplay lists the number of Notes transactions executed during the one-minuteperiod and the number of users who were active during the one-minute period. Forexample:

01/15/00 03:47:04 PM 335 Transactions/Minute, 39 Users



In the NOTES.INI file on the SUT, add the line Server_Show_Performance=1 toenable this feature. This generates a one-line output at one-minute intervals.Server_Show_Performance does not show HTTP, IMAP, POP3/SMTP, or POP3users. To disable the display, change the line to Server_Show_Performance=0.

You can use sh perf at the console to add this line automatically to the NOTES.INIfile. To disable the display, enter sh perf again.

57

Comparing Users on Drivers and the System Under TestThe “Th” (threads) parameter in each driver Results file is the number of users (onethread per user) on each client driver. The total users across all drivers should equalthe “Users” value in the server’s performance display.

The test must reach steady state to generate test results. To reach steady state, thetotal Notes users on the SUT must equal the total users across all child drivers.

If the number of Notes users on the SUT does not increase to equal the total clientdriver users or if the number of users on the SUT subsequently falls below theclient total, do the following:

Run the show users command on the SUT. A list of current users appears,including the user name, the databases open for the user, and the number ofminutes since each database was used.Run show statistics and monitor the Server.Users field values. Compare the show users and show statistics outputs with the minute-by-minuteoutput in the driver results files. This may indicate which users are notaccessing the SUT.Run show stat on the SUT. The network statistics (for example,NET.TCPIP.Sessions.Limit) in the output can indicate that network protocolsession limits are set too low to permit access to all the client users. It may alsotell you that you need to enable a different networking protocol on the serverand drivers with a higher session limit. Run show transactions to display the totals/timing values for numeroustransaction types. This information can be used to show where an inordinateamount of time is spent on any specific transaction.Each server thread equates to a driver client user. Use show transactions andshow users to see how many client threads are active on the server and whattasks each has been executing. The threads should be executing networkingtasks for each of the client users. Examine the NotesBench script for the test to determine which serverdatabases are accessed by the client user. Remember that each client user runsall benchmark specification language statements in the script. Run the show performance command to toggle the option to display or hidethe per-minute user/transaction values. Use Notes to monitor the views of the server’s log file (NOTES.NSF). Use the views in the server’s Notes Log file (LOG.NSF) to check the activityfor the server databases that a NotesBench test uses. For example, theDatabase - Sizes view can indicate whether a CHANGETO statement in a testscript was executed properly.

59

Other views for Notes Log contain details that can be useful for other NotesBenchtests. For example, the Mail Routing Events view shows mail routing details thatyou can use to check for a problem in the R5Mail test. Also, you can use theMiscellaneous Events view to supplement what you see with the SHOW STATserver command.

In the Show Transactions output, shown in the table below, Count is the totalnumber of transactions of the specified type, Minimum is the fastest time inmilliseconds to perform the specified transactions, Maximum is the slowest time inmilliseconds to perform the specified transaction, and Average is the average timein milliseconds to perform the specified transaction.

451201580NAME_LOOKUP

2550520OPEN_DBAverageMaximumMinimumCountTransaction

Using ProbeNote For releases prior to 4.6, Probe is known as Poller. This name alsoappears in the output text.The NotesBench Probe utility displays and records the time it takes for onesimulated user to open and close a discussion database on the SUT. This utility isuseful for observing the total perceived time to complete a typical Notes operation,such as opening the default view of a database, while the SUT is servicing aNotesBench test.

Probe SetupProbe is included in NotesBench 4.5 and higher. Probe requires a correspondingclient. It’s best to run it on the same system that you use as the parent driver. Addthe following setting to your NOTES.INI file:

SUTMachineID=Where System Under Test Machine ID is the reference ID of the SUT, whichyou use when you create an entry in the Capacity Planning database. You canuse any text string that does not include blank spaces.

This environment variable is used by NOTESNUM.EXE to create an entry in theCapacity Planning database, which will record NOTESNUM roll-up informationand the Probe statistics.

59

Running ProbeWhen the child drivers are loading the SUT, start Probe:

NOTEBNCH PROBE

For Multiple Run, add -mr at the end of the line: NOTEBNCH PROBE -mr

Run the probe on the same machine as the parent. The probe uses a database on theSUT. For example, if you are running the R5Mail test, use MAIL1.NSF. If you arerunning R5Mail, use TESTDISC.NSF. Subsequent runs of the probe will use thisdatabase. On one minute intervals, the probe will report statistics as it relates to theprobe script. The following is a sample output of the probe:

7/20/00 04:39:27 PM Script Iteration time: 64683 (ms)

Iteration: 183 LOT : 531 Avg : 571 Min : 401 Max : 902

<1.0n : 183 1.0 - 3.0: 0 3.0-5.0: 0 >5.0: 0



<1.0 n : 183 1.0 - 3.0: 0 3.0-5.0: 0 >5.0: 0



<1.0 n : 183 1.0 - 3.0: 0 3.0-5.0: 0 >5.0: 0

This table contains an explanation of the Probe output.

The number of iterations in which LOT was greater than 5seconds.

> 5.0

The number of iterations in which LOT was between 3 and 5seconds.

3.0 - 5.0

The number of iterations in which LOT was between 1 and 3seconds.

1.0 - 3.0The number of iterations in which LOT was less than 1 second.<1.0

Last Open Time. Refers to the actual time (in milliseconds)taken to execute the database “open” command. This roughlycorresponds to the time it takes for the simulated user to openthe database and the default view.

LOTTime it took to execute the entire probe script.Script Iteration timeThe total number of times the probe script was executed.IterationDescriptionField

61

Considerations for DEBUG_OUTFILEThe Debug_Outfile, mentioned in the section “Running the NotesBench Tests,” isthe file that captures the test results for each child driver. There is a separateDebug_Outfile for each driver.

If you plan to run Notes on a child driver, disable the Debug_Outfile setting usedin NotesBench by commenting out its line. When you are ready to run aNotesBench test on that driver, re-enable Debug_Outfile.

Running the NotesBench TestsInstructions for running the Lotus NotesBench 6 workloads and initialization testsare detailed in the individual topics in this section.

NotesBench workloadsR6Mail

R6iNotes

R6IMAP

R5Mail

R5iNotes

R5IMAP

Cluster Mail Test

WorkGroup

NotesBench Initialization TestsNRPC Mail Initialization

iNotes Initialization

R6IMAP Initialization

Cluster Mail Initialization

61

R6Mail TestThe R6Mail test models an active user on a client reading and sending mail, usingthe calendar and scheduling features to schedule an appointment, send aninvitation, and send an RSVP to an invitation. An average user will run this scriptfour times per hour. Each script iteration reads five documents, updates twodocuments, deletes two documents, scrolls a view once, opens and closes onedatabase, opens and closes one view, sends one memo to three recipients, and doesthree lookups on the Domino Directory (when NthIteration=6). Every 90 minutes,the test schedules one appointment and sends one invitation to the recipients. Thereare server name lookups, and messages are deposited in the SUT mailbox.

MetricsThis test measures:

w Throughput of completed Notes operationsw Average response time at maximum capacityw Maximum number of mail users supported

The resulting capacity metric for a mail-only server is the maximum number ofusers that can be supported before the average user response time becomesunacceptable.

Requirementsw One parent driverw One or more child driversw System under test (SUT)w Results directory

63

Test InitializationReview and apply the NotesBench requirements.

For information about the NotesBench requirements, see the chapter “IntroducingNotesBench.”

Perform all the required setup procedures that apply to your configuration forthis test, including setting up the NOTES.INI file and the destination servers.

For information about the required NotesBench setup procedures, see the chapter“Preparing the Environment.”

If you have not already created mail databases, do so before running the test.For information about creating mail databases, see the chapter “Preparing theEnvironment.”

NOTES.INI Settings for the child driver for R6MailThe following is an example of the NOTES.INI settings for the child driver:

NormalMessageSize=10000 MailServer=server/domainMailTemplate=mail6.ntfNthIteration=6NumMessageRecipients=3NumMailNotesPeruser=100(optional)NBTestReset=1MaxDocToDelete=1000000

63

NOTES.INI settings for the R6Mail SUTw Log_MailRouting. Set to 20 for minimal logging, or to 10 for no logging

(minimizes logging while testing.) Set it to 40 to display additional routinginformation on the console; for example, to verify that messages are beingrouted by the Router and MTA tasks.

w ServerTasks. Specify Router, IMAP, and LDAP.w IMAP_Session_Timeout. Set to 60.

Other suggested SUT settings include:

Mail_Number_Of_MailBoxes=2Max_Users=10100NSF_DBcache_Maxentries=10100Server_Pool_Tasks=100Server_Max_Concurrent_Trans=1000MAILLOGTOEVENTSONLY=1LOG_SESSIONS=0LOG_MAILROUTING=10SERVER_SHOW_PERFORMANCE=1MAILUSERPROCESSES=0No_Force_Activity_Logging=1

Server document settings for the R6Mail testBe sure that the Basics tab of the Server document uses these settings:

w Routing tasks: Mail Routing, SMTP Mail Routingw SMTP listener task: Enabledw Fully Qualified Internet host name: servername.company.com

65

Running the TestOn the parent driver, type NOTESNUM RESET to clear the results of anyprevious test.On each child driver, type NOTEBNCH -c number, where number is thenumber of the child.On the parent driver, type NOTEBNCH R6MAIL number, where number isthe highest child number. After the SUT reaches its steady state maximum value, press ESC on theparent driver to stop the test.On the parent driver, type NOTESNUM R6MAIL number, where number isthe maximum number of child drivers used. Record the NotesMark and Response Time values and the maximum numberof users as the result of this test.

65

R6IMAP TestThis test executes Domino transactions that model a server for mail users at sitesthat rely on IMAP mail for communication. Not only does this test stress the IMAPprotocol by receiving mail messages, but also exercises SMTP and LDAP bysending SMTP messages to a number of recipients and performing LDAP lookupson those recipients.

The R6IMAP test models an active user retrieving and sending mail. An averageuser executes this script no more than four times per hour. For each iteration of thescript, there is a retrieval of IMAP mail messages, one SMTP message sent, and ’n’LDAP lookup requests based on the NumMessageRecipients. In sending messages,each user sends a mail message to NumMessageRecipients not more than onceevery 15 minutes.

The SMTP messages sent by each driver user are delivered to the mail databases ofother driver users on the SUT.

Note The R6IMAP test makes use of the SMTP and LDAP protocols inaddition to IMAP.

System informationA Notes 6 client can run 1500 user threads per 512 MB driver. With less memory,the number of threads must be decreased. You can experiment to see what the CPUand memory usage is on your particular drivers for a given number of threads.

The number of users that can be supported by a server is limited by memory. OnWindows NT, that is 2 GBs and on UNIX it is 4 GBs.

On an 8way Intel® Xeon(TM), Microsoft Windows server the resource limitationis often memory. The maximum memory that the server can use is approximately2.3GBs.

Time RequirementsThe time required to run this test is six hours, minimum, after steady state verified.

67


w Throughput of completed IMAP, LDAP, and SMTP operationsw Average response time at maximum capacityw Maximum number of IMAP mail users supported

The capacity metric for an SMTP/IMAP server is the maximum number of usersthat can be supported before the average user-response time becomes unacceptable.

Requirementsw One parent driverw One or more child driversw System under test (SUT)w Results file





Run R6IMAPINIT to initialize and convert to IMAP mail files. For information about setting up the IMAP service, see the Lotus Domino 6documentation, “Administering the Domino System.”

67

Creating mail databases Use the R6IMAPinit workload in NotesBench to initialize the first database forDomino 6 or use the IMAPINIT workload for Domino R5. Make sure that thelatest MAIL6.NTF (MAIL5.NTF for R5) is copied to the child driver before youcreate the first database and that the mail template NOTES.INI setting is set on thechild.

For information about the NOTES.INI file, see the chapter “Preparing theEnvironment.”

After you create your first mail database, open the database by performing a File -Open from the Notes client or by using Microsoft Outlook®. Opening the databaseinitializes several database fields and uses less memory per user during the test.

NOTES.INI settings for the child driver for R6IMAP testThe following is an example of the NOTES.INI settings for the child driver:

NormalMessageSize=10000 MailRecipientPercentUser=50MailRecipientPercentVolume=50MailRecipientBeginNumber=1MailRecipientEndNumber=2000; EndNumber should be set to maximum entries in Domino DirectoryR5IMAP_LOOP_N=1000 (set to 56 to exit out of loop in 8 hours.)NthIteration=6ClientHost=hostname_of_the_clientSMTPHost=hostname_of_your_SMTP_server_under_testLDAPHost=hostname_of_your_LDAP_server_under_testIMAPHost=hostname_of_your_IMAP_server_under_testRecipientDomain=name_of_your_domainMessageLineSize=100NumMailNotesPerUser=100NumMessageRecipients=3

69

NOTES.INI settings for the R6IMAP SUTThe following are examples of the settings that can be added to the server’sNOTES.INI file:

view_rebuild_dir=l:\temp\Debug_Outfile=g:\server_debug\server_grigsby2k.txtSERVER_SHOW_PERFORMANCE=1Mail_Number_of_mailboxes=2 (can be set in server doc)ServerTasks=Router,IMAP,LDAPIMAP_Session_Timeout=60log_mailrouting=20. Set to 20 for minimal logging, or to 10 for no logging(minimizes logging while testing). Set to 40 to display additional routinginformation on the console; for example, to verify that messages are beingrouted by the Router and MTA tasks.Note The MailRecipientPercentUser and the MailRecipientPercentVolume settings impact each other. IfMailRecipientPercentUser=20 and Mail RecipientPercentVolume=80,80 percent of the messages are addressed to 20 percent of the users.

Server document settings for the R6IMAP testBe sure that the Basics tab of the Server document uses these settings:

w Routing tasks: Mail Routing, SMTP Mail Routingw SMTP listener task: Enabledw Fully Qualified Internet host name: servername.company.com

Special instructions for the R6IMAP TestOn the SUT, edit the Server document - Basics tab as follows, and then save andclose the document.

For example, servername.iris.comFully-qualified Internet host nameEnable this setting.SMTP listener task

Select Mail Routing.Routing TasksSettingField

Open the Server Configuration document and click Edit document. On the Basics tab, locate the field “International MIME setting for thisdocument,” and click Enabled.Edit the Router/SMTP tab as follows and then save and close the document.

69

Select “MIME messages only”SMTP allowed within the local Internetdomain

Select “Always”Servers within the local Notes domain arereachable via SMTP over TCPIP

Select “All messages”SMTP allowed within the local Internetdomain

Enable this setting.SMTP used when sending message outsideof the local Internet domain:

SettimgField

Running the testNote Clear the memory on the client (reboot) before the start of this test.Complete these steps to run the NotesBench software for the R6IMAP test.

If you have not already created mail databases, do so now.On the parent driver, type notesnum reset to clear the results of any previousMail tests.On the SUT start the Domino server. The router, IMAP, LDAP, and SMTPserver processes must be started. You can run additional server programs atyour discretion. On each child driver system, type notebnch -c num where num is the uniquechild number for this system. On the parent system, type notebnch imapinit <MaxChild> where MaxChild isthe highest child number.

After creating IMAP files, do the following:

On each child driver system, type notebnch -c num where num is the uniquechild number for this system. On the parent system, type notebnch R6IMAP <MaxChild> or notebnchR6IMAP <MaxChild> -mr where MaxChild is the highest child number.After the SUT has reached its steady state maximum value, stop the test bypressing ESC on the parent driver system. Issue a SH STAT command at the SUT console prior to exiting the Dominoserver at the end of the test.Disable the setting “DEBUG_OUTFILE” by commenting it out in NOTES.INIwith a semicolon at the beginning of the line.On the parent driver, type notesnum R6IMAP <MaxChild>.

71

Client SetupUse the Lotus Domino 6 Notes client for best results.

Open the Location document and modify these fields:

w On the Servers tab, in the home/mail server field, enter the server namefollowed by the domain name. For example, Servername/IrisTS.

w On the Basics tab, enter the Internet mail address, for example,[email protected] in the Internet mail address field.

71

R6iNotes TestDomino Web Access (iNotes Web Access) is our next generation Web client forWeb-based access to Domino messaging and Personal Information Management(PIM) capabilities.

The R6iNotes test represents an active user sending, retrieving, and deleting mailfrom a browser. An average user runs this script four times an hour. Each time thescript runs, it checks and retrieves mail messages. Additionally, on eachNthIteration, which is one time in six for this workload, the user sends a mailmessage to NumMessageRecipients other users on the server. The message andattachment size distribution will be set according to normal NotesBench 6Workload settings. The user schedules an appointment, sends invitations toNumMessageRecipients other users, and responds to one invitation that it finds inits own inbox.

Time RequirementsThe R6iNotes test must run for six hours, minimum, after steady state is verified.


w Throughput of completed HTTP operationsw Average response time at maximum capacityw Maximum number of R6iNotes users supported before the average user

response time becomes unacceptable

73

System Requirements The system requirements for the R6iNotes test are:

w One parent driverw One or more child driversw System under test (SUT)w Results directory

Test InitializationThis test requires the following:

w Check and complete the NotesBench requirements.For information about the NotesBench requirements, see the chapter “IntroducingNotesBench.”

w Perform all the required setup procedures that apply to your configuration forthis test, including setting up the NOTES.INI file and the destination servers.


The console does not display Web users because they do not update the samecounters as Notes clients. Notes clients use the Database Server task, and Webusers use the HTTP task. Use the show task command to display each task at theserver console.

To determine if mail is being delivered to the server, use this command at theconsole:

show stat Domino.Requests.*

To monitor message counters, use one of these commands at the console:show stat HTTP

show stat mail

If authentication errors appear at the console, check that the password in the HTTPfield of the Person document in Domino Directory on the SUT is set toNotesBench. If necessary, edit the Person document in the Domino Directory. Usethe agent “Set HTTP Password to ’NotesBench’.”

Ensure that the mail template is compatible with the Domino release you arerunning. For Domino Release 6, use the template iNotes6.NTF. In the NOTES.INIfile of the child driver, enter Set MailTemplate=iNotes6.NTF.

To display routing information on the console, include Log_MailRouting=40 in theNOTES.INI file on the SUT.

73

Special Considerations for this workloadw The iNotes6 template, INOTES6.NTF, is available with the server, and must

reside on the server in order to work correctly.w After users and mail files are created, run the “Update ACL of Mail Databases

to include owner” agent on the Domino Directory of the SUT to update theACL for the mail databases. The owner of the mail databases must be namedin the ACL.

Server document settings for the R6iNotes testIn the Server document, use these settings:

w In the “Administrator” field on the Basics tab, enter the administrator’s name,that is, the name of the user that has been defined as the administrator. If youdo not include the administrator’s user name, any server commands issued byProbe are rejected.

w On the Internet Protocols - HTTP tab, set “HTTP Persistent connections” to“Disabled.”

w On the Internet Protocols - HTTP tab (R5 Basics section), set the field“Optimize HTTP performance based on the following primary activity” toAdvanced (Custom Settings). This setting allows you to view/modify thenumber of HTTP threads and is usually set during installation.

NOTES.INI settings for the R6iNotes test SUTCheck these NOTES.INI file settings:

w Verify that the Server Tasks setting in the NOTES.INI file on the SUTincludes HTTP.

w The optimum setting for the number of HTTP threads requires someexperimentation. Start with a value of 100 on a large machine and adjust thisup or down until you obtain the best results.

Access rightsThe user that you designate as the administrator must have Manager access to theDomino Directory in that directory’s ACL.

75

Authenticationw By default, R6iNotes assumes that user authentication is required. For

authenticated users, ACLs for all mail databases and the Domino Directorymust specify Manager access for the - Default - user. In the NOTES.INI file ofthe child drivers and Probe, use the WebAuthenticationOff=0 (or not defined)setting.

w On the Agent Restrictions section of the Server document - Security tab, setthe values of the following properties to * (wildcard character). This allowsanonymous users to run personal agents, run restricted LotusScript/Javaagents, and run unrestricted LotusScript/Java agents.

Running the testR6iNotes requires authenticated users.

Create mail databases with the iNotesInit workload. For information about creating mail databases, see the chapter, “Preparing theEnvironment.”For authenticated users add each user to the ACL in his mail file. To do so, usethe agent “Update ACL of mail dbs to include owner (mail1, mail2, ...)” fromthe NAMAGENT.NSF.On the parent driver, type NOTESNUM RESET to clear the results of anyprevious tests.On each child driver, type NOTEBNCH -c number, where number is theunique child number.On the parent system, type NOTEBNCH R6iNotes number, where number isthe number of the highest child. After the SUT has reached its steady state maximum value, press ESC on theparent driver to stop the test. Enter the SH STAT command at the SUT console before exiting the Dominoserver at the end of the test.On the parent driver, type NOTESNUM R6iNotes number, where number isthe maximum number of child drivers.

75

R5Mail TestThe R5Mail test models an active user on a client reading and sending mail, usingthe calendar and scheduling features to schedule an appointment, send aninvitation, and send an RSVP to an invitation. An average user will run this scriptfour times per hour. Each script iteration reads five documents, updates twodocuments, deletes two documents, scrolls a view once, opens and closes onedatabase, opens and closes one view, sends one memo to three recipients, and doesthree lookups on the Domino Directory (when NthIteration=6). Every 90 minutes,the test schedules one appointment and sends one invitation to the recipients. Thereare server name lookups, and messages are deposited in the SUT mailbox.



The resulting capacity metric for a mail-only server is the maximum number ofusers that can be supported before the average user response time becomesunacceptable.

Requirementsw One parent driverw One or more child driversw System under test (SUT)w Results directory






77

NOTES.INI settings for the child driver for R5MailThe following is an example of the NOTES.INI settings for the child driver:

NormalMessageSize=10000 MailServer=server/domainMailTemplate=mail50.ntfNthIteration=6NumMessageRecipients=3NumMailNotesPeruser=100(optional)NBTestReset=1MaxDocToDelete=1000000

NOTES.INI settings for the SUTw Log_MailRouting. Set to 20 for minimal logging, or to 10 for no logging

(minimizes logging while testing). Set it to 40 to display additional routinginformation on the console; for example, to verify that messages are beingrouted by the Router and MTA tasks.

w ServerTasks. Specify Router, IMAP, and LDAP.w IMAP_Session_Timeout. Set to 60.

Other suggested SUT settings include:

Mail_Number_Of_MailBoxes=2Max_Users=10100NSF_DBcache_Maxentries=10100Server_Pool_Tasks=100Server_Max_Concurrent_Trans=1000MailLogToEventsOnly=1Log_Sessions=0Log_MailRouting=10Server_Show_Performance=1MailUserProcesses=0No_Force_Activity_Logging=1

Server documents settingsBe sure that the Basics tab of the Server document uses these settings:

w Routing tasks: Mail Routing, SMTP Mail Routing

77

w SMTP listener task: Enabledw Fully Qualified Internet host name: servername.company.com

Running the testOn the parent driver, type NOTESNUM RESET to clear the results of anyprevious test.On each child driver, type NOTEBNCH -c number, where number is thenumber of the child.On the parent driver, type NOTEBNCH R5MAIL number, where number isthe highest child number. After the SUT reaches its steady state maximum value, press ESC on theparent driver to stop the test.On the parent driver, type NOTESNUM R5MAIL number, where number isthe maximum number of child drivers used. Record the NotesMark and Response Time values and the maximum numberof users as the result of this test.

79

R5iNotes TestDomino Web Access (iNotes Web Access) is our next generation Web client forWeb-based access to Domino messaging and Personal Information Management(PIM) capabilities.

The R5iNotes test represents an active user sending, retrieving, and deleting mailfrom a browser. An average user runs this script four times an hour. Each time thescript runs, it checks and retrieves mail messages. Additionally, on eachNthIteration, which is one time in six for this workload, the user sends a mailmessage to NumMessageRecipients other users on the server. The message andattachment size distribution are set according to normal NotesBench Workloadsettings. The user schedules an appointment, sends invitations toNumMessageRecipients other users, and responds to one invitation that it finds inits own inbox.

Time RequirementsThe R5iNotes test must run for six hours, minimum, after steady state verified.


w Throughput of completed HTTP operationsw Average response time at maximum capacityw Maximum number of R5iNotes users supported before the average user

response time becomes unacceptable

System Requirements The system requirements for the R5iNotes test are:

w One parent driverw One or more child driversw System under test (SUT)w Results directory

79






Special considerations for this workloadw The iNotes5 template (INOTES5.NTF) is available with the server, and must

reside on the server in order to work correctly. w After users and mail files are created, run the “Update ACL of Mail Dbs to

include Owner mail1, mail2,” agent on the Domino Directory of the SUT toupdate the ACL of the mail databases. The owner of the mail database must benamed in the ACL.

NOTES.INI settings for the Notes client driver systemsSelect the Script Variables pane and then complete these fields:

Define the variable to delete existing documents at thestart of a test. Possible values are:w NBTestReset=1 — To delete existing documents

at the start of the test

NBTest Reset

Number of notes with which to populate the mail fileupon creation. The recommended value is 100.

NumMailNotesPerUser

Pointer to the template that resides on the server usedfor mail file creation. For example,Servername/Acme!!inotes5.ntf

ServerName!!MailTemplate

Database directory relative to the Notes data directory.To have all mail files created in the data\mail\directory set nb_dbdir=mail\. To have mail filescreated in the data directory, enter a space .

nb_dbdir

Canonical Name of MailServer (for example,CN=MailServer1/O=Acme). The MailServer line in the NOTES.INI isautomatically updated when editing the Locationdocument.

MailServerValue to enterScript Values

81

Notes Mail Domain Name of the Domino Server. Forexample, Acme.Note This value must match the value in the MailDomain field of the Location document on theclient.

Domain

Size of the body of the message. The recommendedvalue is 10000.

NormalMessageSize

The TCP/IP address or Hostname of the Domino WebServer.

httphost

w NBTestReset=0 — To ignore existing documentsat the start of a test

The number of documents deleted is dependent on thevalue set for the variable MaxDocToDelete.

Value to enterScript Values

Setting up the Domino Server for the R5iNotes WorkloadUse Domino Designer to copy all agents from the database Server.Load Setupagents to the agents view of the Domino Directory of the SUT. See the “Using This Database” document of the Server.Load Setup Agentsdatabase for more information about these agents.Open the Domino Directory of the test server and run the agent CreateNotesBench Person Documents. In addition to creating the Person documents,this agent also sets the HTTP password. Accept the default settings for the agent except where indicated in the table.

To select POP or IMAP, enter a valueof 6. Default = 1

Mail System1 -- No preferenceMessage Storage Format

Default: Name of the server on whichthe Domino Directory resides.

Mail Server

Default: Read from the server's maildomain

Mail Domain

This value must match the value fornb_dbdir script variable. Default =mail\

Location for Mail Databases1 (the default value)Starting Mailn.NSF file

Enter the total number of mail databasefiles you are creating. This is the samevalue as the value in the field “Numberof Users to Create.” The default =1000.

Number of Mailn.NSF files to use

Enter the total number of test users youare creating. The default = 1000.

Number of Users to Create1 (the default value)Starting value to Create Mail usersUse this valueSetting

81

Default: Hostname of the server onwhich the Domino Directory resides.

Internet Host nameUse this valueSetting

Server document settings for the R5iNotes workloadThe Server document in the SUT’s Domino Directory requires these settings:

w On the Basics tab in R5, or the Security tab in Lotus Domino 6, enter the nameof the administrator in the “Administrator” field.

w On the Basics tab in R5 or the Internet Protocols tab in Domino 6, in the field“Optimize HTTP performance based on the following primary activity”choose Advanced (Custom Settings). This field allows you to view/modify thenumber of HTTP threads and is usually set during installation. The optimumsetting for the number of HTTP threads requires some experimentation. On alarge machine, begin with the value of 100, then adjust up or down,accordingly.

w For Lotus Domino 6, on the Internet Protocols - HTTP tab, locate theTimeouts section. In the field “HTTP Persistent Connections” choose Disable.

NOTES.INI settings for the SUTOpen the NOTES.INI file on the SUT, and verify that HTTP is included in theServer Tasks. To display additional data on the console, in the NOTES.INI file, usethe setting Log_MailRouting=40.

Access Rights settingsOpen the ACL for the Domino Directory and ensure that the person designated asadministrator in the Server document is also designated Manager.

83

Running the testBe sure mail databases are available.

On the parent driver, type NOTESNUM RESET to clear the results of anyprevious tests.On the SUT, start the Domino server. The Router, Update, IMAP, LDAP, andSMTP server tasks must be running. You can run additional server tasks if youwish. On each child driver, type NOTEBNCH -c number, where number is theunique number for the child driver. On the parent driver, type NOTEBNCH INotesINIT number, where number isthe number of the highest child driver.

After creating iNotes mail files, do the following:

On each child driver, type NOTEBNCH -c number, where number is theunique number for the child driver.On the parent driver, type NOTEBNCH R5iNotes number, where number isthe number of the highest child driver.After the SUT has reached its steady state maximum value, press ESC on theparent driver to stop the test.On the console at the SUT, enter a sh stat command prior to exiting theDomino server at the end of the test.On the parent driver, type NOTESNUM R5iNotes number, where number isthe number of the highest child driver.

83

R5IMAP TestThe R5IMAP test models a server for IMAP mail users. The test stresses the IMAPprotocol by receiving messages. The test also exercises SMTP and LDAP bysending SMTP messages to recipients and performing LDAP lookups on thoserecipients. In the R5IMAP test, each iteration of the script retrieves IMAP mailmessages and sends an SMTP message using LDAP lookups for the specifiednumber of recipients.

The SMTP messages sent by each user are delivered to the mail databases of otherusers on the SUT.



w Throughput of completed IMAP, LDAP, and SMTP operationsw Average response time at maximum capacityw Maximum number of IMAP mail users supported

The capacity metric for an SMTP/IMAP server is the maximum number of usersthat can be supported before the average user-response time becomes unacceptable.

Requirementsw One parent driverw One or more child driversw System under test (SUT)w Results file

85





Run IMAPINIT to initialize and convert to IMAP mail files.On each child driver, type NOTEBNCH -c number, where number is theunique number for the child driver. On the parent driver, type NOTEBNCH IMAPINIT number, where number isthe number of the highest child driver.

For information about setting up the IMAP service, see the Lotus Domino 6documentation, “Administering the Domino System.”

NOTES.INI settings for the child driver for R5IMAPThe following is an example of the NOTES.INI settings for the child driver:

NormalMessageSize=10000 MailRecipientPercentUser=50MailRecipientPercentVolume=50MailRecipientBeginNumber=1MailRecipientEndNumber=2000; EndNumber should be set to maximum entries in Domino DirectoryR5IMAP_LOOP_N=1000 (set to 56 to exit out of loop in 8 hours.)NthIteration=6ClientHost=hostname_of_the_clientSMTPHost=hostname_of_your_SMTP_server_under_testLDAPHost=hostname_of_your_LDAP_server_under_testIMAPHost=hostname_of_your_IMAP_server_under_testRecipientDomain=name_of_your_domainMessageLineSize=100NumMailNotesPerUser=100NumMessageRecipients=3

85

NOTES.INI settings for the R5IMAP SUTw Log_MailRouting. Set to 20 for minimal logging, or to 10 for no logging

(minimizes logging while testing). Set it to 40 to display additional routinginformation on the console; for example, to verify that messages are beingrouted by the Router and MTA tasks.

w ServerTasks= Specify Router, IMAP, and LDAP.w IMAP_Session_Timeout= Set to 60.

Server document settings for the R5IMAP testBe sure that the Basics tab of the Server document uses these settings:

w Routing tasks: Mail Routing, SMTP Mail Routingw SMTP listener task: Enabledw Fully-Qualified Internet host name: servername.company.com

Running the testBe sure mail databases are available.Copy the agent “Set HTTP Password to NotesBench’” fromNAMAGENT.NSF and paste it into the Domino Directory on the SUT. Run the agent. The agent sets the HTTP Password field in each Persondocument to “NotesBench”.On the parent driver, type NOTESNUM RESET to clear the results of anyprevious tests.On the SUT, start the Domino server. The Router, IMAP, LDAP, and SMTPserver tasks must be running. You can run additional server tasks if you wish.

After creating IMAP files, do the following:

On each child driver, type NOTEBNCH -c number, where number is theunique number for the child driver.On the parent driver, type NOTEBNCH R5IMAP number, or NOTEBNCHR5IMAP number -mr, where number is the number of the highest child driver.After the SUT has reached its steady state maximum value, press ESC on theparent driver to stop the test.On the console at the SUT, enter a sh stat command prior to exiting theDomino server at the end of the test.On the parent driver, type NOTESNUM R5IMAP number, where number isthe number of the highest child driver.

87

Cluster Mail TestThis test executes Notes transactions that model a cluster for mail users at sitesrelying on a two-way Domino cluster for messaging.

The Cluster Mail Test script models an active user sending and reading mail on aclient. It contains an average of eight minutes of waiting; therefore, an average userexecutes this script no more than seven times per hour. For each iteration of thescript, there are five documents read, two documents added, two documentsdeleted, one view scrolling operation, one database opened and closed, one viewopened and closed, and some miscellaneous operations. One message is sent to therecipients approximately every 90 minutes.

Messages that each driver user sends are delivered to mail users in the cluster. Theupdates are then cluster-replicated to all replica copies of the user’s database. TheCluster Mail test models reading, submitting, delivering, and retrieving messages.As such, the mail users must have a corresponding $Person entry in the DominoDirectories on clients and servers in order to receive mail. Run the agent, “CreateNotesBench Person Documents,” to create these entries.

Since simulated users are connecting to the cluster, instead of to a specific server,they generate their mail from the server they are currently failed-over to. Eachiteration of the script provides an opportunity for the user to fail-over to a differentnode depending on the availability of specific node resources.

See Administering Domino Clusters for options in setting cluster operations, suchas number of nodes, node availability, threshold values, and Single Copy ObjectStore usage.

Metrics UsedThe measurements obtained by this test include:


The resulting capacity metric for a cluster under test is the maximum number ofusers that can be supported before the average user response time becomesunacceptable.

87

Test InitializationComplete the following steps to prepare for running the test:



On the parent driver, type notesnum reset to clear the results of any previousMail tests.On the SUT, start the Domino server. The router, replicator, and update serverprocesses must be started. You can run additional server programs at yourdiscretion. On each child driver system, type notebnch -c num where num is the uniquechild number for this system. Run the workload Cluster Mail Initialization on the parent by entering“notebnch clumailinit MaxChild” to create replicas on the clustered serverdefined by [ClusterServer2] in the child’s NOTES.INI file. The testerdetermines how many, and on which node, replicas should be placed.

89

NOTES.INI settings on the child driver for the Cluster Mail testEnter the following values in each child’s NOTES.INI file.

Enter the frequency for how often amessage is sent. This variable causes themessage to be sent once per n iterations ofthe scrip, instead of being sent on everyscript iteration. The recommended valueis 6.

NthIteration

Enter the number of recipients for eachmessage. Recommended value is 3.

NumMessageRecipients

Enter the number of documents to deletewhen the test starts. After deletingdocuments, the initial document count ishigh.

MaxDocToDelete

Enter one of the following to control howto handle existing documents at the startof the test:w 1 — To delete existing documentsw 0 — To ignore existing documentsNote The number of documentsdeleted is dependent on the value setfor the variable MaxDocToDelete.

NBTestResetActionVariable

Running the testComplete these steps to run the Cluster Mail Test.

On each child driver system, type notebnch -c num where num is the uniquechild number for this system.On the parent system, type notebnch clustermail MaxChild where MaxChild isthe highest child number. After the SUT has reached its steady state maximum value, stop the test bypressing ESC on the parent driver system. On the parent driver, type notesnum mail <MaxChild>. Record the output NotesMark value and the maximum users as the result ofthis test. Disable the setting “DEBUG_OUTFILE” by commenting it out in NOTES.INIwith a semicolon at the beginning of the line.

89

WorkGroup TestThis test models a server for Notes users who send an SMTP message, performLDAP lookups for recipients, schedule an appointment, send an invitation, delete amessage, use HTTP to access a shared discussion database, update a local replicaof the discussion database, and replicate with the discussion database.


Metrics This test measures:

w Throughput of completed LDAP, HTTP, and SMTP operationsw Average response time at maximum capacityw Maximum WorkGroup users supported

The resulting capacity metric for a WorkGroup test is the maximum number ofusers that can be supported before the average user response time becomesunacceptable.

Requirementsw One parent driver w One or more child drivers w System under test (SUT)w Results directory

91



Perform all the required setup procedures that apply to your configuration forthis test, including setting up the NOTES.INI file.


Use SHOW STAT MAIL at the Domino console to monitor the messagecounter.Include the setting Log_MailRouting=40 in the SUT’s Directory to displayadditional routing information on the console to verify that messages are beingrouted by the Router and the MTA tasks.When you are sure that the test is set up correctly, reset Log_MailRouting=10.

NOTES.INI settings for the child driver for the WorkGroup testThe following is an example of the NOTES.INI settings for the child driver.

tmp=tmpMailServer=server/domainMailTemplate=mail50.ntfNthIteration=6NumSharedNotes=100NormalMessageSize=10000NumMessageRecipients=3MessageLineSize=100DiscussionDB=testdisc.nsfDiscTemplate=discsw50.ntfNumNotesLocalReplica=100SMTPhost=hostname_of_your_smtp_server_under_testclienthost=myclient.mydomain.comrecipientdomain=mydomainLDAPhost=hostname_of_your_ldap_server_under_testHTTPhost=hostname_of_your_http_server_under_testSMTPDomain=yoursys.yourhost.comMaxDocToDelete=1,000,000

91

NumMailNotesPerUser=NBTestReset=1

Running the testIf you have not already created mail databases, do so before running the test.

For information about creating mail databases, see the chapter “Preparing theEnvironment.”

Edit the ACL of the Discussion Database to allow Anonymous access, therebyallowing Web access to the database, and enabling the WorkGroup threads touse HTTP to access the database. Replicate the Domino Directory among the SUT and the drivers.On the parent driver, type NOTEBNCH RESET or NOTESNUM RESET toclear the results of any previous test.On the SUT, start the Domino server. The Router, Replicator, Update, HTTP,SMTP, LDAP, and POP3 server tasks must be running on the server. You canrun additional server tasks if you wish.On each child driver, type NOTEBNCH-c number, where number is theunique number of the child driver.On the parent driver, type NOTEBNCH WORKGROUP number, wherenumber is the number of the highest child driver.After the SUT has reached its steady state maximum value, press ESC on theparent driver to stop the test.Enter the sh stat command at the SUT console prior to exiting the Dominoserver.On the parent driver, type NOTESNUM WORKGROUP number, wherenumber is the number of the highest child driver.

93

NotesBench Database Initialization TestsNotesBench initialization tests create and populate databases on the SUT. Runningan initialization script is a required part of the procedure for executing all of theNotesBench workloads except the Idle Usage test. Running initialization testsmakes setting up server databases independent from running the workloads.

Note For information about the Idle Usage test, see the NotesBench 5.0.4or earlier documentation.Each initialization routine executes in a manner similar to the workload tests: eachuses a script that executes benchmark specification language statements,customized with Notes environmental variables. Unlike the workload tests, theinitialization routines do not return a performance metric.

The initialization tests place data in the results file specified byDEBUG_OUTFILE. You can ignore these results. You should, however, delete theresults file before executing the workload test. Then create a newDEBUG_OUTFILE location for the workload test.

If you fail to run initialization tests as part of executing a workload, Notes creates.NSF files for databases, but they do not contain the necessary documents. Forexample, running the R6Mail workload without the MAILINIT initialization testcreates empty mailnn.nsf files. Because these files do not contain the requiredNotes documents, script execution generates NIFOpenErrors.

Initialization scripts automatically drop users once their respective tasks arecompleted. Testers can verify that initialization is complete when they see 0 (zero)users on the server console. The “Show Performance” command toggles thedisplay of database task user connections. It doesn’t currently show user counts forIMAP.

Note When you have achieved 10 minutes of 0 transactions from thedriver systems, and 0 transactions on the SUT, the initialization script hascompleted creating the files and has populated them with Notesdocuments. The procedures for executing each NotesBench workload include a step forrunning initialization tests.

93

NRPC Mail Initialization WorkloadThe NRPC Mail Initialization test opens a mail database on the server andpopulates the mail database. It then opens a discussion database and populates thediscussion database with a number of notes scaled to the SUT. The template usedto create the mail databases is specified in the NOTES.INI variable MailTemplate.The mail databases are populated with the number of notes (messages) specified inthe NOTES.INI variable NumMailNotesPerUser.

Running the Web Mail Initialization or Mail Initialization TestThe Web Mail Initialization and Mail Initialization tests automatically create andset up the mail databases and establish the settings listed below:

w Database ACL settingsw Default user is provided with Manager accessw Anonymous user is provided with Manager access

w Owner set = mailThe MailInit test opens a mail database on the server and populates the maildatabase, and opens a discussion database and populates the discussion databasewith a number of notes scaled to the SUT.

Prerequisitesw The appropriate mail templates must exist on the driver.w All user passwords must be set to NotesBench because authenticated Web

users are required.

Using the "Create NotesBench Mail Persons Documents" AgentUse the “Create NotesBench Mail Person Documents” agent to create Persondocuments. It is stored in NAMAGENT.NSF.

Run mailinit or webmailinit to create the mail databases.Use the “Create NotesBench Mail Person Documents” NAMAGENT.NSFagent on any platform to create Person documents that use the mailinit’smail*.nsf files on the SUT as the users’ mail files. Copy NAMAGENT.NSF tothe Domino Directory and run it from there.

95

Copying an Existing Mail DatabaseComplete this procedure to use the Notes client to create a mail database and thencopy that database to create as many additional mail databases as you need.

Run the Notes client on the Domino server.Create mail1.nsf using the mail template that corresponds to the release ofNotes that you are using.In the ACL for the mail database, assign Manager access to the Default user.Assign No access to the Anonymous user.Exit the Notes client.Copy mail1.nsf to mail2.nsf, etc. Delete the transaction log files so that theycan be rebuilt when the server is brought up.

95

Cluster Mail Initialization TestThe Cluster Mail initialization test creates a mail database on the server andpopulates the mail database with a number of notes scaled to the SUT. The ClusterMail Initialization and Cluster Mail workloads are based on the NRPC mailworkloads, except the mail databases are created on two servers duringinitialization so that failover performance can be tested.

The Cluster Mail Initialization test automatically creates and sets up the maildatabases and establishes the settings listed below:


w Owner set = mail

Prerequisitesw The appropriate mail templates must exist on the driver.w Two clustered servers are required for this test - a primary server and a second

server to test mail file failover performance when the primary server isunavailable.

Using the "Create NotesBench Mail Person Documents" AgentUse the “Create NotesBench Mail Person Documents” agent to create Persondocuments. It is stored in NAMAGENT.NSF.

Use the “Create NotesBench Mail Person Documents” NAMAGENT.NSFagent on any platform to create Person documents that use the mailinit’smail*.nsf files on the SUT as the users’ mail files. Copy NAMAGENT.NSF tothe Domino Directory and run it from there. See the Using this DatabaseDocument in NAMAGENT.NSF for more information about using the agents.Replicate the Domino Directory with newly created users to the second testserver.

97

NOTES.INI settings on the child client drivers

Enter the canonical name of the second mail server— for example, CN=MailServer2/O=Acme.

ClusterServer2

Enter the size of the body of the massage.Recommended value is 10000.

NormalMessageSize

Number of notes used to populate the mail file whenthe mail file is created. Recommended value 100.

NumMailNotesPerUserEnter the name of the mail file template.MailTemplate

Enter a database directory relative to the Notes datadirectory. Recommended value is mail\.

nb_dbdir

Enter the canonical name of the mail server — forexample, CN=MailServer1/O=Acme.

MailServerActionVariable

Running the testRun the workload with the following commands: Parent: notebnch clumailinit MaxChild

where MaxChild is the number of child drivers Child: notebnch -C N

where N is the number of the child driverWhen the workload has completed, verify that all mail files have been createdon both servers.Proceed to the Cluster Mail workload.

97

R6IMAP Initialization WorkloadThe R6IMAP Initialization test opens a mail database on the server and populatesthe mail database. The template, MAIL6.NTF, that is used to create the maildatabases should be specified in the NOTES.INI variable MailTemplate. The mailfile is converted for IMAP use. The mail databases are populated with the numberof notes (messages) specified in the NOTES.INI variable NumMailNotesPerUser.

The R6IMAP Initialization test automatically creates and sets up the mail databasesand establishes the settings listed below:


w Owner set = mailFor information about the NOTES.INI settings for the child drivers for theR6IMAP Initialization workload, see the topic, R6IMAP Test, in the chapter“Preparing the Environment.”

Prerequisitesw The mail template, MAIL6.NTF, must exist on the test driver.w All user passwords must be set to NotesBench because authenticated users are

required.

99

iNotes Initialization workloadThe iNotes Initialization workload creates a set of mail databases on the server andpopulates them with mail. The mail databases are populated with the number ofnotes (messages) specified in the NOTES.INI variable NumMailNotesPerUser.

The iNotesInit test automatically creates and sets up the mail databases for theR5iNotes and R6iNotes tests and establishes the settings listed below:


w Owner set = mailFor information about the NOTES.INI settings for the child drivers for the iNotesInitialization workload, see the chapter “Preparing the Environment” and refer tothe settings for the R6iNotes or R5iNotes tests.

Using the "Create NotesBench Mail Persons Documents" AgentThe agent that you can use to create Person documents is in NAMAGENT.NSF.

Use the “Create NotesBench Mail Person Documents” NAMAGENT.NSFagent on any platform to create Person documents that use the iNotesInit’smail*.nsf files on the SUT as the users’ mail files. Copy NAMAGENT.NSF tothe Domino Directory and run it from there. See the Using this Databasedocument for more information about the agents in NAMAGENT.NSF.

99

Running the testRun iNotesInit to create the mail databases.Run the workload with the following commands: Parent: notebnch iNotesInit MaxChild

where MaxChild is the number of child drivers Child: notebnch -C N

where N is the number of the child driverWhen the workload is complete, verify that all mail files have been created.Run one of these tests:The R6iNotes test, as detailed in the topic “The R6iNotes Test” in this chapter.The R5iNotes test, as detailed in the topic “The R5iNotes Test” in this chapter.

101

Chapter 5Using the Test Results

NotesBench workloads produce data that are subject to further analysis. Thischapter discusses using the results and their interpretation.

NotesBench Test ResultsNotesBench performance metrics can include throughput metrics, capacity metrics,and response times.

Workload Metrics SummaryEach NotesBench test (workload) produces at least one metric that can be used tomeasure server performance. This table lists the metrics provided by each test.

NotesBench Performance Metrics

Throughput of completed Notes operationsAverage response time at maximum capacityMaximum number of mail users supported

R6Mail

Throughput of completed HTTP operationsAverage response time at maximum capacityMaximum number of R5iNotes users supported before theaverage user response time becomes unacceptable

R5iNotes

Throughput of completed IMAP/LDAP operationsAverage response time at maximum capacityMaximum number of IMAP mail users supported

R5IMAP

Throughput of completed Notes operationsAverage response time at maximum capacityMaximum number of mail users supported

R5Mail

Throughput of completed Notes operationsAverage response time at maximum capacityMaximum mail users supported

ClusterMailResultsTest

101

Throughput of completed LDAP, SMTP, and HTTPoperationsAverage response time at maximum capacityMaximum workgroup users supported

WorkGroup

Throughput of completed HTTP operationsAverage response time at maximum capacityMaximum number of R6iNotes users supported before theaverage user response time becomes unacceptable

R6iNotes

Throughput of completed IMAP, LDAP, and SMTPoperationsAverage response time at maximum capacityMaximum number of IMAP mail users supported

R6IMAPResultsTest

Viewing the Test ResultsThe NOTESNUM utility retrieves the NotesBench test results. RunningNOTESNUM is the last step in running a NotesBench test. The NOTESNUMcommand reads the results files in the shared results directory to obtain the data togenerate the test results report.

For more information see the topic “Evaluating the Results” in this chapter.

Here is an example of the NOTESNUM utility for a completed R5Mail test.(NOTESNUM MAIL):

05/03/00 11:04:00 AM (Notes Version: 5.03 - Windows/32)c:\notes\notesnum mail 2

Reading results file - c:\results\res0 ...

Min Start Time = 05/02/00 06:23:19 PM Max Stop Time =05/03/00 07:22:49 AM

Total Test Errors = 0

Total Test Time = 943140 sec

Test Run: Users = 70 NotesMark = 195

Response Time = 515 msec (05/02/00 06:24:00 PM to 05/03/0006:22:00 AM)




103



Test Run: Users = 140 NotesMark = 391

Response Time = 553 msec (05/02/00 07:24:00 PM to 05/03/0006:22:00 AM)

Test Results ExplanationsPath: Set up a path in the NOTES.INI file of each child driver before the test. Eachpath identifies the location of the Results file, one on each child driver. The filename is \resnumber, where number is the number of the child driver.

Reading results file - e:\notestst\results\res1 ...

Reading results file - e:\notestst\results\res2 ...

Timing: Timing shows the start (Min Start Time) and end (Max StopTime) of thetime interval when all child drivers are running simultaneously. The child driversrun the test on a staggered basis. Total Test Time is the interval length, in seconds.The duration of the test is longer than the Total Test Time interval.



Errors: Total Test Errors reports the number of errors that occurred during theinterval between Min Start Time and Max Stop Time.


103

Performance metricsNOTESNUM uses the data from the NOTEBNCH process in the child driverResults files. The performance metrics are for the NotesBench test error-free timeinterval.

Test Run: Users = 70 NotesMark = 195 ResponseTime = 515 msec

(05/02/00 06:24:00 PM to 05/03/00 06:22:00 AM)

w The Users = field is the capacity metric for this test. It is the averagenumber of users on the SUT during the error-free period. The NOTESNUMprocess calculates this as an average value of the number of threadsexecuting across all drivers for the error-free period.

w The NotesMark = field is the throughput metric for this test. It is theaverage number of transactions per minute executed by all threads across allchild drivers during the error-free interval.

w The Response Time = field is the average response time for all transactionsexecuted by all threads across all the child drivers during the error-freeinterval.

105

Evaluating the ResultsThis topic explains how test results are interpreted and evaluated.

Summary Performance Across TestsYou can compute a single summary performance rating for the SUT by averagingthe NotesMark results from all tests that have a NotesMark result.

The notesnum command does not automatically compute the overall result.

Error-Free IntervalFor each NotesBench workload (test), the notesnum testname MaxChild commandcalculates test metrics for a period of time during which NotesBench encounteredno server errors. A set of metrics appears in the notesnum output along with thestart and end times for the error-free interval.

The error-free interval is a candidate for the NotesBench measurement period.

To qualify and to have its associated test-run metrics be valid, the error-freeinterval must:

w be at least 15 minutes but not more than 1 hour in length.w occur during a steady state.

Response timeThe response time cannot exceed 5 seconds.

Specific Results MetricsThe “number of sessions” result of the Idle Usage test provides an upper boundaryon the number of client sessions that the transport protocol and SUT can support.

Note All "number of sessions," "messages transferred per second," "message bytes transferred per second," "maximum users supported," and"throughput NotesMark results" must be measured, not interpolated orextrapolated.

105

Interpreting Web StatisticsThe HTTP-based workloads include several statistics that enable the testingorganization to quantify HTTP server performance and capacity. The Web tests aredesigned to simulate both static and interactive Web use. Benchmarking,quantifying performance and capacity, of an HTTP server is complicated. This isdue to the connectionless nature of the HTTP protocol. Fortunately, the Websimulation features of NotesBench make it easy to quantify not only static Webactivity (retrieving Web information), but also to simulate an interactive usercausing some action to execute on the HTTP server.

The following is a sample of the output generated from the Web Walker test:

HTTP Stats————Min——-Max——-Avg——————-HTTPStats——-Min—-Max——Avg—-

ConnectTime : 9 20 12 SendTime : 9 1 10

ReceiveTime : 1466 8303 5183 TotalTime : 1488 8321 5206

ActionsTime : 0 0 0

Byte/Page : 14467 16187 15900 Object/Page : 35 38 37

—————————Bytes—-Pages——Objs———————————-TimeOut—-Drops—-ActF-

Totals : 95402 6 225 Totals : 27 0 0

——————————Act———SErrs——CErrs———————————————-

Totals : 0 0 0

Summation of ConnectTime, SendTime and ReceiveTime, measuredin milliseconds

TotalTime

Time required to receive the HTTP command from the HTTP Server,measured in milliseconds

ReceiveTime

Time required to send the HTTP command to the HTTP Server,measured in milliseconds

SendTime

Time required to establish an HTTP connection, measured inmilliseconds

ConnectTime

The above times are “summed” across all Web objects on a page. For example, ifwww.iris.com has 4 graphics on it, ConnectTime represents the summation ofconnect times across all 4 graphics. The same principle is applied to all otherstatistics.

Total number of bytes of all text and graphics on aWeb page. Measured in bytes.

Bytes/Page

Time it takes for an HTTP server to execute a CGIprogram, script, or some other program, such assearching. Measured in milliseconds.

ActionTime

107

Total client errors. Examples of client errors areincorrect URLs and authentication failures.

CErrs

Total server errors. For some reason the server couldnot execute or complete a request. This is differentfrom time-outs, drops, and failed actions. In this case,the server detected that it did not have enoughresources, or access to resources, to complete therequest, such as retrieving a Web page or executing anaction.

SErrs

Total actions completed, such as successfullycompleting a Web-based order form, or search query.

Act

Total Actions Failed. An action represents a programexecuted on the HTTP server, such as clicking aSearch button. Errors may be due to occurrences suchas an action that could not complete, or the serverdropped the connection while waiting for an actioncompletion response.

ActF

Total dropped connections. Drops occur while data isin transit between a server and Web client.

Drops

Total number of HTTP server time-outs -- that is -- timed out waiting to connect to a Web server. This isthe most common error and is equivalent to the"Cannot connect to remote host error" as seen from abrowser.

TimeOut

Total Web objects retrieved. This value can also beinterpreted as the total number of hits.

ObjsTotal HTML pages retrieved.PagesTotal bytes transmitted by the HTTP server.Bytes

Total Objects, such as individual graphics,attachments, and other binary data linked on a page.

Objects/Page

Time it takes for an HTTP server to execute a CGIprogram, script, or some other program, such assearching. Measured in milliseconds.

ActionTime

w The first part of the statistics is a real-time computation of minimum,maximum, and average values across all threads (if the run is multi-threaded).The minimum, maximum and average values are computed for all timestatistics, bytes/page, and objects/page.

w The second part of the statistics output represents the cumulative totals fordisplayed statistics. The cumulative totals are generated for all other statistics.

Note that most Web browsers can spawn multiple threads that retrieve Webinformation such as graphics. NotesBench simulates a browser that can create up tofour threads at a time to fetch data.

The following statistics are significant for determining Web performance andcapacity data:

107

w Total Number of hits for a given time period, that is, hits/minute. This isrepresented by Total Number of Obj/Test Time.

w Total Number of Average Total Time and Average Bytes Per Page. Bothstatistics are required in order to derive any meaning.

Average Action Execution Time. Length of time required to complete a specific action given a specific load.

109

Evaluating System CostNotesBench test results can be used to evaluate the performance against the cost ofthe SUT.

This section defines the methodology to use to calculate the numerator of twoprice/performance ratios — price/NotesMark, and price/user. The fundamentalpremise is that whatever is tested is priced, and whatever is priced is tested. Theproposed system to be priced is the total of the SUT and the network componentsrequired to achieve the reported performance level.

Price and PerformanceTo figure out the cost, divide the throughput or capacity result of a NotesBench testinto the price of the tested system.

For example, one performance metric for a test is the maximum number of userssupported. If you calculate the price of the SUT as $4,500, and the maximum userssupported as 1000, the price/performance ratio is $4,500/1000 users or $4.50 peruser.

You can generate price/performance ratios for the performance metric of eachNotesBench test. The following two tables show examples of three systems usingNotesBench capacity and throughput test results. The comparisons can beinteresting. In this example, System A has the lowest price/performance ratio forsystem capacity, while System C has the best throughput ratio.

Price/Performance Results for System Capacity

$7.80/user3900500System C$10.00/user100001000System B$7.50/user4500600System A

Price/PerformanceSystem PriceMaximum UsersSupported

System Under Test

Price/Performance Results for System Throughput

$4.59/tpm3900850System C$4.76/tpm100002100System B$5.00/tpm4500900System A

Price/PerformanceSystem PriceTransactions perMinute (tpm)

System Under Test

109

Pricing Assumptions Pricing methodology assumptions:

w All hardware and software used in the calculations must be announced.Customers must be able to order the hardware and software. Products notalready generally released must include a full disclosure of the committeddelivery date for general availability.

w Generally available discounts for the priced configuration are permissible.w Generally available packaged pricing is acceptable.w Local retail pricing and discount structure should be used in each country for

which results are published.w Price should be represented in the currency with which the customer would

purchase the system.To meet auditing requirements, you must present confirmation of pricinginformation by an independent party. If price/performance is based on list price,you must be able to produce the referenced book; if based on discount price, youmust provide a copy of the quote. For testing organizations that have only indirectsales channels, pricing must be the actual, generally available pricing from indirectchannels and must meet all other requirements.

The intent of the pricing methodology is to allow packaging and pricing that isgenerally available to customers and to exclude promotional and limitedavailability offerings.

The intent is to benchmark the actual system that the customer would purchase.However, vendors typically announce new products and disclose benchmarkresults before the products actually ship. This practice is allowed, but thebenchmark must specifically exclude any use of “one of a kind” hardware/softwareconfigurations that the vendor does not intend to ship in the future.

Products must be generally available in the country where the system is priced.

For publishing results in a country other than the country for which the results areoriginally published, you are permitted to substitute local components that aredifferent from the original report, provided the substituted products are sold to thesame product description or specification. The intent is to encourage local countrypricing by allowing substitution of equipment for country-specific reasons — suchas, voltage or product numbering — which do not affect performance.

Calculating a System PriceInclude the component costs described in the following sections when calculatingthe price of the tested system.

111

For Driver SystemsExclude the price of the driver systems hardware and software from the pricecalculation, but you must describe the driver systems in the NotesBench disclosurereport.

For the Interconnecting NetworkExclude the price of local area network (LAN) or wide area network (WAN)hardware external to the SUT. Do not include the cabling and devices that connectthe drivers and the SUT. Do not include repeaters, bridges, switches, hubs, routers,gateways, or the software and interconnecting cabling for these items.

Include the price of a front-end processor, a communication controller, or otherdevice that executes server tasks in the place of server-based software.

Include the price of the networking hardware and software that runs on the SUT.

For the System Under TestInclude the cost of the server platform, for example:

w CPUsw Memory (level 1, 2, 3, and so on)w Controllersw Disk subsystems w Network interface cards w Data storage media w Other system components

Include server-based software, for example:

w Operating systemw Domino server software w Applications and software components configured to interact directly with

Domino; for example, an email program used in place of Notes mail

111

w Networking software — for example, TCP/IP software or a network operatingsystem such as Novell Netware

w Driver software for network interface cards Exclude maintenance costs.

For Destination SystemsExclude the cost of destination systems, which are the machines to which the SUTsends mail messages. However, you must include the destination systems in theNotesBench Disclosure Report.

Quantity Pricing As a hardware vendor, you may include more than one pricing calculation for aSUT configuration. The first price must be the list price for a quantity of one. Anadditional price may be a discount or quantity price.

For public disclosure reports, you must use a minimum pricing calculation for aquantity of one. Additional pricing calculations are optional.

Disclosing NotesBench Test ResultsThe Disclosure report is an important part of NotesBench benchmarking. Thereport can help ensure the reproducibility of NotesBench test results.

Disclosure Requirements You must include a disclosure report to comply with NotesBench specifications.The report is intended to allow others to duplicate the results of the test, given thesame tools and products.

Disclosure Report Contents This topic provides additional information about the content of the disclosurereport and it’s appendices.

Testing organization — A statement identifying the organizations that participatedin the benchmark.

113

Tuned Parameters — Settings for all customer-tunable parameters and options thathave been changed from the default values for NotesBench tests. These include:

w Notes environment variablesw Operating system parametersw File and disk system parametersw Network parameters

Configuration diagrams — Diagrams of the benchmarked system, the pricedsystem, and if applicable, a description of the differences.

w Show all processor, memory, network, storage, controller, and hardwareinterconnection components. The interconnection components should includethe type of LAN (for example, token ring or Ethernet), the bandwidth of theLAN, LAN devices (for example, bridges and repeaters), the server’s WANinterface protocol (for example, X.25 or Point-to-Point Protocol), a descriptionof router device(s) employed on the WAN, and the WAN networkingprotocols (for example, TCP/IP).

w For the configuration of the tested system, show the distribution of Notes files,that is, Domino Directory, log database, MAIL.BOX database, and testdatabases. Include key operating system files, such as paging file, swap file,and error log across the system’s storage media.

w If any file is mirrored, duplicated, striped or in any way not wholly containedon a single storage device, show the distribution of the file across devices.

w Although not included in the priced system, include a description andconfiguration of all source driver and destination systems.

Steady State Determination — A description of the method used to determine thatthe SUT had reached steady state prior to commencing the measurement interval.The NOTESNUM program will output the earliest time the measurement intervalcould have begun.

Measurement Period — A statement of the duration of the measurement period,greater than 15 minutes, for each test.

Sustained Test Tasks — A description of how work normally performed during asustained test, such as cache flushing, actually occurred during the measurementinterval.

Reproducibility — A description of the method used to determine thereproducibility of the measurement results.

113

Pricing — The total price of the entire tested configuration. A detailed list ofhardware and software used in the priced system. Each item must have vendor partnumber, description, release/revision level, and either general availability status orcommitted delivery date. If package-pricing is used, you must show the contents ofthe package. You should include additional priced items for each country-specificconfiguration. Disclose the basis of all discounts in your pricing calculation.

Performance results — A statement, table, or chart of all performance results.

Auditor’s statement — A letter, or other statement, specifying when the completeaudit report will become available and how to obtain a copy. It must include theauditor’s name, address, phone number, and a brief audit summary report thatstates that the benchmark met NotesBench requirements. The report may specifythat the letter is available upon request, instead of including it in the report.

Disclosure Report Availability If the hardware vendors choose to make reports publicly available, the reports mustbe audited by an authorized NotesBench auditing agency, and made available tothe NotesBench Consortium World Wide Web site for public access at thefollowing Web site:

http://www.notesbench.org

115

Special Disclosure ReportThis topic contains the sample Disclosure Report for your test results.

NotesBench based Test Report

(Audited)

Table of Contents

Optional (For additional vendor-definedinformation)

Appendix HPricingAppendix GGuidelines for Information UsageAppendix FNetwork Configuration File(s)Appendix ENotes Parameters (NOTES.INI)Appendix DOperating System Parameters ValuesAppendix CSystem ConfigurationsAppendix BOverall Test Setup and Software VersionsAppendix AStatement by AuditorSection 8Price/PerformanceSection 7ConclusionsSection 6AnalysisSection 5DataSection 4Test MethodologiesSection 3Benchmarking ObjectivesSection 2Executive SummarySection 1

Section Title

Explanation of the Sample Disclosure ReportEach of the sections in the sample Disclosure Report are described in this topic.The sample outlines the minimum information to be contained within the report.Additional information and data are optional but encouraged.

Executive SummarySummarize the test completion date, organization conducting tests, test setup, ahigh-level description of methodology utilized and the conclusion.

ObjectivesExplain the purpose of the reports. Several examples include relative updates basedon new hardware configurations, updates based on new hardware products, orupdates based on the availability of new software (operating systems or LotusNotes versions).

115

Test MethodologyExplain the test environment, procedures, system drivers, and systems under test.For example, “...The Lotus Domino R5 server was placed under various loadsusing five BlazeOcron 486DX/66 client systems connected over an isolated 802.3LAN using 10baseT medium. The server platform OS xxx parameter was variedbased on the xxxxxx test to optimized yyyyy. All other OS parameters wereconstant for the remaining benchmark sequences. For the replication throughputtest, an identically configured Domino server platform [was/was not] utilized.Other than administrative OS programs, no other programs were executing on theserver platform.”

DataSummarizes test results in graphical terms. Additional information can be providedin tabular and narrative format. The charts shown below are examples of the resultsyou can expect to see. Similar charts for all values are reported by NOTESNUM.

Lotus NotesMarks values for benchmarked configurations

117

Idle Usage

AnalysisOn a model-by-model basis, provide a narrative description of the data results andsystem behavior.

ConclusionsProvide a customer-centric recommendation for each server configuration.Recommendations can include price/performance information, extensibility ofplatform to support concurrent Companion Notes and/or Non-Notes applications,etc.

Price/PerformanceBasis of pricing method and price/performance results.

Statement by AuditorStatement from auditing organization that certifies the contents of the test reportand conformance to all test procedures.

117

Appendixes

Appendix A -- Overall Test Setup and Software VersionsNumber of Client SystemsNumber of Server platformsNetwork — For example: Localized, single, coaxial Ethernet segment with noWAN linksHigh-level test setup diagramDiscuss any special techniques — For example, disk caching, filecompression, etc.

Appendix B -- System ConfigurationsServer

All hardware and software components.Clients

All hardware and software components.

Appendix C -- Operating System ParametersConfiguration and parameter files. Highlight differences from default values.

Appendix D -- Notes ParametersInclude contents of NOTES.INI file.

Appendix E -- Network Configuration FilesLAN Protocol(s) TCP/IP and vendor name.Listing of Server and Client stack parameter settings.

Appendix F -- Guidelines for Information UsageProvide a pre-drafted statement based on the license.

Appendix G -- PricingProvide pricing information.

Appendix H -- OptionalAdditional vendor-defined information.

119

Auditing OrganizationsLotus works with independent auditing organizations to perform audits ofNotesBench results. Testing organizations interested in auditing must beindependent of a specific hardware manufacturer or operating system developer.

Information for specific auditing organizations is available on the Web athttp://www.notesbench.org.

NotesBench Auditing RulesThe auditing of NotesBench tests is an important part of Notes/Dominobenchmarking. Auditing ensures that workload results are both credible andreproducible. This section discusses the general rules for various types of reportsand the specific rules for each workload.

General Rules - Full Disclosure Reportw The Full Disclosure Report must contain all of the sections as defined in this

NotesBench Guide. For more information on the Full Disclosure Report, see the topic in thischapter “Sample Disclosure Report.”

General Rules - OnSite Auditw The number of Person documents in the Domino Directory on the SUT must

be at least the maximum number of users reported.w The number of Person documents in the Local address book must be at least

the maximum number of users reported.w For workloads that include mail routing, at least 90 percent of the mail

generated during the test must be routed during the test.w There is no limit on the amount of time that can be used to ramp-up users for a

test; however, the ramp-up time for each test must be disclosed in the FullDisclosure Report.

w Data volumes that use RAID must be configured as RAID5. However, iSeriessystems may run a combination of RAID1 and RAID5.

119

General Rules - OffSite AuditOnly NotesBench Consortium Member Companies can perform an off-site audit.

w General rules for on-site audit apply.w System configuration file for the SUT must be saved immediately prior to

starting the test and immediately after the test has shut down. For WindowsNT systems, use MSDRTP, or another diagnostic program that verifies theSUT configuration.

w Save the SUT files NAMES.NSF, LOG.NSF, NOTES.INI, RES files,MAILn.NSF (1 sample).

w Save the NOTES.INI file (1 sample) for the child driver.w Save the NOTES.INI file for the parent driver.w Save the destination server files NOTES.INI, LOG.NSF, Un.NSF (1 sample).w Run the SHOW STAT MEMORY and SHOW STAT DISKSPACE

commands once during the test. Run SHOW STAT MAIL once every halfhour after ramp-up.

General Rules - Partitioned Server AuditsSingle Domain (One Domino Directory)

w General rules for on-site and off-site audits apply.w Each partitioned server must be able to route mail to any/all destination

servers.w Run NOTESNUM across all res files to produce one NOTESNUM output.w Use one or more parent drivers.

Multiple Domain (Multiple Domino Directories)

w General rules for on-site and off-site audits apply.w The number of Person documents in the Domino Directory on the SUT for a

particular partition/domain must be at least the maximum number of usersreported for that partition/domain.

w The number of Person documents must equal at least the maximum number ofusers for the partition or domain of which the child driver is a member.

w Each partition or domain must use a separate parent driver.w Run NOTESNUM across the res files for each domain to produce a

NOTESNUM output for each partition or domain.

Auditing RulesThe following sections detail the specific auditing rules for each workload.

121

Cluster Mail TestChild Driver System NOTES.INI Variable Values:

w NormalMessageSize=1000w NumMessageRecipients=3w NumMailNotesPerUser=100

SUT NOTES.INI Variable Values:

w ServerTasks=REPLICA, ROUTER, UPDATEw SERVER_SHOW_PERFORMANCE=1w DEBUG_OUTFILE=server.txt

Minimum Test Runtime Duration after ramp-up - Domino R4.6 and later Dominoversions:

w 6 hours - Notesnum reported errors are not allowedMinimum Test Runtime Duration after ramp-up - Domino R4.5x and prior Dominoversions:

w N/A - new workload with NB 4.6R5Mail Test and R6Mail Test

NOTES.INI file settings on the child driver:

w NthIteration=6w NormalMessageSize=10000w NumMessageRecipients=3

NOTES.INI file settings on the SUT:

w ServerTasks=Router, LDAP, IMAPw Server_Show_Performance=1w Debug_Outfile=server.txt

Minimum test runtime duration after ramp up:

w 6 hours — NOTESNUM reported errors are not allowed.Miscellaneous:

w Average response time reported buy NOTESNUM must be less than or equalto 5 seconds.

R5IMAP Test and R6IMAP TestNOTES.INI file settings on the child driver:

IMAPHost=sut.company.comSMTPHost=sut.company.com

121

LDAPHost=sut.company.comClientHost=client.company.comRecipientDomain=company.comNthIteration=6NormalMessageSize=10000NumMessageRecipients=3NumMailNotesPerUser=100MailRecipientPercentUser=50MailRecipientPercentVolume=50

NOTES.INI file settings on the SUTServerTasks=Router, IMAP, LDAPServer_Show_Performance=1Debug_Outfile=server.txt

Minimum test runtime duration — after ramp up:

w 6 hours — NOTESNUM reported errors are not allowed.Miscellaneous:

w Average response time reported by NOTESNUM must be less than or equalto 5 seconds.

123

WorkGroup TestNOTES.INI file settings on the child driver:

HTTPHost=sut.company.comSMTPHost=sut.company.comLDAPHost=sut.company.comClientHost=client.company.comRecipientDomain=company.comNthIteration=6NumSharedNotes=(0.1 x maximum number of concurrent users)NumNotesLocalReplica=(0.01 x maximum number of concurrent users)NormalMessageSize=10000NumMessageRecipients=3NumMailNotesPerUser=100

NOTES.INI file settings on the SUT:

ServerTasks=Replica, Router, Update, HTTP, IMAP, LDAPServer_Show_Performance=1Debug_Outfile=server.txt

Minimum test runtime duration - after ramp up:

w 6 hours - NOTESNUM reported errors are not allowed. Miscellaneous:

w Average response time reported by NOTESNUM must be less than or equalto 5 seconds.

123

Auditing NotesBench ResultsAuditing NotesBench tests is an important part of Domino/Notes benchmarkingbecause it helps to ensure workload results that are both credible and reproducible.

Auditing StandardsAuditing standards for NotesBench include requirements that define the auditableperiod of time and the specifics for performing “audit runs” that create auditabledata sets.

Auditable PeriodTo meet NotesBench Consortium standards, the auditable period must meet theserequirements:

w The workload ran for the specified time after ramp up.w The SUT did not generate any errors that prevented a successful run of

NOTESNUM.w Extend uninterrupted for at least 15 minutes.w Response times must be a maximum of 5 seconds.w The NthIteration variable must be set to 6.

Note You must document all errors in the disclosure report.

Audit RunsUse the process below to run NotesBench for the purpose of creating auditable datasets that are to be made available to the public. NotesBench is used withServer.Planner for this purpose. Server.Planner rejects any data sets that have notbeen audited.

The process consists of 6 steps. Steps 1 - 3 are the responsibility of the vendor:

Run NotesBench workloads with Probe.The NOTES.INI file for the Probe driver must contain the MachineID andVendorDB variables. The MachineID must be unique for each SUTconfiguration tested. The VendorDB is usually the same for each company ordivision doing testing. For example, the XYZServerCompany VendorDBwould contain several data sets, each with a different MachineID. RunningNotesBench with Probe produces the RES files and the POUT file.

125

Run Notesnum.This will produce the DEBUG_OUT file containing the notesnum metrics forthe particular workload. It also places the RES file data and the POUT datainto the VendorDB, using the MachineID as the data set identifier.Submit VendorDB with unaudited data sets to the NotesBench Auditor.Vendors will also submit the associated report and additional files as requestedby the auditor for verification of data.Steps 4 - 6 are the responsibility of the auditor:Audit and certify the Server.Planner data set(s) within the VendorDB.Auditors have a special Server.Planner template that allows them to add aNotes digital signature to certified data sets. Data sets that have a digitalsignature for certification are then “readable” by Server.Planner for queryselections.Submit the VendorDB with the newly-certified data sets to the NotesBenchConsortium.The NotesBench Consortium WebMaster then posts the VendorDB with thenewly-certified data sets to the Consortium Web Site for public consumption.

Price InformationTo meet auditing requirements, you must present confirmation of pricinginformation by an independent party. If price/performance is based on list price,you must be able to produce the referenced book; if price/performance is based ondiscount price, you must provide a copy of the quote.

125

Auditors ChecklistNotesBench requires an independent audit of NotesBench results for disclosurereports that are publicly accessible. Auditors must be on the approved auditor list.

The primary focus of the auditing process is to provide Notes/Domino withcredible, reproducible Domino server information. To achieve this, the auditorverifies that the NotesBench benchmark was run correctly and that the NotesBenchreport includes all required information.

Hardware vendors may include additional information at their discretion.

When an audit is scheduled, specific requirements for the benchmark will besupplied as part of the statement of work. The following is an example of theauditor’s checklist.

Verify that:

The version of NotesBench is compatible with the release of Domino that isbeing tested.The test was run correctly, including the configuration, execution, andduration. The configuration of the test environment and SUT are correct. See the samplereport in this chapter. This includes items which may not be part of a “generic”product offering, such as additional I/O caching systems, or CPU cachingsystem. Compliance with NotesBench scaling rules was maintained. These not onlyinclude NOTES.INI file settings but also configurations — for example, thesize and content of the NAMES.NSF file.The method used to determine steady state was correct.Capacity and performance information is provided by, not extrapolated from,NOTESNUM.EXE outputs. All costs for the SUT are included in the price.Information about participating organizations is included.Any changes to default software installation files are included in the report.

127

Any software-based component removal, as compared to actual hardwareremoval, is included in the report. For example, OS-based CPU removal as inlimiting a dual-CPU configuration to uni-processor operation based onoperating system settings.Proper configuration of driver systems was used. Verify that any deviation in documented execution procedures are included inthe report. For example, delayed initiation of driver systems.

127

Appendix A Benchmark Specification Language

NotesBench scripts consist of statements of a simple procedural language — theNotesBench benchmark specification language. Each statement corresponds to oneor more Notes API functions with a similar name. Each script statement causes thesystem under test (SUT) to perform a specific Notes task. Each NotesBenchworkload is comprised of some number of these tasks.

Commands used with NotesBench Specification LanguageMany benchmark specification language statements include Notes environmentalvariables, taken from a NOTES.INI file. The language uses the values of theenvironmental variables in this file to customize the script for your configuration.

This section defines the syntax and semantics for each command used in thestatements in the Benchmark Specification Language.

ADDnum_notes summary_length nonsummary_length attachment_filename

folder_noteid

Creates num_notes new notes in a database. If num_notes is not specified, one noteis created. If summary_length is not specified, the length of the summary data is auniform random number between 1 and 100 bytes. If nonsummary_length is notspecified, the length of the nonsummary data is a uniform random number between100 and 300 bytes.

If attachment_filename is specified, the named file is attached to the new note as aseparate attachment field (that is, not part of the body). Each new note consists of:an author field with the current user’s name; a recipient’s field with the currentuser’s name; the ordinal number of the note as a summary item; the subject(summary) text item; the optional attachment item; and the body (non-summary)text item. If folder_noteid is specified, the new note is inserted into that folder.

BEGINLOOPMarks the start of the loop and the point to which the REWIND statement returnscontrol. A script can have one loop.

129

CHANGETO database_filename template_filename keepopen

Switches the current database context to database_filename. All followingstatements operate on the specified database. If the database doesn’t exist, a newdatabase is created using template_file. If the keepopen option is specified (whichis the string “-KeepOpen”), the database is not closed and reopened if it is alreadyopen.

CLOSECloses the current view.

CREATEFILE filename size

Creates a file of size length in bytes full of textual data. The distribution of wordsin the text file follows the distribution of words in typical English languagedocuments. The file is built either by generating properly distributed random wordsor by extracting a random size section from a public domain English textdocument.

DELETE num_notes

Deletes randomly selected num_notes from the database. If num_notes is notspecified, one note is deleted.

129

ENTRIESfirst_entry num_entries navigator

Starting at row first_entry in a view, returns information on the followingnum_entries rows. The num_entries rows of information is only returned if thereare more than first_entry rows in the view and first_entry + num_entries is lessthan the total number of rows in the view. Possible values for navigator are:NEXT, PREV, CURRENT, PARENT, CHILD, NEXT_PEER, PREV_PEER,FIRST_PEER, LAST_PEER, CURRENT_MAIN, NEXT_MAIN, PREV_MAIN,ALL_DESCENDANTS, NEXT_UNREAD, NEXT_UNREAD_MAIN, andPREV_UNREAD. The default value of navigator is NEXT. The informationreturned from the server to the client for each row consists of the NoteID (4 bytes),NoteClass (2 bytes), and all summary data.

FTENTRIESfirst_entry num_entries

Starting at row first_entry in the current full-text search results, returns informationon the following num_entries rows. The num_entries rows of information is onlyreturned if there are more than first_entry rows in the full-text search result andfirst_entry + num_entries is less than the number of hits in the full-text searchresult. The information returned from the server to the client for each row consistsof the NoteID (4 bytes), NoteClass (2 bytes), and all summary data.

FTSEARCHquery_kind

Performs a full-text search of the current database and establishes a full-text searchresult that can be read with FTENTRIES. The only kind of full-text query used inNotesBench is a Boolean OR of two random words, which is specified withquery_kind as TWO_OR_TERMS. Both of the chosen random words occur“infrequently” in the current database.

GETALLFetches the ID Table of all NoteIDs from the database. This command must bedone before other commands that operate on random documents in the database,such as STAMP, because those commands pick random notes out of this table.

131

NABUPDATEtype num_notes

Updates num_notes random type notes in the NAB database. If num_notes is notspecified, one note is updated. Possible values for type are PERSON, GROUP, andCONNECTION.

NAVIGATEnum_notes navigator

For num_notes times, opens a note and then navigates within the current view toanother note specified by navigator. If num_notes is not specified, it defaults to 1.Possible values for navigator are: NEXT, PREV, CURRENT, PARENT, CHILD,NEXT_PEER, PREV_PEER, FIRST_PEER, LAST_PEER, CURRENT_MAIN,NEXT_MAIN, PREV_MAIN, ALL_DESCENDANTS, NEXT_UNREAD,NEXT_UNREAD_MAIN, and PREV_UNREAD. The default value of navigator isNEXT.

NEWREPLICATEDBsource_filename new_filename

Makes a new replica database, called new_filename, based on source_filename .Initializes the new replica but doesn’t pull data from the source database.

OPENview_name view_options View_noteid

Opens the collection specified by either view_name or view_noteid and makes itthe current collection. If view_noteid i is used, this command translates to aNIFOpenCollection function. If view_name is used, this command translates to aNIFOpenCollection of the design collection, one or more NIFReadEntries, aNIFCloseCollection for the design collection, and then a NIFOpenCollection forthe specified view. The default value for view_options is none. Possible values forview_options are: REBUILD, INVALIDATE, VERIFY, NOUPDATE, andUPDATE.

PAUSEduration

Wait duration until performing the next command in the script.

131

POPULATEnum_notes folder_name

Ensures that there are num_notes notes in the current database. This commandlocks the database to prevent other users from simultaneously performing anotherPOPULATE command; gets the number of notes currently in the database; if thenumber of notes in the database is less than num_notes, adds enough notes to addup to num_notes; if folder_name is specified, adds the new notes to the namedfolder view; closes and unlocks the database.

REPLICATE server_name direction filename options

Replicates local filename with server_name by direction. If filename is notspecified, all files are replicated. Possible values for direction are PUSH, PULL,and BOTH. Possible values for options are UPDATE_COLL, CLOSE_SESSION,SUMMARY_ONLY, TRUNCATE, PRI_LOW, PRI_MED, and PRI_HIGH.

REWINDiterations

Moves the point of command execution in the script. If the script contains aBEGINLOOP statement, the next command executed is the one immediatelyfollowing the BEGINLOOP. Otherwise, the next command executed is the firstcommand in the script. The rewind operation is performed iterations times. Ifiterations is not specified, the REWIND command is executed indefinitely.

SENDMESSAGEmessage_size num_recipients NthIteration

Creates a mail message with message_size body text and sends it to num_recipientsrandom users. The random body text in the message is created by the same methodas in CREATEFILE. Message recipients are selected with a uniform distributionfrom the People in the NAB on the source driver system. (All replica copies of theNAB on the source driver system(s), system under test, and destination system(s),have the same contents.) The NthIteration parameter allows a message to be sentevery Nth script iteration.

SESSIONSCLOSECloses all open sessions. This statement only closes sessions opened withSESSIONSOPEN.

133

SESSIONSOPENserver_name num_sessions

Establishes num_sessions sessions between the client and server_name.

STAMPnum_notes summary_length

Selects num_notes random notes from the list of NoteIDs returned from GETALL.STAMP modifies a summary data field of length summary_length in each notewith the same random value.

UNREADnum_notes

Sets the database unread list for the current collection to contain num_notesrandom notes. This command may be used before a NAVIGATE with one of theunread navigation options to simulate reading a specific number of new documents.

UPDATEnum_notes summary_length nonsummary_length attachment_filename

Updatesnum_notes random notes in a database. If num_notes is not specified, onenote is updated. If summary_length is not specified, the length of the summary datais a uniform random number between 1 and 100 bytes. If nonsummary_length isnot specified, the length of the nonsummary data is a uniform random numberbetween 100 and 300 bytes. If attachment_filename is specified, the named file isattached to each updated note.

133

AADD command

described, 128Audit

OffSite, 120OnSite, 119

Audit runs, 124Auditable data sets

creating, 124Auditing Organizations, 119Auditing Rules, 119Auditing standards, 124Auditor's Checklist, 126Auto Check routine, 55Auto initialization, 8

BBEGINLOOP command

described, 129

CCapacity Planning Database

and Notesnum roll-up information,10

CHANGETO commanddescribed, 129

Checklistfor auditors, 126

Child driverand Location documents, 24installing templates on, 11NOTES.INI file, 29

CLOSE commanddescribed, 129

Cluster Mail Initialization testsetting up and running, 96

Cluster Mail testsetting up and running, 87

CREATEFILE commanddescribed, 129

DData

in Disclosure Report, 116Data sets

creating for audits, 124Databases

creating mail databases, 31creating vendor databases, 33

Debug_Outfileconsiderations for using, 61in initialization tests, 93

DELETE commanddescribed, 129

Destination serverssetting up, 31

Disclosure Reportcontents of, 112sample, 115

Driver filessetting up, 24

DSPA.NTFNotesBench Results Analyst, 34

DSPD.NTFNotesBench Results Decision Maker,

34DSPV.NTF, 33

NotesBench Results Decision Maker,34

Duration of workloads, 20

EENTRIES command

described, 130Environmental variables, 8Error-free interval, 105Executive Summary

in Disclosure Report, 115

FFTENTRIES command

described, 130FTSEARCH command

described, 130

GGETALL command

described, 130

HHard disk requirements, 7

IInitial Check routine, 54Initialization

auto, 8Initialization tests

explained, 93Installing, 21

LLocation documents

setting up for parent and childdrivers, 24

MMachine Identifier form

creating, 33Mail databases

creating, 31Memory requirements, 6Metric Summary

for workloads, 101Metrics

performance, 104Multiple Run

defined, 22MultiRun

defined, 22

NNABUPDATE command

described, 131NAVIGATE command

Index 1

Index

described, 131NBCONFIG.INI

using to override settings, 30Network Protocol Requirements, 6Network protocols

using with Probe_Interval, 11NEWREPLICATEDB command

described, 131NOTEBNCH

using to run tests, 55Notebnch executable

defined, 9NOTES.INI file

for child driver, 29for NotesBench, 35for parent driver, 25sample for ClusterMail test, 51sample for NotesBench, 45sample for Parent/Probe driver, 46sample for R5IMAP test, 50sample for R5iNotes test, 51sample for R5Mail test, 49sample for R6IMAP test, 48sample for R6iNotes test, 49sample for R6Mail test, 47sample for WorkGroup test, 52

NotesBenchinstalling, 21requirements, 5

NotesBench Configuration file, 30NotesBench Consortium

explained, 4NotesBench Results database, 33NotesBench scripts

explained, 12NOTESNUM

using to obtain test results, 56Notesnum executable

defined, 9NRPC Mail Initialization test

setting up and running, 94number of users

estimating, 22

OOffSite Audit, 120OnSite Audit, 119OPEN command

described, 131override settings

for NotesBench tasks, 30

PParent driver

and Location documents, 24NOTES.INI file, 25

PAUSE commanddescribed, 131

Performancedefined, 2

Performance metrics, 104Poller, 59POPULATE command

described, 132Price

defined, 2Price and Performance

calculating, 109Price/Performance

defined, 2Pricing assumptions, 110Probe

defined, 10running, 60setting up, 59used with MultiRun, 22using, 59

Probe commandused with evironmental variables, 10

Probe_Interval, 11

RR5IMAP test

setting up and running, 84R5iNotes test

setting up and running, 79R5Mail test

setting up and running, 76R6IMAP Initialization test

setting up and running, 98R6IMAP test

setting up and running, 66R6iNotes test

setting up and running, 72R6Mail test

setting up and running, 62REPLICATE command

described, 132Report

Disclosure, 112NotesBench test results, 33

Requirementshard disk, 7

memory, 6NotesBench, 5

Resultsauditing, 124evaluating, 105monitoring, 56

Results database, 33REWIND command

described, 132Rules

Auditing, 119

SScalability

defined, 2Scaling

NotesBench tests, 21Scaling rules

based on number of users, 8SENDMESSAGE command

described, 132Server

destination, 31SESSIONSCLOSE command

described, 132SESSIONSOPEN command

described, 133Settings

override with NBCONFIG.INI, 30Specification language commands

list of, 128STAMP command

described, 133Standards

used for auditing, 124Steady state

defined, 3SUT, 23System cost

evaluating, 109System price

calculating, 111System under test

and number of users, 58and Results file, 57

TTemplates

installing on child drivers, 11Test Methodology

in Disclosure Report, 116Test results

2 Index

evaluating, 105monitoring, 56reporting of, 33viewing, 102

TESTDISC.NSF, 10Tests

monitoring, 56planning duration of, 20

Threads parameterand driver Results file, 58

UUNREAD command

described, 133UPDATE command

described, 133Users

estimating number of, 22

VVariables

for publishing test results, 8NotesBench environmental, 8user-settable, 9

Vendor databasecreating, 33

WWeb statistics

interpreting in test results, 106WorkGroup test

setting up and running, 90Workload Metrics Summary, 101Workloads

defined, 2monitoring, 56planning duration of, 20

Index 3