Raj JainThe Ohio State University

1

95-1347R1

PerformanceBenchmarking BOF

Contact: Jain@cis.ohio-State.Eduhttp://www.cis.ohio-State.Edu/~Jain/

Raj JainProfessor of Computer and Information Science

The Ohio State UniversityColumbus, OH 43210-1277

Raj JainThe Ohio State University

2

What is Performance Benchmarking?Why ATM Forum should work on it?Initial goals

Overview

Raj JainThe Ohio State University

3

Dictionary DefinitionBenchmark v. trans. To subject (a system)to a series of tests in order to obtainprearranged results not available oncompetitive systems.

From: The Devil’s DP DictionaryS. Kelly-Bootle

Raj JainThe Ohio State University

4

Networking BenchmarksBenchmarking Methodology Working group(BMWG) formed in January 1990RFC 1242 “Benchmarking Methodology forNetwork Interconnection Devices” written inJuly 1991.Defined a number of terms that arecommonly (mis)used by vendors

Raj JainThe Ohio State University

5

Terms Defined in RFC 1242Back-to-backFrame SizeFrame Loss RateLatencyThroughput

Raj JainThe Ohio State University

6

Throughput

Is the throughput 100 Mbps?NO!Correct definition: The maximum rate atwhich none of the offered frames aredropped by the deviceIssues: Must specify the frame size, path

155 Mbps 100 Mbps

Raj JainThe Ohio State University

7

Why not QoS Group?Quality of Service definitions:

Cell error ratioCell misinsertion ratioCell loss ratio

QoS parameters may be OK for VC setupbut not for customer buying the equipment.Benchmarks allow customers to comparedifferent products and make an apple-to-apple comparison.

Raj JainThe Ohio State University

8

Why not Testing Group?Testing group is working on conformancetesting.Conformance ≠ PerformanceConformance is mostly “Yes-No”Performance is mostly quantitative.

Raj JainThe Ohio State University

9

Can we use BMWG?Yes, we can. The terms and methodologyhave to be extended for ATM.ATM provides many variations of manyservices not provided by traditional networksWe need to identify meaningfulcombinations and their measurementconditions.

Raj JainThe Ohio State University

10

Why Do This at ATM Forum?ATM Forum has the most interest of makingATM successful (compared to IETF)Confusion caused by differing terminologyand differing benchmarks will eventuallylead to customer dis-satisfactionBetter customer information will contributeto more customer satisfaction and more salesand hence success of ATM.

Raj JainThe Ohio State University

11

Initial Goals ofPerformance Benchmarking BOFIdentify performance terms used by vendorsDefine these terms consistentlyDefine measurement methodology

Define key configurationsIdentify key issues and declarations

Define scripts or benchmark that can be usedby vendors and independent test labs tocompare product offerings

Raj JainThe Ohio State University

12

Summary

Performance benchmarking is important toavoid customer confusionIt would work only if there is a majorityagreement among the vendors/users.Benchmark ≠ QoS ≠ Conformance