effect of message size and number of clients on ws frameworks for cis*6650.01 service oriented...
TRANSCRIPT
Effect Of Message Effect Of Message Size and Number Size and Number of Clients on WS of Clients on WS
FrameworksFrameworksFor CIS*6650.01 Service Oriented For CIS*6650.01 Service Oriented ComputingComputing
Dariusz GrabkaDariusz GrabkaGerett CommefordGerett Commeford
Jack ColeJack Cole
April 3, 2007 2Effects on WS Frameworks
BackgroundBackground
Web Service Frameworks provide Web Service Frameworks provide functionality to applications that they functionality to applications that they host, including:host, including: SOAP message handlingSOAP message handling XML parsingXML parsing data marshalling into and out of application data marshalling into and out of application
spacespace Performance doing these mundane but Performance doing these mundane but
intensive chores can greatly affect the intensive chores can greatly affect the overall throughput of an applicationoverall throughput of an application
April 3, 2007 3Effects on WS Frameworks
GoalsGoals
The purpose of this experiment is to The purpose of this experiment is to compare major Web Service frameworks compare major Web Service frameworks to better under-stand their performance to better under-stand their performance under conditions of:under conditions of: increasing number of clientsincreasing number of clients increasing XML message lengthincreasing XML message length
We were particularly interested in We were particularly interested in conditions leading to unacceptable conditions leading to unacceptable service response (a denial-of-service service response (a denial-of-service situation)situation)
April 3, 2007 4Effects on WS Frameworks
WSTestWSTest
WSTest is a Web Service benchmark WSTest is a Web Service benchmark developed by Sun and augmented by developed by Sun and augmented by MicrosoftMicrosoft
Results published by both sides in 2004Results published by both sides in 2004 Each showed results that demonstrated superior Each showed results that demonstrated superior
performance for their own platformperformance for their own platform http://msdn2.microsoft.com/en-us/vstudio/aa700840.aspxhttp://msdn2.microsoft.com/en-us/vstudio/aa700840.aspx
Tests results were published for just a couple Tests results were published for just a couple of casesof cases
Tests were performed on a single client, and Tests were performed on a single client, and on a network of 50 clients, on a network of 50 clients, with similar resultswith similar results
April 3, 2007 5Effects on WS Frameworks
WSTestWSTest
Four testsFour tests EchoVoid – bi-directional empty messagesEchoVoid – bi-directional empty messages EchoStruct – echo an array of structures EchoStruct – echo an array of structures
containing an int, float, and stringcontaining an int, float, and string EchoList – echo a linked list of structuresEchoList – echo a linked list of structures GetOrder – receive a simulated purchase GetOrder – receive a simulated purchase
orderorder We have implemented EchoStruct as it We have implemented EchoStruct as it
provides a suitable range of testing provides a suitable range of testing optionsoptions
April 3, 2007 6Effects on WS Frameworks
Test EnvironmentsTest Environments
Hardware:Hardware: 3 identical 2.4 GHz Pentium 4 systems3 identical 2.4 GHz Pentium 4 systems 512 Mb RAM512 Mb RAM 1 Gigabit throughput switch1 Gigabit throughput switch
Software (running under Windows Server SE Software (running under Windows Server SE 2003):2003):
Each platform hosts a different frameworkEach platform hosts a different framework IBM WebSphere Application Server Express 6.1IBM WebSphere Application Server Express 6.1 Microsoft .NET with IIS 6.0Microsoft .NET with IIS 6.0 Sun JAVA Application Server PE 9Sun JAVA Application Server PE 9
April 3, 2007 7Effects on WS Frameworks
Test ConditionsTest Conditions
One server servicing one or two One server servicing one or two clients; no additional programs clients; no additional programs activeactive
All systems were removed from All systems were removed from external networksexternal networks Benchmarked at 11MBps total Benchmarked at 11MBps total
throughput, 98% efficiency between throughput, 98% efficiency between two machinestwo machines
April 3, 2007 8Effects on WS Frameworks
Test ApplicationsTest Applications The EchoStruct service was implementedThe EchoStruct service was implemented Servers were developed in the native Servers were developed in the native
language for the frameworklanguage for the framework Java for WebSphere and Sun ASJava for WebSphere and Sun AS C# for .NETC# for .NET A framework-free gSoap server was also A framework-free gSoap server was also
developeddeveloped Two types of clients were developed Two types of clients were developed
C# (using .NET), creating client threadsC# (using .NET), creating client threads standard C (using gSoap), creating client standard C (using gSoap), creating client
processesprocesses Document-literal style was used Document-literal style was used
April 3, 2007 9Effects on WS Frameworks
Test ApplicationsTest Applications Timing performed using the Windows Multi-media Timing performed using the Windows Multi-media
library with 1 mS resolutionlibrary with 1 mS resolution Two metrics: Two metrics:
Response time (start to end)Response time (start to end) User's view of service responseUser's view of service response
Latency (round-trip time)Latency (round-trip time) Removes client-side overhead from the measurementRemoves client-side overhead from the measurement
Care was taken to launch multiple clients Care was taken to launch multiple clients simultaneouslysimultaneously
Multiple transactions were used to ensure we Multiple transactions were used to ensure we were measuring service time and not client were measuring service time and not client deploymentdeployment
April 3, 2007 10Effects on WS Frameworks
Test ApplicationsTest Applications
Detailed experiments of:Detailed experiments of: Variable number of clients from 1 to 1200Variable number of clients from 1 to 1200 Variable number of array structures (and Variable number of array structures (and
therefore message length) from 1 to 1000therefore message length) from 1 to 1000 Variable number of iterations from 1 to Variable number of iterations from 1 to
20002000 Single experiments to provide insight Single experiments to provide insight
into possible future directionsinto possible future directions These will be discussed at the end of this These will be discussed at the end of this
reportreport
April 3, 2007 11Effects on WS Frameworks
Assumptions Assumptions
All hardware platforms and underlying All hardware platforms and underlying software perform essentially the samesoftware perform essentially the same Benchmarks shows some small variation in Benchmarks shows some small variation in
hard-drive write speeds (not considered)hard-drive write speeds (not considered) Windows client scheduling algorithms Windows client scheduling algorithms
are reasonableare reasonable Frameworks are reasonably configured Frameworks are reasonably configured
"out of the box""out of the box"
April 3, 2007 12Effects on WS Frameworks
Array Tests & Results Array Tests & Results
EchoStruct Array TestEchoStruct Array Test Each structure contains an integer, a Each structure contains an integer, a
float, and a string (which we set to 30 float, and a string (which we set to 30 characters)characters) Serializes to approximately 120 bytesSerializes to approximately 120 bytes
Tests were run for array sizes of n=5, 50, Tests were run for array sizes of n=5, 50, 100, 200, 500, and 1000.100, 200, 500, and 1000.
In each trial 5 simultaneous clients In each trial 5 simultaneous clients submitted 30 repetitions of the array echo submitted 30 repetitions of the array echo test => 150*n echoed transactionstest => 150*n echoed transactions
30 trials were performed and times 30 trials were performed and times gathered from each individual clientgathered from each individual client
April 3, 2007 13Effects on WS Frameworks
Array Test – Total Client Array Test – Total Client TimeTime
April 3, 2007 14Effects on WS Frameworks
Array Test – Latency Array Test – Latency TimeTime
April 3, 2007 15Effects on WS Frameworks
Client Tests & Results Client Tests & Results
EchoStruct Client TestEchoStruct Client Test Same structure as other testsSame structure as other tests Tests were run for n simultaneous Tests were run for n simultaneous
clients, n = 1, 5, 10, 20, 40, 60clients, n = 1, 5, 10, 20, 40, 60 In each trial, clients submitted 30 In each trial, clients submitted 30
repetitions of the 50 element array echo repetitions of the 50 element array echo test => 1500*n echoed transactionstest => 1500*n echoed transactions
Multiple trials were performed and Multiple trials were performed and times gathered from each individual times gathered from each individual clientclient
April 3, 2007 16Effects on WS Frameworks
Client TestsClient Tests
April 3, 2007 17Effects on WS Frameworks
Client Test – 600 ClientsClient Test – 600 Clients
April 3, 2007 18Effects on WS Frameworks
Client Test – 1200 Client Test – 1200 ClientsClients
April 3, 2007 19Effects on WS Frameworks
Iteration Tests & Results Iteration Tests & Results
EchoStruct Iteration TestEchoStruct Iteration Test Same structure as other testsSame structure as other tests Tests were run for n repetitions by each Tests were run for n repetitions by each
client, n = 5, 30, 100, 200, 500, 1000, client, n = 5, 30, 100, 200, 500, 1000, 20002000
In each trial, 5 simultaneous clients In each trial, 5 simultaneous clients submitted n repetitions of the 5 element submitted n repetitions of the 5 element array echo test => 25*n echoed array echo test => 25*n echoed transactionstransactions
Multiple trials were performed and times Multiple trials were performed and times gathered from each individual clientgathered from each individual client
April 3, 2007 20Effects on WS Frameworks
Iteration TestIteration TestIteration Tests (Simultaneous C Clients)
0
5
10
15
20
25
0 500 1000 1500 2000
Iterations
To
tal
Lat
ency
(S
)
WebSphere Sun AS9 .NET
April 3, 2007 21Effects on WS Frameworks
ConclusionsConclusions Performance of the 3 frameworks is similarPerformance of the 3 frameworks is similar Contrary to expectations, service times Contrary to expectations, service times
increase ~ linearly with both number of increase ~ linearly with both number of clients and size of transaction clients and size of transaction within limits within limits of our testingof our testing
As expected, number of iterations also As expected, number of iterations also produces a linear effectproduces a linear effect
Some results show interesting anomaliesSome results show interesting anomalies We believe that there is still much to be We believe that there is still much to be
discovered by pushing the frameworks discovered by pushing the frameworks harderharder
April 3, 2007 22Effects on WS Frameworks
Further InvestigationFurther Investigation
1. Comparison to non-Framework-1. Comparison to non-Framework-based Web servicesbased Web services By using a gSoap C/C++ server, we can By using a gSoap C/C++ server, we can
perform the same tests with minimal perform the same tests with minimal overheadoverhead
This will give a near best-case This will give a near best-case performance metricperformance metric Preliminary testing shows more than 50% Preliminary testing shows more than 50%
improvementimprovement
April 3, 2007 23Effects on WS Frameworks
Further InvestigationFurther Investigation
2. Distribute the clients over multiple platforms2. Distribute the clients over multiple platforms Running multiple clients on a single platform will Running multiple clients on a single platform will
eventually lead to saturation of the client, and eventually lead to saturation of the client, and results will no longer be reflecting only the server results will no longer be reflecting only the server loadload
We postponed this effect by developing very fast We postponed this effect by developing very fast clients. Better to use multiple client platforms. clients. Better to use multiple client platforms.
In a single test (WebSphere, 50 clients, array size In a single test (WebSphere, 50 clients, array size 50) we found that time was 5% 50) we found that time was 5% worseworse when 2 when 2 platforms were usedplatforms were used
Implies some optimization for single clientsImplies some optimization for single clients Implies non-saturation to that point in our testingImplies non-saturation to that point in our testing
April 3, 2007 24Effects on WS Frameworks
Further InvestigationFurther Investigation
3. Comparison with client and server 3. Comparison with client and server on same platformon same platform Some network transfer time can be Some network transfer time can be
removed from the experiment and then removed from the experiment and then used to characterize other resultsused to characterize other results
Preliminary testing shows that this Preliminary testing shows that this introduces interesting scheduling introduces interesting scheduling complicationscomplications
April 3, 2007 25Effects on WS Frameworks
Further InvestigationFurther Investigation
4. Comparison of different message 4. Comparison of different message stylesstyles RPC-literal and Document literal styles RPC-literal and Document literal styles
involve different activities from the involve different activities from the frameworksframeworks
Additional clients and servers are Additional clients and servers are requiredrequired Versions have been created using gSoap Versions have been created using gSoap Operation has been verified but no Operation has been verified but no
comparisons have been performedcomparisons have been performed
April 3, 2007 26Effects on WS Frameworks
Further InvestigationFurther Investigation
5. Comparison of message wrapping5. Comparison of message wrapping Even though the services are Even though the services are
functionally identical, the wrappings functionally identical, the wrappings may be slightly differentmay be slightly different
We have used EtherReal to snoop the We have used EtherReal to snoop the SOAP messages, but have not pursued SOAP messages, but have not pursued this investigation yetthis investigation yet