ibm mashup center: v2public.dhe.ibm.com/software/dw/lotus/mashups/perf/mash_tuning.pdf · when...

IBM Mashup CenterRelease 2.0

Performance Tuning Guide

��

NoteBefore using this information and the product it supports, read the information in “Notices” on page 27.

This edition applies to version 2 of IBM Mashup Center and to all subsequent releases and modifications untilotherwise indicated in new editions.

© Copyright IBM Corporation 2010, .US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Introduction to Mashup Centerperformance tuning. . . . . . . . . . 1

Understanding the benchmarkenvironment for performance tuning . . 3

Tuning the WebSphere ApplicationServer instance of Lotus Mashups . . . 5

Tuning the WebSphere ApplicationServer instance of MashupHub . . . . . 7

Tuning IBM HTTP Server . . . . . . . 9

Tuning IBM WebSphere Edge Server . . 11

Tuning the IBM DB2 database . . . . . 13

Tuning MashupHub feeds . . . . . . 15

Understanding general performancecharacteristics of the catalog, feedgenerators, and data mashup builder . 17

Addendum: Performance tuning forMashup Center 2.0.0.2 . . . . . . . . 21

Notices . . . . . . . . . . . . . . 27Trademarks . . . . . . . . . . . . . . 29

© Copyright IBM Corp. 2010, iii

iv IBM Mashup Center: V2.0

Introduction to Mashup Center performance tuning

When tuning your Mashup Center environment for optimum performance, youshould tune and configure the individual systems and components used in yourenvironment. This guide discusses some general concepts about performancetuning and describes the configuration details used in the measurementenvironments used by the IBM® performance team.

Although Mashup Center is a single product, it is composed of the following twomain components: Lotus® Mashups and MashupHub. The Lotus Mashupscomponent includes the mashup builder environment where you create the pagesand spaces that make up your mashup applications. The MashupHub componentincludes the catalog, data mashup builder, and set of feed generators. When tuningyour environment, each of these components can be configured separately.

When tuning Lotus Mashups, we observed the following details during thebenchmark testing:v The main factor that affected performance was the speed of the WebSphere

Application Server central processing unit (CPU). As a general rule, the fasterthe CPU, the higher the throughput.

v The Java™ virtual machine (JVM) heap did not restrict performance in the IBMtesting environment.

v The bandwidth of our network did not restrict performance when using a100–megabit network with browser cache settings and GZIP encoding. For moreheavy workloads, we recommend that you connect a reverse proxy server suchas IBM WebSphere Edge Server, IBM HTTP Server, and WebSphere ApplicationServer using a 1000–megabit network.

v The throughput results from the IBM testing cannot be achieved with defaultsettings.

We also observed some recommendations for tuning performance of MashupHub.For example, the level of performance mainly depends on the feeds in the catalog.This dependency is described in more detail at “Tuning MashupHub feeds” onpage 15. In addition, we observed the following details:v When server-side cache is enabled for MashupHub feeds, performance is limited

mostly by the database disk I/O or the MashupHub application server's CPU.As a general rule, increasing the number or speed of the database hard disks orapplication server CPU may result in increased throughput.

v In our testing environment, we found that the JVM heap did not restrictperformance when the MashupHub feed cache was disabled. If yourMashupHub feed cache is enabled, do not set server-side cache memory toolarge, and make sure that MashupHub has enough JVM heap. We recommendthat you start at 25% of the maximum heap size

v Feed performance mainly depends on the computation complexity of the feed.As a general rule, the more complex the computation, the more limited theperformance of the feed. Increasing the number or speed of the CPUs mayincrease feed throughput.

v MashupHub performance depends mainly on the size of the feeds. Larger feedsizes may decrease throughput.

© Copyright IBM Corp. 2010, 1

2 IBM Mashup Center: V2.0

Understanding the benchmark environment for performancetuning

This topic describes the tuning and configuration settings used by the IBMperformance team to conduct performance measurements of specific MashupCenter workload scenarios.

The goal of the following information is to make you aware of the parameters thathad the greatest impact on performance. Although we describe the exact values weused to in our testing environment, we do not necessarily recommend that you usethe same parameters and values in your environment. In general, tuning andcapacity depend on your workload scenario, measurement environment, and otherfactors. When tuning individual systems, we recommend that you start with abaseline measurement that is specific to your actual environment. Then, afterestablishing a baseline, monitor performance metrics to evaluate each parameterchange for effectiveness. Be sure to make changes in a controlled, incrementalmanner.

Note: We used the following parameter settings strictly to measure performanceand scalability in a test environment, not a production environment. If you aretesting in a production environment, you may need to analyze your individualsettings in a more comprehensive manner.

A key characteristic of highly scalable configurations is a distributed topology inwhich individual software components run on dedicated hardware. Thisdistributed topology minimizes the need for shared resources and allows granularincreases in capacity for individual components as they are needed. In an effort toadhere to this topology model, we used the following five separate serversconnected by 1000 Mbps Ethernet in our benchmark testing:v WebSphere Edge Components Version 6.1 to act as the reverse proxy serverv A Mashup Center server, which consisted of the following components:

– IBM HTTP Server 6.1– WebSphere Application Server instance of Lotus Mashups– WebSphere Application Server instance of MashupHub

v IBM DB2® database server for Lotus Mashupsv IBM DB2 database server for MashupHubv IBM Tivoli® Directory Server

Here is a graphical diagram of our deployment topology:

Note: In the following diagram, note the target database and server. These are theterms that we used internally for our benchmark testing to represent thethird-party or legacy server, service, or database you would use in a productionenvironment. They are not part of Mashup Center but are accessed by MashupCenter.


All of the servers pictured in the diagram above are connected by a 1000 Mbpsnetwork.


Tuning the WebSphere Application Server instance of LotusMashups

This topic describes the details and results of the performance testing of theWebSphere Application Server instance of Lotus Mashups.

Note: In our measurement environment, Lotus Mashups and MashupHub wereinstalled on the same server but with separated instances of WebSphereApplication Server. Lotus Mashups is composed of standard Java Web applications.Therefore, typical tuning practices for WebSphere Application Server applicationswere used in our testing. In your environment, at a minimum, you should size andmonitor database connection pools, the application server thread pools, and theJVM heap. See the following sections for details about each of these components.

Database connection pools

The default maximum WebSphere Application Server Java Database Connectivity(JDBC) pool size is only 10 Lotus Mashups connections. You should increase thisnumber of connections for all types of installations. In our benchmark testing, eachapplication that we measured had a maximum of 100 Lotus Mashups connections.Certain applications had an even higher number of connections. We recommendthat you set this value to 100 after installation and tune your testing environmentas required by your workload. Also, be sure to confirm that the back end databasewill accept the total number of Lotus Mashups connections from all JDBC pools.

In addition, the initial prepared statement cache size is also too small. As a result,we set the cache size value 100.

WebContainer thread pools

The default maximum size of the WebSphere Application Server WebContainerthread pool of Lotus Mashups is 50, which is too small. In our measurements, weset this size to at least a maximum of 100, or 25 threads per processor core, toallow more simultaneous requests on the WebSphere Application Server instance ofLotus Mashups.

You can monitor the actual number of threads being used in the pool by enablingthe appropriate performance counters and launching the IBM Tivoli PerformanceMonitor.

JVM heap

The default size for each WebSphere Application Server JVM is not set. Themaximum heap size of Lotus Mashups should be at least 768 MB. To obtain ourresults, we used a maximum heap size of 1024 MB.

If your machine has more than one processor, we recommend parallel garbagecollection. You can enable this setting for Lotus Mashups with –XgcthreadsN, whereN is double the CPU core number.

Note: Note that when running larger JVM sizes, all the JVMs running on aphysical server must fit into the physical memory of the server.


In our measurements, one critical setting was enabling the generational garbagecollector. Enabling this setting with -Xgcpolicy:gencon for Lotus Mashups meansthat the JVM reserves several small areas of the heap (two by default) for all newobjects called nursery spaces. When one of these areas becomes full, only this areais scanned for dead objects. Any objects still living are moved to a differentnursery space. If, after another garbage collection, the second nursery space stillhas an object in it, these objects are moved to the main heap. This action results inmany short garbage collection cycles instead of less frequent, longer garbagecollection pauses. In our measurements, with generational collection enabled,garbage collection cycles occur every one to two seconds but take less than 50milliseconds to complete. This allows higher throughput since there is less chanceof transactions being held up by a one-second pause for garbage collection, whichwould happen with the default collector.

We recommend that you enable verbosegc logging and monitor heap use with anyof the settings described above.

Multipart

Multipart is a new Mashup Center 2.0 feature. Server-side Lotus Mashups providesa way to handle multiple requests to a POC DataSource or a POC DataSink withone single request to the mashup source or sink. This is interesting if the mashuphandler is accessed through an HTTP request, because then multiple logicalrequests can be tunneled through one single physical HTTP request.

In addition, the multipart requests sent by the HTTP GET method can be cachedby the HTTP Server or Edge Server, so enabling the multipart feature correctly canimprove browser-side performance and result in almost no regression forserver-side performance. The negative impact on server-side performance ofmultipart is minimal. You can locate the following five multipart attributes of theWebSphere Application Server instance of Lotus Mashups by navigating toResource → Resource Environment → Resource Environment Providers →Mashups_ConfigService → Custom properties.v com.ibm.mashups.multipart.enabled and pageLoadOptimization: These two

attributes enable and disable the multipart feature. Values are either false ortrue. Set these values to true to enable multipart.

v com.ibm.mashups.multipart.correlatehosts: With the default value of false, thebrowser sends only one multipart request to retrieve resources, even theresources are distributed on a different host. This may take a long time. Set thevalue to true to allow the browser to send different multipart requests fordifferent hosts.

v pageLoadOptimizationTheme: Keep the default value of true to enable themultipart feature for theme files. Enabling this feature may improve client-sideperformance. Also, enabling this feature has no negative impact on server-sideperformance.

v pageLoadOptimizationAppWidgets: Keep the default value of false to disable themultipart feature for application widget files. Enabling this feature decreasesserver-side performance, especially under complex deployment and networkconditions.


Tuning the WebSphere Application Server instance ofMashupHub

This topic describes the details and results of the performance testing of theWebSphere Application Server instance of MashupHub.

Similar to Lotus Mashups, tuning the WebSphere Application Server instance ofMashupHub also involves setting database connection pools, application serverthread pools, and JVM heap properties. An additional configuration component isWebSphere® DynaCache for MashupHub feeds.

Database connection pools

In our testing, the maximum number of database connection pools is 50. Inaddition, the default prepared statement cache size is also too small. As a result,we set the cache size value to 100.

You can monitor the results of these settings in various ways, for example byenabling the IBM Tivoli Performance Monitor or from tracking high-water marksin database snapshots.

WebContainer thread pools

The default maximum size of the WebSphere Application Server WebContainerthread pool of MashupHub is 50, which is too small. In our measurements, we setthis size to at least a maximum of 100, or 25 threads per processor core, to allowmore simultaneous requests on the WebSphere Application Server instance ofMashupHub.

You can monitor the actual number of threads being used in the pool by enablingthe appropriate performance counters and launching the IBM Tivoli PerformanceMonitor.

JVM heap

The default size for each WebSphere Application Server JVM is not set. Themaximum heap size of MashupHub should be at least 768 MB. To obtain ourresults, we used a maximum heap size of 1024 MB.

If your machine has more than one processor, we recommend parallel garbagecollection. You can enable this setting for MashupHub with –XgcthreadsN, where Nis double the CPU core number.

Note: Note that when running larger JVM sizes, all the JVMs running on aphysical server must fit into the physical memory of the server.

In our measurements, one critical setting was enabling the generational garbagecollector. Enabling this setting with -Xgcpolicy:gencon for MashupHub means thatthe JVM reserves several small areas of the heap (two by default) for all newobjects called nursery spaces. When one of these areas becomes full, only this areais scanned for dead objects. Any objects still living are moved to a differentnursery space. If, after another garbage collection, the second nursery space still


has an object in it, these objects are moved to the main heap. This action results inmany short garbage collection cycles instead of less frequent, longer garbagecollection pauses. In our measurements, with generational collection enabled,garbage collection cycles occur every one to two seconds but take less than 50milliseconds to complete. This allows higher throughput since there is less chanceof transactions being held up by a one-second pause for garbage collection, whichwould happen with the default collector.

We recommend that you enable verbosegc logging and monitor heap use with anyof the settings described above.

WebSphere DynaCache

MashupHub supports cache in the server-side JVM memory. MashupHub usesWebSphere DynaCache to cache feeds. The cache instance is called MashupHub feedcache, which can be found in Resources → Cache instances → Object cacheinstances.

You can modify the cache size according to your needs. We recommend that yourJVM heap setting have 40 percent of free heap after caching. In a defaultinstallation, memory size is set to 256 MB, which is the value we used in ourtesting.

CacheSize is the number of entries to put in the cache. This value should be largeenough to hold the cache entries that are associated with the more frequently usedfeeds in the application mix. In a default installation, the number of entries is setto 2000, which is the value we used in our testing. When a feed is enabled forserver-side cache, the feed data will be cached in the DynaCache.


Tuning IBM HTTP Server

This topic describes the details and results of the performance testing of IBM HTTPServer running alongside WebSphere Application Server.

During the testing process, we changed the following default settings in the HTTPServer httpd.conf file:v We enabled mod_status.so in order to monitor the number of worker threads in

use.v We set MaxKeepAliveRequests to 0.v We set KeepAliveTimeout to 5 so that the value was less than the think time in

the workload scripts. The purpose of using a value of 5 was to prevent theautomated test clients from consuming the entire pool of HTTP workers. This isnot a risk for real users, so plan to increase this value for live deployments.

v We increased ThreadsPerChild to 750 and ThreadLimit to 2048.v We set MaxRequestsPerChild to 0.

We recommend that you enable monitoring and adjust the values described aboveas appropriate for your environment. When deploying a proxy server, for example,you will need fewer numbers of threads (ThreadsPerChild directive).

Caching rules

In our testing, we added rules to the httpd.conf file to allow caching of staticcontent. These rules, which are specified by a URI pattern, append aCache-Control: max-age header to all content that contains the pattern. Thisheader forces browsers to cache content internally, and no subsequent requests aremade for that content for an entire day. See the following HTTP Serverconfiguration example:LoadModule headers_module modules/mod_headers.so

SetEnvIfNoCase Request_URI (?:/widget-catalog/|mum/undefined/js/|mum/registry/|mum/themes/|mum/js/|mum/images/|mum/_combined.css|/charts/|/dataviewer/|/themes/|/regularExpressionAdapter/|/scriptAdapter/)file-pathHeader set Cache-Control "public,max-age=86400" env=file-path

SetEnvIfNoCase Request_URI (?:mashuphub/client/images|mashuphub/client/other|mashuphub/client/styles)hub-file-pathHeader set Cache-Control "public,max-age=86400" env=hub-file-path

SetEnvIfNoCase Request_URI (?:mum/mycontenthandler/mm/mashup/)multipart-file-pathHeader set Cache-Control "public,max-age=86400" env=multipart-file-path

The following pattern should not be cached:SetEnvIfNoCase Request_URI (?:d3mmtgt12.perflab.com|mum/enabler|mum/mycontenthandler|bootstrap|mashuphub/client/plugin|mashuphub/client/mm) ignore-file-pathHeader set Cache-Control no-cache env=ignore-file-path


In addition, if you want to modify the Cache-Control of other default IBM widgetsor your custom-made widgets, you can add the URI patterns of your widgets inthe httpd.conf file, for example:SetEnvIfNoCase Request_URI (?:/my_widget_context_root/)my-file-pathHeader set Cache-Control "public,max-age=86400" env=my-file-path

The lifetime is the time it takes to cache the feed in seconds. The cache for one dayis 86400 seconds.

Note: By default, the Cache-Control feature is enabled for mm.was.ear, which willcause little regression on server-side performance. Here is an example:<filter><filter-name> mmCacheHeaderFilter</filter-name><filter-class> com.ibm.mm.was.filter.CacheHeaderFilter</filter-class></filter>

In order to achieve the best performance on both the client and server, werecommend that you comment out the mmCacheHeaderFilter in filter_web.xml andweb.xml in mm.was.ear\mm.runtime.was.war\WEB-INF\ and then useWebSphere Application Server to redeploy mm.was.ear.

Compress rules

To reduce network traffic, we enabled GZIP for JavaScript™:LoadModule deflate_module modules/mod_deflate.soAddOutputFilterByType DEFLATE application/x-javascript

Note: By default, the GZIP feature is enabled for mm.was.ear, which will causesome regression on server-side performance. Here is an example:<filter><filter-name> mmCompressionFilter</filter-name><filter-class> com.ibm.mm.was.filter.CompressionFilter</filter-class></filter>

As described in this set of tuning topics, almost all the resources are compressedon the edge server. The exception is JavaScript, which is compressed on the HTTPServer. The purpose of the JavaScript compression is to avoid functional problems.

In order to achieve the best performance on both the client and server, werecommend that you comment out the mmCacheHeaderFilter in filter_web.xml andweb.xml in mm.was.ear\mm.runtime.was.war\WEB-INF\ and then useWebSphere Application Server to redeploy mm.was.ear.


Tuning IBM WebSphere Edge Server

This topic describes the details and results of the performance testing ofWebSphere Edge Server.

Required directives

In order for caching to work correctly in Mashup Center, you must set several keyvalues in the ibmproxy.conf file for WebSphere Edge Server. For example, youshould set the SendRevProxyName value to yes. In addition, you should set theReversePass rules to something similar to the following example:ReversePass http://appserver/* http://proxyserver/*ReversePass https://appserver/* https://proxyserver/*

These values ensure that HTML links to the application are rewritten with thereverse proxy server name instead of the application server host name. Note that itis safe to pass all URLs for the server if only Mashup Center applications areinstalled on the application server. In the example above, these URLs are noted bythe * character.

When a cache object is re-validated with the back end server, the requests for thesame resource continue to be sent to the back end server. This means that a largenumber of requests for the same resource may hit the back end server, which cancause high resource usage. You can prevent this situation by enabling the followingdirective: KeepExpired On. An expired or stale copy of the resource gets returnedfor the brief time that the resource is updated on the proxy.

Tuning directives

The default memory sizes for caching are very small. If you are working with adedicated server, you can increase the memory cache size to a significant fractionof the total system memory. For our benchmark testing, we configured the proxywith a memory cache of one gigabyte using the following directive: CacheMemory 1G

Note: Because the proxy server is a 32–bit application, cache memory is limited toabout 1500 MB. Larger sizes must be moved to a cache that is backed up by a disk.

Edge Server has its own set of threads used for incoming Mashup Center requests.You should change the following setting to be at least as large as theThreadsPerChild setting for the IBM HTTP Server:MaxActiveThreads 500

By default, a connection coming into the proxy requires a connection to the backend servers if the request content is not cached. This connection can cause ahigher-than-necessary usage of the HTTP Server connection threads. By turning onthe proxy’s connection pool, as shown in the following example, Mashup Centerrequests from front end clients can share existing back end Mashup Centerconnections and reduce overhead on the HTTP Server:ServerConnPool OnMaxSocketPerServer 60


Finally, you can configure another setting to allow more requests to use the samepersistent HTTP connection to the server. By default, only five requests are allowedper connection. You can modify this setting to allow more connections. This updatecan be especially beneficial when using Secure Sockets Layer (SSL), since an SSLhandshake can take a significant amount of time to complete. Allowing morerequests per connection allows the SSL connection to stay open longer, whichmeans that less handshakes are necessary. You can change this setting with theMaxPersistRequest directive. Note that this setting was not required for our testingenvironment.

To reduce network traffic, we also enabled GZIP on the proxy server for all tests,as shown in the following example. After GZIP is enabled, the zipped files arespecified by MIME type. For our testing, all non-image files were compressed.CompressionFilterEnable <Edge Server install location>\Bin\modz.dll

CompressionFilterAddContentType text/html,Image/bitmap,text/xml,application/atom+xml,application/xml,text/css,multipart/mixed


Tuning the IBM DB2 database

This topic describes the details and results of the performance testing of DB2.

General tuning for both Lotus Mashups and MashupHubdatabases

DB2 tuning generally revolves around these three R actions:v runstats: update statistics using the following command: db2 "runstats on

table <schema>.<tablename> with distribution and detailed indexes all".You should update statistics regularly to ensure that queries run optimally.

v reorg: reorganize tables using the following command: db2 "reorg table<schema>.<tablename>". Based on our measurements, you only need toreorganize tables occasionally after the database is populated.

v rebind: rebind the database using the following command: db2rbind -l logfileall -u <db2admin id> -p <db2admin password>. Rebinding the database isneeded only intermittently, for example after any database schema change suchas an index addition or column change.

To review table and index reorg needs, run the following command from a DB2command window: db2 "reorgchk current statistics all table all" >reorgreport.txt. This command creates a TXT file with output that you canreview. In the last column, if you see a * character, then you should plan toreorganize the table or index in the near future. If you see two or more *characters, you should reorganize the table immediately. Most times, the indexesfor a table will have at least some * designators. This is because a clean andsequential physical order of many indexes for a single table is not possible.

To review your database configuration settings, run the following command from aDB2 command window: db2 "get db cfg for <yourdb>".

We also recommend that you review the DB2 general error log frequently. There,you may see true process stopping errors as well as general warnings. For aMicrosoft® Windows® server, you can locate this file at %DB2PATH%/db2/db2diag.log,where db2 reflects that your database is in the default DB2 instance named DB2. Itcould specify a different folder if you are using another instance name.

Buffer pool tuning for both Lotus Mashups and MashupHubdatabases

Since disk I/O can be a performance bottleneck for Mashup Center, werecommend that you set up and configure a powerful disk array. To improveperformance, you can also separate the Lotus Mashups and MashupHub databasesto different hosts as well as add more disks to the table space to share the I/Oworkload.

The default size of the buffer pool MASHUPSBP32K is too small. We recommend thatyou issue the following command to increase the size to 2000: db2 alterBUFFERPOOL MASHUPBP32K IMMEDIATE SIZE 20000.


Large object (LOB) data tuning for the MashupHub database

The queries to LOB data on the MashupHub database are not very efficient. Thiskind of data does not stay in the buffer pool memory, so each LOB query is a diskaccess, and no real DB2 caching takes place. Several optimization methods areavailable. One possible method is to reorganize the binary large object (BLOB) andcharacter large object (CLOB) data and turn off the file system caching for the tablespace.

To reorganize the BLOB and CLOB data, issue the following two commands:db2 "reorg table <schema>.resources longlobdata"db2 "reorg table <schema>.plugindata longlobdata"

To turn off the file system caching for the table space, issue the followingcommand:db2 alter tablespace <tablespacename> no file system caching

In addition, to improve server-side capacity, you may consider adding a new diskstorage for the MashupHub table space in order to share the overhead on the diskI/O.


Tuning MashupHub feeds

This topic describes the details and results of the performance testing ofMashupHub feeds.

Enabling cache

You can enable caching for MashupHub feeds to reduce the resource cost on theMashupHub server as well as the MashupHub database server. To enable cachingfor a feed, do the following steps:1. Locate the feed that you want to modify in the catalog.2. Open the details page for the feed.3. In the Advanced section, expand the Caching data section.4. Select the Cache data from the feed option, and specify the number of seconds

the data should be cached. The lifetime is the time to cache the feed in seconds.86400 is cache for one day.

Now, you feed details page should look similar to the following example:


Limiting feed size

To improve performance, try to limit the size of your feed by filtering outunnecessary data. As a general rule, smaller feeds perform better than larger feeds.

Improving disk I/O

Disk I/O is an important performance factor for MashupHub feed access. Thebetter the disk I/O, the greater the feed performance. For example, to improve diskI/O, you can add a new disk storage for the MashupHub table space to share theoverhead on the disk I/O and improve server-side capacity.

Avoiding negative impact on the target database

The content in relational feeds is stored in a separate database. In our environment,we call this type of database the target database. When accessing a relational feed,MashupHub connects to, logs in to, queries, and disconnects from the targetdatabase. Frequent feed access creates significant overhead when connecting to,logging in to, and disconnecting from the database. For example, when workingwith some version of DB2 using Microsoft Windows, a heavy disk I/O occurs onthe target database for authentication, which may be a limiting factor for databasefeeds. Here are two possible tuning methods that you can try in your ownenvironment.

In this first approach, you can create a data source in WebSphere ApplicationServer for the target database. Without a data source, each time a request is sent toa database feed, a database connection has to be created between server and targetdatabase. These connection operations use a lot of disk I/O resources on the targetdatabase. To learn how to configure a data source in WebSphere ApplicationServer, see Configuring the WebSphere Application Server data source athttp://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/twim_fedmap_datasconf.html. Tolearn more about creating a database using a data source, see Creating a feed froman enterprise database (JDBC) at http://publib.boulder.ibm.com/infocenter/mashhelp/v2/index.jsp?topic=/com.ibm.muh.useradmin.doc/userhelp/feedhelp/help_feeds_sql.htm.

In this second approach, if you are using a DB2 server that is older than version9.3, set the following DB2 variable to improve the connection times to the DB2server: DB2_GRP_LOOKUP = LOCAL,TOKENLOCAL or DB2_GRP_LOOKUP = token. For moreinformation, see the following IBM support document: http://www-01.ibm.com/support/docview.wss?uid=swg1JR26395.


Understanding general performance characteristics of thecatalog, feed generators, and data mashup builder

This topic describes general performance characteristics of the catalog, feedgenerators, and the data mashup builder. This information was included in theMashup Center 2.0 release notes, which are located on the Mashup Center supportsite.

Performance characteristics of the catalog, feed generators, andthe data mashup builder

The main performance characteristics of the catalog, feed generators, and the datamashup builder performance are as follows:v The aggregated size of data that is processed by MashupHub.v The complexity and the number of operations in typical data flow mashups.v The number of concurrent users.v The size and usage patterns of the mashup catalog, for example, the number of

objects in the catalog and the number of searches that are supported.v The caching and feed response characteristics when using network data sources.v The memory allocated to the server

The guidelines in this section are based on WebSphere Application Server with thefollowing configuration:v JVM initial heap size: 384 MBv Maximum heap size: 1024 MBv JVM parameter: -Xgcpolicy:gencon -XgcthreadsN (N is double the CPU core

number)v Generational garbage collector enablement: Set by adding the setting

-Xgcpolicy:gencon to the JVM generic arguments in WebSphere ApplicationServer. Enable verbosegc logging and monitor heap use with the previoussettings. All the JVMs running on a physical server must fit into the physicalmemory of the server. See the instructions for changing the JVM settings inWebSphere at the following Web site: http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/trun_jvm.html.

v Web container Thread pool: minimum 50, maximum 100.v Database Connection Pool for the MashupHub data source: minimum 10,

maximum 50, Statement Cache Size of 100 statements.v Servlet caching: enabled.v Performance monitoring on the server: disabled.

Following are the general performance characteristics for a regular low-end serverwith a typical hardware configuration, which includes a dual-core chip on 2-wayservers and 4 GB memory:v Data feed size: Feeds up to 30 MB output size can be created. The optimal size

for feeds is up to the 2 MB range for the feed output. The input size for feedsfor a 2 MB output feed depend on the nature of the input feed. For example, a500 KB Excel input file can generate a 2 MB output XML feed.


http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/trun_jvm.html



v Complexity and the number of operations in typical data mashups: There is nolimit for the number of operators. The optimal number of operators is less thaneight operators and the data within the flow is 100 - 1,000 rows.

v For large feeds that generate an output feed of 10 MB or more, the server will beconstrained by the amount of memory that it can use. If multiple users executelarge feeds at the same time, it can exhaust the Java heap memory. Increase themaximum heap setting to 1400 MB.

v For feeds that use back end data sources, for example, relational databases,enable feed caching to improve performance. Feed caching will show asignificant improvement for relational, Web services, and other feeds that requireaccess to a back end server. For the feeds that are generated from CSV and Excelfiles, it is not required to specify feed caching.

v Data mashup Source operator: If caching is turned on, performance will improvethe second and subsequent times that the URL for the Source operator isaccessed.

v Preview and Load in the Data mashup builder might perform slowly for inputXML feeds greater than 2 MB. When you specify large feeds that have severalmegabytes or more of data in the Source operator, it might take several minutesfor the Source operator to finish loading. The time depends on the volume ofdata and the current server load.

v Active Content Filtering (ACF): Enabling ACF for a feed increases CPU usageand increases the response time for reads of the feed. Add ACF only to a feedwhose data source has potentially malicious active content, such as injectedJavaScript code.

v Feed Policy Processing: Enabling feed policy processing decreases throughput.Depending on the client application workload when you enable feed policyprocessing, MashupHub might need additional resources:– Increase the web container thread pool size on the server to at least twice the

number of concurrent clients expected in typical client application workloads.Choose the web container thread pool, and then change the maximum size.See the following Web site for information about changing the thread poolsettings:http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/uejb_rthrd.html

– Increase the database connection pool size for the MashupHub data sourceto twice the number of concurrent clients expected in typical clientapplication workloads. See the following Web site for information aboutincreasing the database connection pool size:http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/udat_conpoolset.html

v Data mashups guidelines. As the complexity of a feed transformation increases,the performance can inversely decrease. Following are the key elements oftransformation complexity:

Number of operatorsTypes of sourcesCachingSize of feeds and the amount of data within feedsTypes of operatorsSize of the resulting feedsMemory


http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/uejb_rthrd.html

http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/uejb_rthrd.html

http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/udat_conpoolset.html

http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/udat_conpoolset.html

– Caching data mashup feeds can impact the capacity and server response.Working from cached feeds eliminates the computation and I/O costsassociated with a mashup creation, similar to that of a database cache. Enablecaching whenever it is appropriate.For example, consider a dashboard mashup application for the total sales bymonth. It is probably sufficient to work from orders received an hour or aday earlier and caching can be enabled. In contrast, if the mashup applicationsupports a service representative scenario, then waiting until the cache expiresbefore seeing the most recent order is probably not acceptable.As the number of feeds in a data mashup increases without caching, feedperformance decreases. Enable data mashup source caching as appropriate.Some sources can be fast such as an XML file or a spread sheet, whereasother feeds sources can be expensive to retrieve.

– Sources from external sites can affect performance because of network andsite availability. To offset the network cost, enable caching whenever possibleat the feed, data mashup, or mashup application level.

– Feed size: MashupHub does not perform as well as for large feeds as it doesfor small feeds. To reduce feed size, filter the feed source, for example, for anexternal feed or data from a relational data source. For a relational source,filter the data using an SQL query. When creating a mashup, reduce theamount of data by adding a Filter operator after the Source operator.

– Some operators affect performance:- The For Each operator affects performance because of the network cost

associated with this operator. The For Each operator takes values from eachentry in a feed and fetches data from another feed based on a variable inthe second feed. The variable in the second feed is substituted with thevalues from the first feed. For each entry in the first input, there is acorresponding URL for the second input from which data is fetched. Thenetwork cost of fetching data can increase and slow the mashup. Forexample, if the first feed has 500 entries, for each of those entries thesecond source is fetched 500 times. If the cost of retrieving from an externalsite is 1 second, retrieving time is 500 seconds, which is over 8 minutes.

- The Merge operator affects performance because of the complexity of themerge operation.

- The Sort operator is an expensive operator in terms of the CPU usage.v Eliminate any unnecessary data in the output feed and generate the mashup

tailored to your specific needs. The serialization cost depends on the size of theoutput feed and also the network cost of transferring of data from the server tothe client.

v Data mashups are more CPU-intensive than base feeds.

Understanding general performance characteristics of the catalog, feed generators, and data mashup builder 19

Addendum: Performance tuning for Mashup Center 2.0.0.2

This section provides additional tuning information for the 2.0.0.2 release ofMashup Center. If you are working with Mashup Center 2.0.0.2, be sure to read the2.0 performance tuning information that precedes this addendum. This addendumonly points out the changes in performance tuning for Mashup Center 2.0.0.2. Noother changes since Mashup Center 2.0 were recorded.

Understanding the benchmark environment

The deployment scenario for Mashup Center that was used by the IBMperformance team is similar to the Mashup Center 2.0 deployment scenario. Thescenario consists of the following five separate servers connected by 1000 MbpsEthernet:v WebSphere Edge Components Version 6.1 to act as the reverse proxy serverv WebSphere Edge Components Version 6.1 to act as the reverse proxy serverv A Mashup Center server deployed on AIX 6.1, which consisted of the following

components:– IBM HTTP Server 6.1– WebSphere Application Server instance of Lotus Mashups– WebSphere Application Server instance of MashupHub

v Oracle database server for Lotus Mashupsv Oracle database server for MashupHubv IBM Tivoli Directory Server

These are the key differences between the Mashup Center 2.0 and 2.0.0.2deployment scenarios:v The Mashup Center server is deployed on AIX6.1 instead of Windows 2003.v The database server is Oracle instead of IBM DB2.

Here is a graphical image of the Mashup Center 2.0.0.2 deployment topology:


Tuning the AJAX proxy

The implementation of the AJAX Proxy has changed from Mashup Center 2.0 to2.0.0.2. The 2.0.0.2 implementation has no HTTP connection pool at the applicationlevel. Compared to 2.0, the overhead of the HTTP connection opening and closingis larger. As a result, in order to improve performance, we recommend that you dothe following steps to enable the system level HTTP connection pool and set anappropriate pool size. This tuning works applies to all operating systems.1. Log into the WebSphere Application Server administrative console.2. Click Servers → Server Types → WebSphere application servers → Java Virtual

Machine.3. In the Application servers section in the main pane, click server1.4. In the Server Infrastructure section, click Java and Process Management →

Process definition.5. Under Additional Properties, click Java Virtual Machine.6. In the Generic JVM arguments field, add the following line:

-Dhttp.maxConnections=200 -Dsun.net.http.errorstream.enableBuffering=true

See the following image for an example:


Tuning the data source (AIX 6.1)

On an AIX platform, the large reap time of the data source may lead to a spike inCPU usage under a heavy workload. To solve this usage issue, do the followingsteps to set the preferred values for reap time and aged timeout:1. Log into the WebSphere Application Server administrative console.2. Click Resources → JDBC → Data sources.3. In the Data sources section in the main pane, click mashupDS.4. In the Additional Properties section, click Connection pool properties.5. In the Reap time field, set the value to 0.6. In the Aged timeout field, set the value to 0.

Now the fields should look like this:

Tuning JVM initial and maximum heap size (AIX 6.1)

The value of the JVM heap size is directly related to the amount of physicalmemory on the system. Never set the JVM heap size larger than the physicalmemory on the system. When setting the JVM heap size, be sure to follow therecommendations for Mashup Center 2.0. In our performance tuning, we took thefollowing steps to set the value of the JVM heap size to 1400MB:1. Log into the WebSphere Application Server administrative console.2. Click Servers → Server Types → WebSphere application servers → Java Virtual


Process definition.5. Under Additional Properties, click Java Virtual Machine.6. In the Maximum heap size field, set the value to 1400.

Tuning JVM heap for large pages

Large pages can reduce the CPU overhead needed to keep track of heap size. Withthis setting, we have seen 5% throughput improvement in our measurements.

Although this setting improves performance on Windows, we did not set it for ourmeasurements because Mashup Center does not start reliably when –Xlp is set and

Addendum: Performance tuning for Mashup Center 2.0.0.2 23

sometimes requires a system reboot before starting or restarting a JVM. However, ifyou want to set the JVM heap size, do the following steps on Windows platforms:1. Log into the WebSphere Application Server administrative console.2. Click Servers → Server Types → WebSphere application servers → Java Virtual


Process definition.5. Under Additional Properties, click Java Virtual Machine.6. In the Generic JVM arguments field, add the following line:

-Xlp

Tuning JVM large pages on AIX

To use JVM large pages on AIX, do the following steps to add support for largepages:1. Use the following values to allocate 4GB of RAM as large pages (16MB). We

chose this amount based on the amount of physical memory in our systems,which was 8GB. You may need to adjust these values for systems with differentamounts of physical memory.vmo -r -o lgpg_regions=256 -o lgpg_size=16777216bosboot -ad /dev/ipldevicereboot -qvmo -p -o v_pinshm=1chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE $USER

2. Do the following steps to add the JVM heap size:a. Log into the WebSphere Application Server administrative console.b. Click Servers → Server Types → WebSphere application servers → Java

Virtual Machine.c. In the Application servers section in the main pane, click server1.d. In the Server Infrastructure section, click Java and Process Management →

Process definition.e. Under Additional Properties, click Java Virtual Machine.f. In the Generic JVM arguments field, add the following line:

-Xlp

3. Still in the administrative console, do the following steps:a. Click Servers → Server Types → WebSphere application servers → Java

Virtual Machine.b. In the Application servers section in the main pane, click server1.c. In the Server Infrastructure section, click Java and Process Management →

Process definition.d. Under Additional Properties, click Environment Entries.e. Click New.f. In the General Properties section, in the Name field, type EXTSHM. In the

Value field, type OFF. Note that when the value is set to ON, it prevents theuse of large pages.

g. Click OK. Now you environment entries page should show your new entry,for example:


4. Restart the WebSphere Application Server server. To verify if large pages arebeing used, run the following AIX command: vmstat -l 1 5. Now check thealp column, which is the active large page that is currently being used. Thevalue should not be 0 if large pages are being used.

You can also use large pages for memory areas other than the Java heap. Theseother memory areas include the program text (machine instructions), data area (allother data outside of the Java heap), and stack. For the AIX operating system, onemechanism for using large pages for other data areas is by using environmentvariables, for example:export LDR_CNTRL="TEXTPSIZE=64K@DATAPSIZE=64K@STACKPSIZE=64K"

Running Java after setting the environment variable, as in the example above,causes 64 KB pages to be used for all of the text, data, and stack areas. Moredetails about changing the page sizes used by programs are available in the AIXdocumentation.

Tuning AIX networking

In our measurement environment, we changed the following network tuningparameters on all the AIX systems. For AIX, use the no command to set thefollowing TCP layer parameters. These take effect immediately to improve thenetwork layer performance in high volume environments.v /usr/sbin/no -o tcp_sendspace=65536

v /usr/sbin/no -o tcp_recvspace=65536

v /usr/sbin/no -o udp_sendspace=65536

v /usr/sbin/no -o udp_recvspace=65536

v /usr/sbin/no -o somaxconn=10000

v /usr/sbin/no -o tcp_nodelayack=1

To make these changes permanent across reboots, add these statements to the/etc/rc.net file.

Addendum: Performance tuning for Mashup Center 2.0.0.2 25

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user's responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not grant youany license to these patents. You can send license inquiries, in writing, to: IBMDirector of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785U.S.A.

For license inquiries regarding double-byte (DBCS) information, contact the IBMIntellectual Property Department in your country or send inquiries, in writing, to:Intellectual Property Licensing, Legal and Intellectual Property Law, IBM Japan,Ltd., 3-2-12, Roppongi, Minato-ku, Tokyo 106-8711 Japan

The following paragraph does not apply to the United Kingdom or any othercountry where such provisions are inconsistent with local law: INTERNATIONALBUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFNON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULARPURPOSE. Some states do not allow disclaimer of express or implied warranties incertain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM Web sites are provided forconvenience only and do not in any manner serve as an endorsement of those Websites. The materials at those Web sites are not part of the materials for this IBMproduct and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way itbelieves appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact: IBM Corporation, 5Technology Park Drive, Westford Technology Park, Westford, MA 01886


Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

Any performance data contained herein was determined in a controlledenvironment. Therefore, the results obtained in other operating environments mayvary significantly. Some measurements may have been made on development-levelsystems and there is no guarantee that these measurements will be the same ongenerally available systems. Furthermore, some measurements may have beenestimated through extrapolation. Actual results may vary. Users of this documentshould verify the applicable data for their specific environment.

Information concerning non-IBM products was obtained from the suppliers ofthose products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy ofperformance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to thesuppliers of those products.

All statements regarding IBM's future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subjectto change without notice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject tochange before the products described become available.

This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to the names and addresses used by an actual businessenterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. The sampleprograms are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work, mustinclude a copyright notice as follows:


© (your company name) (year). Portions of this code are derived from IBM Corp.Sample Programs. © Copyright IBM Corp. _enter the year or years_. All rightsreserved.

If you are viewing this information softcopy, the photographs and colorillustrations may not appear.

TrademarksThe following terms are trademarks of International Business MachinesCorporation in the United States, other countries, or both:

AIXAIX 5L

Adobe, Acrobat, Portable Document Format (PDF), PostScript, and all Adobe-basedtrademarks are either registered trademarks or trademarks of Adobe SystemsIncorporated in the United States, other countries, or both.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo,Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks orregistered trademarks of Intel Corporation or its subsidiaries in the United Statesand other countries.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks ofMicrosoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in theUnited States, other countries, or both.

Linux is a registered trademark of Linus Torvalds in the United States, othercountries, or both.

UNIX is a registered trademark of The Open Group in the United States and othercountries.

Other company, product, or service names may be trademarks or service marks ofothers.

Notices 29

��

Printed in USA

ibm mashup center: v2public.dhe.ibm.com/software/dw/lotus/mashups/perf/mash_tuning.pdf · when...

Documents