ia6.5 performance tuning

TAKING ADVANTAGE OF THE EMC CAPTIVA ARCHITECTURE

Applying Best Practices to Optimize

1© Copyright 2011 EMC Corporation. All rights reserved.

Applying Best Practices to Optimize Performance

Christopher Lund

EMC

Agenda

• How InputAccel works

• InputAccel 6.5 Benchmark Results

• Tuning– InputAccel Server

– Batches

– InputAccel Database


– InputAccel Database

– Client Modules

– Capture Workflow

• Diagnosing Performance Issues

The EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and Tuning Guide – which is available on PowerLink – provided much of the data for this presentation.

A multi-machine capture application server

• Server is the data tier (memory mapped)

• Server manages task queues

• Server is multi-threaded…

• VBA execution is single-threaded

How InputAccel Works

InputAccel System

Export Modules

Export Modules

Processing Modules

Processing Modules

Capture Modules

Capture Modules

Capture Modules

Export Modules

Processing Modules


•

• DB writes are queued, but single-threaded

• Server uses asynchronous I/O

• Most work done from a thread pool

• Clients are the executing tier (where scaling comes from)

InputAccelServers

WIP&Reports Tables

How InputAccel WorksAn execution pipeline

• Task queues are not FIFO

• Tasks are scheduled based on Priority, then creation date

• Recovery is through reprocessing

Execution Pipeline

B2 B3 B4

B3 B4

B4t1

t2

t3

m1 m2 m3 m4


reprocessing

• Not a repository

• Assume short duration tasks

• Work is pushed, no polling

• Tasks may be prefetched

B1

B1

B2

B2

B3

B3

B4

A4

B1 B2A4

B1A3 A4A2

A3 A4A2A1

B2 B3 B4

A3A2A1

t3

t5

t4

t7

t6

t8

t9

How InputAccel WorksLike a Petri net

• A ProcessFlow defines the steps and trigger levels.

• Implicit fire when data is available

• There is no predefined execution

Petri Net

AA


• There is no predefined execution order

• There is no end state – IADonetriggered implies completed

D

CB CB

InputAccel 6.5 Benchmark Results• One 8-core IA Server, over 300 Client Modules

• In-house, ideal conditions, your mileage will vary

• Performance similar to IA 6.0 SP1

Performance Level Overall Task Processing Rate(tasks/hour)

Processing Rate/CPU Core(tasks/hour)

Avg. CPU Utilization/CPU Core


(tasks/hour) (tasks/hour) PU Core

50 active batches w/reporting disabled

2,672,007 324,001 67%

1000 active batches w/reporting disabled

1,990,892 248,862 53%

1000 active batches w/reporting enabled

1,384,910 173,114 32%

VMware ESX 4.x degrades throughput by approx 27%

Tuning – InputAccel Server…

• It is a server application. It will use all available resources

• Recommendation: use Windows 2008 R2 for best performance

• CPU– ias.exe is multi-threaded – Recommendation: use at least a 4-core CPU for optimum throughput, 8+ is

better

• RAM


• RAM – Recommendation: 4-8 GB RAM, no less than 4 GB– ias.exe is a 32-bit app, so 4 GB address space max– BatchMaxAddressSpaceK controls how much RAM IAS uses

• 1.5 GB is the default• Set to 2 to 2.5 GB on 32-bit Windows with /3G option• Set to 3 to 3.5 GB on 64-bit Windows

– Only as many batches as will fit in BatchMaxAddressSpaceK are kept in RAM, when there are more, swapping occurs• Have your working set of batches small enough to all fit in RAM• Delete batches when you are done with them

…Tuning – InputAccel Server

• Disk

– Used heavily for batches and processes

– RAID 1+0 is best – RAID 5 usually is not fast enough and is not recommended• Recommendation: use a caching controller with Read Ahead and Write Back

– SAN is OK, NAS not recommended

– Turn off anti-virus scanning on IAS folder

– IAS folder should be on a dedicated disk drive or array so it is not shared by other programs or the Windows swap file


programs or the Windows swap file

– #2 cause of slow performance: a slow hard drive

• Network

– InputAccel has a “chatty” protocol between client modules and the InputAccel Server

– For best performance, client modules, the InputAccel Server, Administration Console, and DB should be on the same sub-net

– WANs usually have low bandwidth with high latency, which may make it unsuitable • Connecting client modules to InputAccel Server via a WAN is doable with adequate performance – depends on

the client modules, # batches on IA server, and IPP

• InputAccel Server and IADB should not be connected by a WAN

Tuning – Batches…

• Batch size (IAB file)– 5 – 20 MB is ideal

– < 100 MB is OK

– > 100 MB not recommended, but is allowed

– 10 – 100 pages per batch recommended


– 1000 pages per batch degrades throughput by 10% or more

– < 10 pages per batch leads to too many small batches and too much batch swapping

…Tuning – Batches

• Number of Batches– Throughput is best when the working set of batches fit in RAM

• Typically 50 – 500 fit depending on batch size and BatchMaxAddressSpaceK

– When there are more batches than fit in RAM, IA Server swaps batches in/out as needed• Swapping decreases IA Server throughput

– Up to 9,000 idle batches is possible with adequate performance


– Up to 9,000 idle batches is possible with adequate performance• Idle means that all tasks within the batch have been processed and finished

– IA Server startup is slower with 1000’s of batches because IAS must load all batches into RAM to extract data• After startup they are swapped to disk and do not consume too many resources

– #1 cause of slow performance: too many active batches causing excessive swapping

Tuning – InputAccel DatabaseSystem Requirements

• InputAccel Server requires a database to run

• Only MSSQL is supported, versions 2005 and 2008– Express supported only for low volume and no IA Reporting

– Standard or Enterprise recommended for medium to high volume


• IADB stores:– IA configuration data

– Reporting data on completed batches/tasks

– Work In Progress (WIP) status

– Web services data

Tuning – InputAccel DatabaseSystem Requirements

• Recommendation: 64-bit MSSQL server for best performance– 2-4 CPU cores

– 4-8 GB RAM

– RAID 1+0 with read/write caching controller w/ fast disks (15k RPM)

• InputAccel Server requires fast, uninterrupted access to the DB


• InputAccel Server requires fast, uninterrupted access to the DB– If the DB goes offline, IA Server pauses

• Recommendation: put the DB and InputAccel Server on the same low-latency, high-bandwidth subnet

• Reporting decreases the InputAccel Server throughput by about 10-30%, although it can be more

Tuning – InputAccel DatabaseData Volume and Rates

• IADB File Size and Growth Rates

– Configuration data - typically < 100 MB

– WIP data - 1 MB × # batches• WIP is transient and grows and shrinks as needed

– Error/Warning Log data – typically negligible and can be purged as needed

– Reporting data


– Reporting data • All reporting log rules off – 0 MB• Some or all reporting log rules are on

� Typically data grows 100 MB – 3 GB per hour� Shrinks only when purged� But…

» Growth rate depends on page volume and which log rules are on» Overall size depends on # days of data retained

– Audit Log data

• All audit log rules off – 0 MB

• All audit logs rules on – 2× the growth rate of Reporting data

Tuning – InputAccel DatabaseData Volume and Rates

• InputAccel Database transaction rates

Log Rules EnabledLog Rules EnabledLog Rules EnabledLog Rules Enabled Estimated Transaction RateEstimated Transaction RateEstimated Transaction RateEstimated Transaction Rate

None IAS Tasks / Hour × 0.075

Reports only (no Audit) IAS Tasks / Hour × 2.5


Reports only (no Audit) IAS Tasks / Hour × 2.5

Audit only (no Reports) IAS Tasks / Hour × 5.0

Reports + Audit IAS Tasks / Hour × 7.5

Tuning – InputAccel DatabaseImproving Performance

• Defragment and rebuild indexes– up_ReorganizeIndex – defragments all indexes

– up_RebuildIndex – rebuilds all indexes

• Purge reporting and auditing data– Reporting and Auditing tables grow continuously


– Reporting and Auditing tables grow continuously

– You must schedule purges via the Admin Console

• Recommendation: generate reports during non-peak hours– Generating Reports runs complex queries that place a heavy load on

IADB and MSSQL

• Store MSSQL transaction logs and data files on separate hard drives/controllers

Tuning – Client Modules

• Each client module has its own unique tuning characteristics– Example: val2xml slows as more IA Values are exported and is slower when triggered at

level 1 than level 7

– See “EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and Tuning Guide” for details

• Parameters on the client machines can be modified to optimize performance– Stored in settings.ini located in %ALLUSERSPROFILE%\EMC\InputAccel


– Stored in settings.ini located in %ALLUSERSPROFILE%\EMC\InputAccel

– PrefetchDefault (default = 2) – the number of additional tasks the InputAccel Server sends to each client module

– FileCacheSize (default = 8)

– CacheSize (default = 1,048,576)

– CacheCount (default = 200,000), previously 20,000 – the number of files and IA Values the client module caches

– IAClientDebug (default = 0) – set to 1 to capture the debug log iaclient.log in %ALLUSERSPROFILE%\EMC\InputAccel (previously was created in C:\)

Tuning – Client Modules

• CPU intensive client modules like NuanceOCR & ImageEnhancement– Require fast CPUs

– Run one instance for each CPU core.

• e.g. 4 CPU cores, run 4 instances of NuanceOCR

• Non-CPU intensive client modules like val2xml & Documentum


• Non-CPU intensive client modules like val2xml & Documentum Export– Performance is limited by other resources (disk, network)

– Run at least one instance for each CPU core and possibly more

Tuning – Client ModulesUsing InputAccel over a WAN…

• InputAccel Server, Database and Administration Console– Require high-speed, low-latency connections

– Must be on the same LAN as each other

• Unattended client modules – Often require high-speed, low-latency connections for best throughput


– Generally do not need to be remote from the InputAccel Server

– Should be on the same LAN as the InputAccel Server

• Attended client modules– Performance varies by environment and module

– Detailed guidance is in the EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and Tuning Guide

– Recommendation:Recommendation:Recommendation:Recommendation: do benchmark testing to ensure adequate performance

Tuning – Client Modules…Using InputAccel over a WAN

• Attended client module details– ScanPlus

• Not recommended, but may perform acceptably with scanner hardware compression or small scanned images

• Works best when bandwidth is ≥50 Mbps and round trip latency ≤25 ms

– IndexPlus• Generally performs well over a WAN (except for thumbnail display) • Displaying the batch list takes about 25% longer


• Displaying the batch list takes about 25% longer• Works best when bandwidth is ≥1.5 Mbps and round-trip latency ≤50 ms

– Dispatcher Classification Edit and Dispatcher Validation• Should not be used over a WAN

• Recommendation:Recommendation:Recommendation:Recommendation: consider using VMware View or Citrix for remote operators– Module executes on the LAN, screen display is over the WAN– Supports remote scanning

• ScanPlus is on the LAN, the scanner is on the remote machine• Use scanner hardware compression for best results

– Maximizes InputAccel Server-to-client module throughput– EMC OnDemand uses VMware View

Tuning – Capture WorkflowTrigger Levels

• Triggering at level 0 or 1 – Usually gives better throughput than level 7– The tasks within a batch can be distributed among many client modules– Which results in faster end-to-end processing of any single batch

• Triggering at level 7– Is less work for InputAccel Server as it has fewer tasks to manage


– Is less work for InputAccel Server as it has fewer tasks to manage– Under some circumstances may provide better overall throughput at the

expense that any single batch may take more time to process

• Unsupported: accessing external resources in IPP VBA code– VBA execution with InputAccel Server is single-threaded– The external resource may be slow or not present– If InputAccel Server needs to wait for the resource, all other tasks block– Put more complicated custom code logic on the client through the .NET Code

module or client scripting

Diagnosing Performance Issues

• IA Server Performance Counter– Batches loaded and loads/second– Connections– Disk bytes written & read/second– VBA calls/second & queue length– Processing Message Count– Network bytes written & read/second– Packets send & received/second


– Packets send & received/second– Pending I/O (~ # of asynchronous sends in progress)– Event (db) queue length

• Data Access Layer Performance Counters– Data Requests/second– % Load Factor– Avg. Execution Time Millisec– Current connection count

EMC OnDemandCaptiva Instant Cloud Implementation


IIG Applications on EMC OnDemand

ENTERPRISECAPTURE

CONTENTMANAGEMENT

CUSTOMERCOMMUNICATIONS

CASEMANAGEMENT

DocumentumDocumentumxCPxCP

CaptivaCaptiva DocumentumDocumentumECMECM

DocumentDocumentSciencesSciences

INFORMATION

GOVERNANCE

SourceOneSourceOne


NetworksNetworks

StorageStorage

Virtualization and SecurityVirtualization and Security

Cloud ManagementCloud Management

Summary/Key Takeaways

• Focus on system throughput, not per-task time

• Use high-speed hardware – disk drives and networks

• Minimize disk I/O where possible– Keep active batches in memory

– Avoid excessive reporting


– Avoid excessive reporting

• Parallelize – multi-core CPUs and task granularity

• Use performance counters to find bottlenecks

Q&A


Chris Lund

[email protected]

(858) 320-1215

Learn More About EMC Captiva


Go to: www.EMC.com/Captiva

Captiva @ Momentum 2011

Thursday

10:00 AM AP Automation: Best Practices for Capturing and Integrating Paper Invoices into your Accounts Payable Processes

Galileo 705


GET SOCIAL


Come to the Momentum Lounge todayCome to the Momentum Lounge todayCome to the Momentum Lounge todayCome to the Momentum Lounge todayto play, win, learn and more.to play, win, learn and more.to play, win, learn and more.to play, win, learn and more.

DOWNLOADDOWNLOADDOWNLOADDOWNLOAD

Apple: Apple: Apple: Apple: http://bit.ly/MMTM11http://bit.ly/MMTM11http://bit.ly/MMTM11http://bit.ly/MMTM11

Other: Other: Other: Other: http://bit.ly/MMTMemchttp://bit.ly/MMTMemchttp://bit.ly/MMTMemchttp://bit.ly/MMTMemc

THANK YOU


ia6.5 performance tuning

Documents

emc corporation

server application

core ia server

gb ram

task queues server

singlethreaded server

core cpu

ram recommendation