accelerating file check-outs with the starteammpx cache agent randy guck, chief scientist, borland
TRANSCRIPT
Accelerating File Check-outs with the StarTeamMPX Cache Agent
Randy Guck, Chief Scientist, Borland
Overview
StarTeam architecture overview Configuring MPX Cache Agents Using bundled Cache Agent-aware clients Checking-out files via the StarTeam SDK
The basic file check-out application The new CheckOutManager class
Cache Agent performance comparison New Cache Agent options
StarTeamClient
SDK
StarTeamClient
SDK
StarTeam Basic Architecture
Vault
StarTeamServer
Data-base
Command API
StarTeamClient
SDK
StarTeamConfiguration
workspace
workspace
workspace
StarTeam Architecture with Basic MPX
StarTeamClient
SDK
StarTeamClient
SDK
Vault
StarTeamServer
Data-base
StarTeamClient
SDK
MessageBroker
publish/subscribeupdate events
StarTeam with MPX Cache Agents
StarTeamClient
SDK
StarTeamClient
SDK
Vault
StarTeamServer
Data-base
StarTeamClient
SDK
RootMessageBroker
RootCache AgentRemote
Cache Agent
RemoteMessageBroker
file check-out
LocalCache
forwardingand catch-up
File Transmitter
Event Transmitter
MPX/Cache Agent Configuration Order
Vault
StarTeamServer
Data-base
RootMessageBroker
RootCache Agent
RemoteCache Agent
RemoteMessageBroker
StarTeamClient
SDK
1) MPXEventTransmitter.xml
2) MPXFileTransmitter.xml5) STMessageBroker64.ini
7) Enable MPX andCache Agent
3) STMessageBroker64.iniand optionallySTMulticastService64.ini
4) RootCAConfig.xml(example)
6) RemoteCAConfig.xml(example)
Bundled Cache Agent-aware Clients
Using Cache Agent check-out wth the StarTeam Cross-Platform Client
Tools -> Personal Options
File -> Check Out…
After check out
Bundled Cache Agent-aware Clients (cont.)
StarTeam command-line (stcmd) utility Add “-useMPXCacheAgent <address>” to “co” command
Use “autolocate” to use the network-nearest Cache Agent Use “host:port” for <address> to use a specific Cache Agent
Add “-mpxCacheAgentThreads <#>” to set fetch threads 3 is the default
Note: Cache Agent mutually exclusive with keyword expansion Example:
stcmd co -p "Administrator:Administrator@localhost:49201/StarDraw/Beta Release"-is -filter OM -useMPXCacheAgent autolocate
Bundled Cache Agent-aware Clients (cont.)
StarTeam bulk check-out (BCO) utility Similar to “stcmd co”, but optimized for large check-outs Options:
-useca <address>: “autolocate” or host:port -threads <#>: to set fetch threads (3 is default) -t: to get post-check out statistics -verbose: to get a warm fuzzy feeling about what it’s doing
Limitation: can’t lock/unlock files during check-out Note: Cache Agent mutually exclusive with keyword expansion Example:
bco -autologon -p "localhost:49201/StarDraw/Beta Release " -is -filter OM –t-useca "192.168.0.4:5201"
Quick Refresher on SDK Flavors
StarTeam SDK supports multiple language “flavors” Pure Java applications: starteam80.jar Pure .Net applications: Borland.StarTeam.dll Cross-compilable Java/J# applications: starteam80.jar for Java;
Borland.StarTeam.Core.dll for .Net Hybrid core/edge applications: Java/J# mixed with pure .Net:
starteam80.jar for Java; Borland.StarTeam.dll and Borland.StarTeam.Core.dll for .Net
COM application languages: StarTeamSDK80.dll VB, C++, scripting languages, etc.
Checking-out files via the SDK
The basic check-out application Most SDK applications work in the context of a view
Open a Server and log-on Find the Project you want Select the View you want
Change ViewConfiguration if needed: tips, label, date, or promotion state
Shortcut: StarTeamFinder
Find the base Folder of interest The old way to check-out
Examine each File in a Folder If you want a file, call File.checkOut() or File.checkOutXxx() Recurse to subfolders
Old Way to Check-out Files in Java
// Find view as a URL.String viewURL = "Administrator:Administrator@localhost:49201/StarDraw/Beta Release";View view = StarTeamFinder.openView(viewURL);
// Fetch all the file properties we're going to need in one request.Server server = view.getServer();String[] filePropNames = new String[] { /* list of property names here */ };Folder folder = view.getRootFolder();folder.populateNow(server.getTypeNames().FILE, filePropNames, -1);
// Check-out all files in this folder whose status is "missing" or "out of date".Item[] itemArray = folder.getItems(server.getTypeNames().FILE);for (int i = 0; i < itemArray.length; i++) { File fileItem = (File)itemArray[i]; // StarTeam File, not Java.io int fileStatus = -1; try {fileStatus = fileItem.getSmartStatus();} catch (IOException ex) {} if (fileStatus == Status.MISSING || fileStatus == Status.OUTOFDATE) { try { fileItem.checkout(Item.LockType.UNCHANGED, false, true, true); } catch (IOException ex) { // Couldn't check-out file } }}
// Recurse to subfolders...
Old Way to Check-out Files in C#
// Find view as a URL.string viewURL = "Administrator:Administrator@localhost:49201/StarDraw/Web1";View view = StarTeamFinder.OpenView(viewURL);
// Fetch all the file properties we're going to need in one request.Server server = view.getServer();string[] filePropNames = new string[] { /* list of property names here */ };Folder folder = view.RootFolder;folder.PopulateNow(server.TypeNames.FILE, filePropNames, -1);
// Check-out all files in this folder whose status is "missing" or "out of date".ItemCollection folderItems = folder.GetItems(server.TypeNames.FILE);foreach (Item item in folderItems){ // Get this item as a file and check its status File fileItem = (File)item; // StarTeam File, not Java.io int fileStatus = fileItem.SmartStatus; if (fileStatus == Status.MISSING || fileStatus == Status.OUTOFDATE) { // Check out this file try { // Checkout could throw an IOException. fileItem.Checkout(Item.LockType.UNCHANGED, false, true, true); } catch (IOException ex) { /* Error checking-out this file. */ } }}
// Recurse to subfolders...
The New Way to Check-out Files
New CheckoutManager class “The mother of all check-out classes” Checks-out a single file or many files in one call Can check-out from an MPX Cache Agent New CheckoutOptions class for bundling options together New CheckoutListener (Java) or CheckoutEventSource (.Net)
class for monitoring progress and setting per-file options New CheckoutProgress class provides statistics on snapshot or
final basis New ContentFilter for in-stream manipulation
“SDK New Features” session covers each in more depth
Steps for Cache Agent Check-out
1. Create a CheckoutOptions object Create from View or StarTeamClientOptions to initialize with
workstation settings Set desired options; some non-default examples:
Set ViewConfiguration by date, by label, by promotion state Insert a ContentFilter into the check-out stream Override project setting for keyword expansion and file encoding Lock files during check-out (exclusive or non-exclusive) Mark non-locked files as “read only” Set workfile timestamp to check-out time instead of revision
modification timestamp Enable “delta check-out” for out-of-date text files
Steps for Cache Agent Check-out (cont.)
2. Create a CheckoutManager object Initialize with CheckoutOptions or just from View Set Cache Agent options
setMPXCacheAgentEnabled(boolean bEnabled) setMPXCacheAgentAddress(String address)
null or not called means “auto locate” setMPXCacheAgentPort(int port)
default is 5201 setMPXCacheAgentThreadCount(int nThreads)
default is 3
Call addCheckoutListener() to add a CheckoutListener (Java) Call NewCheckoutEventSource() to create a
CheckoutEventSource for .Net
Steps for Cache Agent Check-out (cont.)
3. Call one CheckoutManager.checkOut or checkOutTo void checkout(File file): single file void checkout(Folder folder): all files in a folder void checkout(Folder folder, int depth): all files in a folder and child
folders to the specified depth (-1 means “all children”) void checkout(Item[] files): all files in Item array void checkout(ItemList files): all files in an ItemList void checkout(Items files): all files specified by Items interface void checkoutTo(File file, java.io.File workingFile): single file to an
alternate location void checkoutTo(File file, java.io.OutputStream stream): single file to
a stream (System.IO.Stream in .Net) Note: all check-outs are “force” check-outs
Steps for Cache Agent Check-out (cont.)
4. While the check-out is in progress… If a CheckoutListener object (or CheckoutEventSource
delegates) are registered onStartFile() and onNotifyProgress() get called once for each file
onStartFile() can modify each file’s CheckoutOptions or its workfile
onNotifyProgress() can see each file’s result
CheckoutManager.setCanceled() can be called to cancel the check-out
CheckoutManager.getProgress() can be called to get a snapshot of in-progress or final statistics
Be careful of Cache Agent check-out with threads > 1 ContentFilter and listeners must be thread safe!
Cache Agent Check-out: Simple Example
// C#void SimpleCacheAgentCheckout(View view){ CheckoutManager checkoutManager = new CheckoutManager(view); checkoutManager.MPXCacheAgentEnabled = true; checkoutManager.Checkout(view.RootFolder, -1); // all files in view}
// Javavoid SimpleCacheAgentCheckout(View view){ CheckoutManager checkoutManager = new CheckoutManager(view); checkoutManager.setMPXCacheAgentEnabled(true); checkoutManager.checkout(view.getRootFolder(), -1); // all files in view}
Cache Agent Check-out: More Sophisticated
// Initialize CheckoutOptions from StarTeamClientOptionsStarTeamClientOptions defaultClientOptions = null;CheckoutOptions checkoutOptions = null;try { // StarTeamClientOptions.getDefault() could throw an exception. defaultClientOptions = StarTeamClientOptions.getDefault(); checkoutOptions = new CheckoutOptions(defaultClientOptions);} catch (IOException ex) { checkoutOptions = new CheckoutOptions(view);}
// Create a CheckoutManager and enable the use of an auto-located Cache Agent.CheckoutManager checkoutManager = new CheckoutManager(view, checkoutOptions);checkoutManager.setMPXCacheAgentEnabled(true);
// Add ourself as a CheckoutListenter and check-out all files from root.checkoutManager.addCheckoutListener(this);checkoutManager.checkout(view.getRootFolder(), -1);
// Get some final statisticsCheckoutProgress finalStats = checkoutManager.getProgress();System.out.println("Total files checked-out: " + finalStats.getTotalFilesCheckedOut());System.out.println("Total bytes checked-out: " + finalStats.getTotalBytesCheckedOut());System.out.println("Total check-out time: " + ((float)finalStats.getTotalCommandTime() / 1000) + " seconds");
Performance Comparison
CPC w/remote CA CPC w/no CA BCO w/remote CA BCO w/no CA
FCO time (secs) Bytes/sec
FCO time (secs) Bytes/sec
FCO time (secs) Bytes/sec
FCO time (secs) Bytes/sec
33.94 3,392,343 103.47 1,112,738 25.36 4,540,472 93.27 1,234,350
29.17 3,946,687 94.68 1,216,071 27.87 4,131,064 103.90 1,108,111
28.19 4,084,170 97.10 1,185,713 28.58 4,028,297 93.70 1,228,804
Average 3,807,733 Average 1,171,507 Average 4,233,277 Average 1,190,421
Test #1: Project Documents• 238 files: Word docs, PPTs, PDFs, etc.• 115,132,745 total bytes; 483,751 average file size• Check-out over cable mode, VPN• Remote Cache Agent network-near over 100Base-T
Performance Comparison (cont.)
Test #2: Source Files• 172 files: source files, IDE workspaces• 857,044 total bytes; 4.983 average file size• Check-out over cable mode, VPN• Remote Cache Agent network-near over 100Base-T
CPC w/remote CA CPC w/no CA BCO w/remote CA BCO w/no CA
FCO time (secs) Bytes/sec
FCO time (secs) Bytes/sec
FCO time (secs) Bytes/sec
FCO time (secs) Bytes/sec
5.67 151,207 13.94 61,481 5.09 168,477 10.32 83,087
4.88 175,732 11.62 73,775 5.01 171,169 10.87 78,874
2.25 380,401 10.51 81,584 4.98 172,201 10.34 82,926
Average 235,780 Average 72,280 Average 170,616 Average 81,629
New Cache Agent Options for 2005 R2
Remote Cache Agent tracking-by-project Allows a remote Cache Agent to cache a subset of each
StarTeam server’s files Enabled by adding <project> options to <ContentSource> group:
<ContentSource> <ServerGUID>bd1352a8-68a3-4ac1-9f28-0604b3d9267a</ServerGUID> <Projects> <Project>FelixTools</Project> <Project>WebFoundation1</Project> <Project>Bank*</Project> <Project>Insurance*West*</Project> </Projects> <UpstreamCache> <AutoLocate/> </UpstreamCache></ContentSource>
New Cache Agent Options for 2005 R2 (cont.)
Multiple Cache Agents on the same machine CacheAgentService.exe makes it easier to register multiple
Cache Agent services on the same machine -name <name>: sets service display name -dependson <service list>: declares service start dependency -log <log file>: sets name of log file (default based on <name>) -verbose: generates secondary debug log file
Unregister supports -name <name> as well Example:
CacheAgentService -register Auto"C:\Program Files\Borland\StarTeamMPX Cache Agent 2005 R2\RemoteCAConfig.xml"-dependson StarTeamMessageBroker6.4 -verbose
New Cache Agent Options for 2005 R2 (cont.)
New Cache Agent Configuration Options New common Cache Agent options
CacheCheckInterval: Frequency (seconds) for checking cache limit InitialRequestThreads: Start-up request threads MaxCatchupSize: Maximum content transfer provided (root Cache
Agent) or requested (remote Cache Agent) for catch-up requests RequestReadTimeout: Connection timeout (seconds)
New remote Cache Agent options CatchupCheckInterval: Catch-up check frequency after a network
outage PrechargeSize: Initial “pre-charge” size for new remote Cache
Agents
