building vsphere perf monitoring tools

Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

Developing Performance Monitoring Tools Using vSphere APIs: Techniques and Best Practices

TEXIBP 1005

Balaji Parimi & Ravi Soundararajan, VMware Inc.

2Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.


This session may contain product features that are curr ently under development.

This session/overview of the new technology represent s no commitment from VMware to deliver these features in a ny generally available product.

Features are subject to change, and must not be inclu ded in contracts, purchase orders, or sales agreements of any k ind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or feat ures discussed or presented have not been determined.

“These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”

Disclaimer



t

Linkedin Search for VMware Partner Network

Follow Us

TwitterSearch for VMware_Partners

Facebook Search for VMware Partner Network

http://blogs.vmware.com/powerofpartnership/

Help us help you! Please fill out the survey at the end of the course.

Join the conversationGet the latest scoopBe a part of the network

Subscribe to RSS

Follow this event LIVE!#VMwarePEX2010



Who is our Target Audience?

System Administrators:• Monitor the performance of their virtual infrastructure

• Understand the bottlenecks in the datacenter to help reconfigure their storage or network topology

• Plan for future growth in the datacenter

VMware ISV Partners:• Collect the relevant/important performance counters and hand them off to a

performance analysis tool

• Create a wizard/tool that helps system administrators troubleshoot their datacenter



Agenda

Use Cases

Building a Performance Monitoring Application

Q & A



Use Cases

Use Cases: What do I look for and why?• CPU• Memory

• Disk

• Network



CPU

ESX is designed to run Virtual Machines

Schedulable entity = “world”• Virtual Machines are composed of worlds (mks, vCPUs)

• Service Console (agents like vpxa, hostd) (Classic ESX)

Proportional-share scheduler for resource management • Limits, Shares, and Reservations

World states (simplified view):• ready = ready-to-run but no physical CPU free

• run = currently active and running

• wait = blocked on I/O



So, How Do I Spot CPU Performance Problems?

One common issue is high CPU ready time

• High ready time � possible contention for CPU resources among VMs

• Many possible reasons

� CPU over commitment (high %rdy + high %used)

� Workload variability

� Limit set on VM

• No fixed threshold, but > 20% for a VCPU �Investigate further



CPU: Useful Metrics

Per-Host

Metric (Client) Metric (esxtop)

Metric (sdk) Description

Usage (%) %USED cpu.usage.average CPU used over the collection interval (%)

Usage (MHz) n/a cpu.usagemhz.average CPU used over the collection interval (MHz)



CPU: Useful Metrics

Per-VM


Metric (SDK) Description

Usage (%) %USED cpu.usage.average CPU used over the collection interval

Used (ms) %USED cpu.used.summation CPU used over the collection interval)*

Ready (ms) %RDY cpu.ready.summation CPU time spent in ready state*

Swap wait time (ms) [ESX4.0 hosts]

%SWPWT cpu.swapwait.summation CPU time spent waiting for host-level swap-in

* Units different between esxtop and vSphere client



Spotting CPU Overcommitment in esxtop

• 2-CPU box, but 3 active VMs (high %used)

• High %rdy + high %used can imply CPU overcommitment



Caveat on Ready Time: Workload Variability

Ready time < used time

Used time

Ready time ~ used time

Some caveats on ready time

� Used time ~ ready time: may signal contention. However, might not be overcommitted due to workload variability

� In this example, we have periods of activity and idle periods: CPU isn’t overcommitted all the time

(screenshot from VI Client)



Further ready time examination

High Ready TimeHigh MLMTD: there is a limit on this VM…

�High ready time not always because of overcommitment�When you see high ready time, double-check if limit is set



Further Ready Time Investigation: No Fixed Threshol d…Why?

ESX 1 ESX 2

vSphereVM

Probe VM vSphereDB

ODBC

Snifftraffic

vSphere communicates with DBProbe VM monitors vSphere-to-DB trafficThe more traffic, the more work done by Probe VM



Why Isn’t There A Fixed Ready Time Threshold?

Ready time jump from 12.5% (idle DB) to 20% (busy DB ) – didn’t notice until responsiveness suffered!



Summary of Possible Reasons for High Ready Time

CPU overcommitment• Possible solution: add more CPUs or VMotion the VM

Workload variability• A bunch of VMs wake up all at once

• Note: system may be mostly idle: not always overcommitted

Limit set on VM• 4x2GHz host, 2 vcpu VM, limit set to 1GHz (VM can consume 1GHz)

• Without limit, max is 2GHz. With limit, max is 1GHz (50% of 2GHz)

• CPU all busy: %USED: 50%; %MLMTD & %RDY = 150% [total is 200%, or 2 CPUs]



Where do we go from here?

Use Cases: What do look for and why• CPU

• Memory

• Disk

• Network



Memory

ESX must balance memory usage for all worlds

• Virtual machines, Service Console, and vmkernel consume memory

• Page sharing to reduce memory footprint of Virtual Machines

• Ballooning to relieve memory pressure in a graceful way

• Host swapping to relieve memory pressure when ballooning insufficient

ESX allows overcommitment of memory

• Sum of configured memory sizes of virtual machines can be greater than physical memory if working sets fit



Ballooning vs. Swapping (1)

Ballooning: Memctl driver grabs pages and gives to ESX� Guest OS choose pages to give to memctl (avoids “hot” pages if possible ): either

free pages or pages to swap

- Unused pages are given directly to memctl

- Pages to be swapped are first written to swap partition within guest OS and then given to memctl

VM1

Swap partition w/inGuest OS

ESX

VM2

memctl

1. Balloon

2. Reclaim

3. Redistribute

F




Swapping: ESX reclaims pages forcibly� Guest doesn’t pick pages…ESX may inadvertently pick “hot” pages (�possible VM

performance implications)

� Pages written to VM swap file

VM1

SwapPartition (w/in guest)

ESX

VM2

VSWP(external to guest)

1. Force Swap2. Reclaim3. Redistribute




Bottom line:

• Ballooning may occur even when no memory pressure just to keep memory proportions under control

• Ballooning is vastly preferably to swapping

� Guest can surrender unused/free pages

- With host swapping, ESX cannot tell which pages are unused or free and may accidentally pick “hot” pages

� Even if balloon driver has to swap to satisfy the balloon request, guest chooses what to swap

- Can avoid swapping “hot” pages within guest



Ok, So Why Do I Care About Memory Usage?

If running VMs consume too much host memory…

• Some VMs do not get enough host memory

• This forces either ballooning or host swapping to satisfy VM demands

• Host swapping or excessive ballooning � reduced VM performance

If I do not size a VM properly (e.g., create Windows VM with 128MB RAM)

• Within the VM, swapping occurs, resulting in disk traffic

• VM may slow down

• But…don’t make memory too big! (High overhead memory)



Important Memory Metrics (Per VM)

One rule of thumb: > 1MB/s swap in or swap out rate may mean memory overcommitment



Swap in rate (ESX4.0 Hosts)

SWR/s mem.swapinRate.average Rate at which mem is swapped in from disk

Swap out rate(ESX4.0 Hosts)

SWW/s mem.swapoutRate.average Rate at which mem is swapped out to disk

Swapped SWCUR mem.swapped.average (level 2 counter)

~swap out – swap in

Swap in (cumulative)

n/a mem.swapin.average Mem swapped in from disk

Swap out (cumulative)

n/a mem.swapout.average Mem swapped out to disk



Important Memory Metrics (Per Host, sum of VMs)

One rule of thumb: > 1MB/s swap in or swap out rate may mean memory overcommitment



Swap in rate (ESX4.0 Hosts)

SWR/s mem.swapinRate.average Rate at which mem is swapped in from disk

Swap out rate(ESX4.0 Hosts)

SWW/s mem.swapoutRate.average Rate at which mem is swapped out to disk

Swap used SWCUR mem.swapused.average (level 2 counter)

~swap out – swap in

Swap in (cumulative)

n/a mem.swapin.average Mem swapped in from disk

Swap out (cumulative)

n/a mem.swapout.average Mem swapped out to disk



Example of Swapping

Increased swap activity may be a sign of over-commitment

Lots of swapping

No swapping



A Stacked Chart (per VM) of Swapping

Lots of swapping

No swapping





• Memory

• Disk

• Network



Disk Performance Problems 101

What should I look for to figure out if disk is an i ssue?

• Am I getting the IOPs I expect?

• Am I getting the bandwidth (read/write) I expect?

• Are the latencies higher than I expect?

• Where is time being spent?

What are some things I can do?

• Make sure devices are configured properly (caches, queue depths)

• Use multiple adapters and multipathing

• Check networking settings (for iSCSI/NAS)



Useful Disk Metrics

Metric (Client)

Metric (Esxtop)


Commands CMDS/s disk.commands.summation commands issued in the sampling interval

Read rate MBREADS/s disk.read.average KB/s read*

Write rate MBWRTN/s disk.write.average KB/s written*

Device latency DAVG/cmd disk.deviceLatency.average Average latency at device

Kernel latency KAVG/cmd disk.kernelLatency.average Average latency in vmkernel

Command latency

GAVG/cmd disk.totalLatency.average Total latency for command




Disk Performance Example: vSphere Client

SAN cache enabled:High Write Throughput

SAN cache disabled:Poor throughput



Another Disk Example: Slow VM Power On

Trying to Power on a VM

• Sometimes, powering on VM would take 5 seconds

• Other times, powering on VM would take 5 minutes!

Where to begin?

• Powering on a VM requires disk activity on host � Check disk metrics for host



Let’s look at the vSphere client…

�Max Disk Latencies range from 100ms to 1100ms…very high! Why?(counter name: disk.maxTotalLatency.latest)



Using Esxtop to Examine Slow VM Power On

Note very large DAVG/cmd and GAVG/cmd

Rule of thumb: GAVG/cmd > 50ms = high latency!

What does this mean?• Latency when command reaches device is high

• Latency as seen by the guest is high

• Low KAVG/cmd: command isn’t queuing in VMkernel

• What’s up?



High Disk Latency: Mystery Solved

Host events: disk has connectivity issues �� high latencies!

Bottom line: monitor disk latencies; issues may not be related to virtualization!



A More Complex Disk Example

ESX

VM1 VM2 VM3 VM4 VM15… NAS

Ethernet

Disk Traffic

VMs communicate with NAS (network traffic)NAS writes to disk (disk traffic)



Combining Metrics: A More Complex Disk Example

Setup:

• Group of Virtual Machines running on a host

• Each Virtual Machine talks to a Virtual Machine serving as a NAS device

Problem:

• Suddenly, I cannot log in to any of the Virtual Machines (really slow!)

Initial Speculation

• Virtual Machines are saturated in some resource



A More Complicated Disk Example, Part I (CPU)

Predictable CPU usage,Host not saturated

“Chaotic” CPU usage,Host saturated



Complicated Disk Example, Part 2: Disk Usage

Uneven, reduced disk usage

Predictable, balanced disk usage



Complicated Disk Example, Part 3: Write Rate

Increased write traffic, zero read traffic

Read and write traffic



Complicated Example: Putting it all Together

Each App Virtual Machine reads to & writes from same N AS

Something caused excessive writes from each Virtual Machine

• Increased CPU usage per Virtual Machine

• Increased write traffic per Virtual Machine

• Ton of writes on NAS VM!

Issue

• Bug in application within Virtual Machine caused error condition

• Error condition caused excessive writes to same NAS

- Network traffic for application VMs, disk traffic on NAS VM

• Each Virtual Machine is so busy writing that it never reads





• Memory

• Disk

• Network



Network Performance Problems 101

What should I look for to figure out if network is an issue?

• Am I getting the packet rate that I expect?

• Am I getting the bandwidth (read/write) I expect?

• Is all traffic on one NIC, or spread across many NICs?

• [more advanced…not available through counters]: out-of-order packets?

What are some things I can do?

• Check host networking settings (full-duplex/half-duplex, 10Gig network vs 100Mb network?, firewall settings)

• Check VM settings: all VMs on proper networks?



Useful networking metrics



Packets transmitted (in sampling interval)

PKTTX/s net.transmitted.average Packets transmitted in sampling interval

Packets received (in sampling interval)

PKTRX/s net.received.average Packets received in the sampling interval

Data transmit rate (KBps)

MbTX/s net.transmitted.average Amount of data transmitted per second*

Data receive rate (KBps)

MbRX/s net.received.average Amount of data received per second*




Network Performance Troubleshooting

Customer complains about slow network� She’s running netperf on a GigE Link

� She sees only 200Mbps

� Why? I bet it’s that VMware stuff!!

- Note to reader: Please don’t blame VMware first ☺

Where do we start?



Where do we begin? Check VM Bandwidth.

Measure VM Bandwidth (net.transmitted.average)� 200 Mb/s

� Screenshot from the vSphere client



Check Host Bandwidth

Measure Host Bandwidth (net.transmitted.average)� Host sees around 900Mbps…why is VM at 200Mbps?

� Hmm…are we sharing this NIC with multiple VMs?



All VMs using same NIC (VM Network)

All VMs using “VM Network” and sharing 1 physical NIC



All Traffic is Going Through One Nic!

Measure per-physical-NIC traffic

� Hmm…all VM traffic is going through 1 NIC

� Let’s split the VMs across NICs

All traffic through oneNIC on this host



Split VMs Across Multiple NICs. Bingo!



So Complicated! Where do I begin?

So much to track…

But I have different needs! For example…



What if I want esxtop for multiple VMs at once?

vTop: A stats display similar to esxtop, but showing multiple metrics per VM in one screen



What if I want to see if DRS is ‘working’?

Highlight the effectiveness of DRS by overlaying stats and events data



What if I Want to Compare Guest/Host Metrics on One Chart?

Overlay stats data from the ESX and the GOS



What If I Want To Compare the Throughput of 2 RPs?

Aggregate stats data by resource pools, clusters, group of VMs(Windows, Linux), vApps, etc.



Sounds Great…but how do I do this stuff?

Basic Steps

1. Collect the Stats you Need

2. Interpret the Stats

3. Present the stats

Observations

1. Stats collection is similar across customer environments

2. Interpretation/Presentation is specific to customer

Stats collection is a major pain point

1. For customers: so many stats, complex API

2. For us: we get the same questions over and over again

So…we’ve created a reference application for Stats Col lection



Agenda

Use Cases


Q & A




Monitoring Applications consist of a few standard pa rts:

• Selecting entities to monitor (hosts, VMs, etc.)

• Selecting metrics to monitor (CPU, Memory, etc.)

• Choosing collection frequency

• Keeping abreast of inventory changes

• Collecting stats

• Presenting data in a meaningful way



StatsFeeder: A Reference Monitoring Application

Present data in a meaningful way

Collect statistics(note: many sources!)

Keep track of inventory changes

Select Entities andMetrics to monitorand set collectionfrequency



A Brief Walkthrough of StatsFeeder

1. Select Entities/Metrics to Monitor

2. Keep Track of Inventory Changes

3. Collect Statistics

4. Present Statistics



Selecting Entities/Metrics to Monitor

Relevant Modules in StatsFeeder

1. Counter Info Initializer

2. User Input Reader

3. Entity MoRef Retriever

4. Entity Metrics and Performance Provider Summary



Counter Info Initializer

Retrieves available counters.

Tip:

• Counter information is static: retrieve only once.



Counter Info Initializer: Code Snippet

The PerformanceManager managed object contains a proper ty with the name ‘perfCounter’

It is an array of type PerfCounterInfo.

Use PropertyCollector to retrieve this property and get the supported counters.

PropertySpec pSpec = new PropertySpec();

pSpec.setType(“PerformanceManager”);

pSpec.setPathSet(new String[]{“perfCounter”});

ObjectSpec oSpec = new ObjectSpec();

oSpec.setObj(serviceContent.getPerfManager());



User Input Reader

Reads the configuration file for:

• The entities for which the performance statistics are being retrieved.

� Specify individual entities by name

� Specify the name of the container and the child type to monitor the entities in that container

• The performance statistics to retrieve.

• The frequency at which the performance data should be retrieved.



User Input Reader: Sample Configuration File



Entity MoRef Retriever

Retrieves MoRefs of entries in configuration file.

Tricks:

• MoRef does not change once an item is added to the vCenter inventory.

• MoRef does change if entity is removed and re-added.

Therefore:

• Maintain cache of MoRefs

• Monitor changes to the inventory

• Use Views to retrieve MoRefs



Entity MoRef Retriever

Retrieve the MoRef of individual entities using Prope rtyCollector or SearchIndex.

Retrieve the MoRefs of entities in a container using ContainerView:

• Obtain the MoRef of the container

• Create the ContainerView

• ContainerView has a property with the name ‘view’ and it is an array of type ManagedObjectReference. This contains the MoRefs of all the types specified during the ContainerView creation.

vimPort.createContainerView(

serviceContent.getViewMgr,

containerMoRef,

new String[]{“VirtualMachine”},//MO type true //recursively look in the container

);



Entity Metrics and Performance Provider Summary

Retrieve the metric IDs for all the entities.

• E.g.: For dual CPU VM, metric IDs are 0 and 1 for CPU 0 and CPU 1.

• Caveat: Can’t get this for powered off VMs and host systems, disconnected hosts and orphaned VMs

Tricks:

• Performance Metric IDs for entities do not change if hardware does not change.

Therefore:

• Cache Metric IDs for entities.

Tip:

• Use wild card for metric IDs



Entity Metrics

vimPort.queryAvailablePerfMetric (

serviceContent.getPerfManager(),

entityMoRef,

null,

null,

intervalID //If not specified, defaults to

//available metrics for historical stats.

);



Entity Metrics and Performance Provider Summary

Performance Provider Summary provides information about:

• The type of statistics it supports (real-time or historical)

• The refresh rate for statistics generation

Performance Provider Summary data is the same for managed objects of the same type.

Tricks:

• You don’t have to retrieve this data for all entities.

• Make sure the interval between two performance data queries is equal to at least the refresh rate to avoid getting the same value.



Performance Provider Summary

vimPort.queryPerfProviderSummary(

serviceContent.getPerfManager(),

entityMoRef,

);



Keeping Track of Inventory Changes

Relevant Module in StatsFeeder: Inventory/Entity Properties Monitor



Inventory/Entity Properties Monitor

Tracks changes to the inventory (to keep EntityMoRefCache up to date)

Tracks changes to the virtual hardware of the VMs (to keep Entity Metrics cache up to date)

Use Views to monitor these changes




To monitor for changes to inventory:

• Create InventoryView

• Open the inventory view folders to monitor for changes

• Monitor for changes to ‘view’ property, as it contains the MoRefs of all the managed objects in the inventory

vimPort.openInventoryViewFolder(

inventoryViewMoRef,

new ManagedObjectReference[]{}//MoRefs of

//all the containers to expand

);




To monitor for changes to VM hardware:

• Create ListView with the MoRefs of all the VMs

• Create the appropriate TraversalSpec to start from the created ListView

• Monitor for changes to VM hardware using PropertyCollector

vimPort.createListView (

serviceContent.getViewManager(),

new ManagedObjectReference[]{}//MoRefs of

//all the VMs

);

TraversalSpec tSpec = new TraversalSpec();

tSpec.setType(“ListView”);

tSpec.setPath(“view”);




PropertySpec pSpec = new PropertySpec();

pSpec.setType(“VirtualMachine”);

pSpec.setPathSet(propertyArr);//Props String array

ObjectSpec oSpec = new ObjectSpec();

oSpec.setObj(viewObj);//View object

oSpec.setSelectSet(new SelectionSpec[]{tSpec});



Collecting Statistics

Relevant Modules in StatsFeeder

1. PerfQuerySpec Cache

2. Stats Reader

3. Stats Receiver



PerfQuerySpec Architecture

To grab counters:

QueryPerf(PerfQuerySpec[] querySpec)

PerfQuerySpec: Specifies which counters to grab

PerfQuerySpec[]: [pQs1, pQs2, pQs3, …]

• Array of PerfQuerySpec objects pQs1, pQs2, pQs2

• Can grab multiple stats using single QueryPerf call

Entity (host, VM)

Format (CSV, normal)

MetricId StartTime EndTime IntervalID (20s, 300s)

maxSample



Complexities of QueryPerf

How Does vSphere Process QueryPerf(querySpec[])?

1. vCenter receives queryPerf request with querySpec[]

2. vCenter takes each querySpec one at a time

3. vCenter gets data for each querySpec before processing next one

Options for querySpec[]:

1. 1 entry � 1 stat or set of stats for a single entity (e.g., all CPU)

2. Multiple entries. Examples: • Each entry for a different entity …• Each entry for a different stat type, same entity

VM1,cpu.* VM2,cpu.* H3,mem.*

VM1,cpu.* VM1,net.* VM1,mem.*

pQs1 pQs2 pQs3



Implications of QuerySpec

Format of QuerySpec Allows Multiple Client Options

1. Grab each stat one at a time

2. Grab a group of stats per entity at once

3. Grab all stats for all entities at once

4. Grab stats for a subset of entities at once

Some Tradeoffs:

1. Network processing (large result sets vs. small result sets)

2. Client aggregation overhead

3. vCenter processing (Each QueryPerf handled in a single thread)



PerfQuerySpec Cache

Reuse PerfQuerySpec, as long as there are no changes to the inventory and changes to the hardware of the corresponding entity.

Tricks:

• Keep the PerfQuerySpec cache up to date using the monitoring module.

• Use wild card for metric IDs to automatically account for hardware changes.



PerfQuerySpec

PerfQuerySpec pqSpec = new PerfQuerySpec();

pqSpec.setIntervalId(intervalId);

pqSpec.setMaxSample(maxSamples);

perfQuerySpec.setFormat(“csv”);//Set format to CSV

PerfMetricId pmid = new PerfMetricId();

pmid.setCounterId(counterId);

pmid.setInstance("*");//Wild card

pqSpec.setMetricId(new PerfMetricId[]{pmid});

pqSpec.setEntity(entityMor);



Stats Reader

Retrieves the performance data from the vCenter Server and various guest specific performance tools like xperf, perfmon, or top etc.

Tip:

• Use parallelism



Stats Reader - Using Parallelism

class StatsReader implements Runnable {

private PerfQuerySpec[] perfQuerySpecArr = null;

public StatsReader(PerfQuerySpec[] pqsArr) {

perfQuerySpecArr = pqsArr;

}

public void run() {

if (perfQuerySpecArr != null) {

vimPort.queryPerf (serviceContent.getPerfManager(),

perfQuerySpecArr);

}

}

}



Stats Receiver

This is a simple interface to process the output data from the StatsReader.

• By default CSVFileStatsReceiver and DefaultStatsReceiver are provided.

� CSVFileStatsReceiver – Writes the performance data in CSV format to a file

� DefaultStatsReceiver – Writes the performance data to the console.

Users can simply implement this interface to create their own StatsReceivers to:

• Write the performance data to a database.

• Publish the data to a message queue.



Stats Receiver

/*

All the receivers need to implement an IStatsReceiv er interface that has ‘receiveStats’ method

*/

public void receiveStats(String entityName,

PerfEntityMetricBase[] pembArr) {

writeStatsToDB(entityName, pembArr);

}



Present Data in Meaningful Way

Relevant modules in StatsFeeder1.Stats Interpreter

2.Stats Presenter



Stats Interpreter

Interprets the performance data to identify potential problems.

You can write your own rules engine to interpret the performance data and plug it in easily.



Stats Presenter

Interpret the data using your own rules and present it in your own way.

• E.g.: Present the data in graphical format using histograms, charts etc.



Stats Presenter Sample



Summary of Pieces

What counters are supported?

Get entities to monitor, stats to retrieve,and collection interval from config file

Retrieves MoRefs of entities in config file

Retrieves metadata for entities to be monitored and stats to be collected

Watch for inventory changesMaintain cache of querySpecs

Collect Data

Store data (CSV, DB, etc.)

Sample Rules Engine

Present data



Conclusion: Performance API Tips and Tricks

Collect static data once

• CounterIDs, metricIDs, MoRefs etc.

• Use Views to keep this data up to date.

• Reuse PerfQuerySpec as much as possible

Use CSV format

• Reduces serialization cost and the size of metadata

Choose metrics and query intervals carefully

• Query the real-time stats at a slower rate than the refresh rate

• Query over small time increments

• Choose correct stats levels

Use parallelism (multi-threaded clients)



Resources for Developers

StatsFeeder Source Code:

http://communities.vmware.com/servlet/JiveServlet/do wnloadBody/11933-102-1-11789/StatsFeeder.zip

Developer Community:

http://communities.vmware.com/community/developer/foru ms/managementapi

Code Samples:

http://communities.vmware.com/community/developer/cod ecentral

Blogs:

http://blogs.vmware.com/developer



Q & A

• Questions ???• Please complete the survey.



Ready. Set. Go! Take Action

GO! Get Educated

•Visit Partner University for information on partner programs and resources

•Access to VMware Partner Exchange presentations

GO! Get Competent

•Enroll in training and certification that counts toward earning competencies

GO! Register Your Deals to Earn More Margin

•Visit Partner Central to learn about VMware’s registration programs



Any Questions?

building vsphere perf monitoring tools

Sports

affect final

intellectual

relieve memory

important

inventoryentity

performance

average average

average mem