INTELLIGENT, POLICY-DRIVEN ORCHESTRATION OF SENSORS AND EFFECTORS ACROSS THE

DATA CENTER IN REAL-TIME This document describes the technical methodology of Sychron's "Infrastructure Orchestration" solution, giving detail on the methodology required to create a a real time closed loop system that intelligently coordinates 'sensor' information from performance, utilization, systems management, and service level management monitors with appropriate activation of "effector' tools such as auto-provisioning, single server resource management, clustering, load balancing, and virtualization across a diverse infrastructure.

BILL MCCOLL

FOUNDER AND CTO, SYCHRON INC.

mccoll@sychron.com

2 CONFIDENTIAL

TABLE OF CONTENTS

INTRODUCTION …………………………………………………………………………………..3

Business Accelerates to Real Time …………………………………………………………3

Moving to a Real-Time Infrastructure ………………………………………………………4

Orchestrating IT for Business ……………………………………………………………….4

THE SYCHRON POLICY-DRIVEN ORCHESTRATOR ……………………………………....5

Policy Conditions and Triggering …………………………………………………………...9

Aggregation and Virtualization ………………………………………………………………9

Demand-Driven Health and Performance Monitoring …………………………………..10

Policy Variables ……………………………………………………………………………...11

Policy Expressions …………………………………………………………………………..12

Goal-Oriented Policy-Driven Orchestration ………………………………………………12

Intelligent Optimized Decision Making ……………………………………………………13

Highly Scalable Policy Architecture ……………………………………………………….14

BUILDING A COMPLETE DATA CENTER AUTOMATION SOLUTION ………………..15

Component Matrix …………………………………………………………………………..16

ALTERNATIVE APPROACHES ………………………………………………………………..17

Scripting Best Practices: IBM Intelligent Orchestrator …………………………………18

Monitoring and Root Cause Analysis: Vieo ………………………………………………19

Grid Scheduling: Platform Computing ……………………………………………………20

CASE STUDY: GLOBAL FINANCIAL SERVICES COMPANY …………………………….21

CONCLUSION ……………………………………………………………………………………23

3 CONFIDENTIAL

INTRODUCTION

Rarely has there been such a degree of unity within the computing industry. Today, every major hardware and software vendor agrees that during the decade 2000-2010 we will witness a wholesale transformation of the IT world to one in which computer power is delivered on demand, in real-time, to business applications and services as they require it. Each vendor has their own language to describe how they will adapt, evolve and enhance their product line to embrace and deliver on the promise of the “real-time utility vision”. IBM has its autonomic and on-demand initiatives, HP has its utility data center plans, Sun has the N1 initiative. The other major players, such as Veritas, Microsoft, CA and Dell, all have similar visions of how the computing landscape will be transformed by this real-time, utility megatrend.

What seems to be missing from these various announced initiatives, at least at this stage, is a clearly defined technical strategy for “connecting the dots” between where we are today in 2004 and where we all agree the industry wants to reach sometime during the period 2007-2010. In particular, there is no clear statement on how the critically important “real-time policy engine” component will be realized. In this white paper we address that issue. We describe the main characteristics of the Sychron™ policy-driven resource Orchestrator, the first example of a fully extensible general purpose language and run-time system for specifying and automatically enforcing business goals and policies across a large scale data center infrastructure. Leveraging, integrating and enhancing the various data center technologies already available today, this new open standards based product provides a major leap forward in terms of delivering on the vision of a truly scalable data center infrastructure that is driven automatically in real-time by high level business goals and policies.

BUSINESS ACCELERATES TO REAL-TIME

A recent report by Gartner Research showed that in the space of just four years (1998-2002) the velocity of a whole range of standard business processes had accelerated by factors of several hundred to several thousand. Some examples were: trading analytics down from thirty minutes to five seconds (1800x acceleration), call center enquiries down from eight hours to ten seconds (2880x), tracking finances down from one day to five minutes (288x), supply chain updates down from one day to fifteen minutes (96x), and data warehouse renewal down from one month to one hour (720x). Even in an industry that has lived with the dramatic impact of Moore’s Law for several decades, and become accustomed to exponentially “Better, Faster and Cheaper” systems, these rates of acceleration are staggering. They show clearly that all critical areas of business are now becoming real-time. The reasons for this shift to real-time are also equally clear.

As business become more and more competitive, companies

• need to be able to adjust plans on the fly in reaction to unexpected and unpredictable fluctuations in demand, supply, competition, pricing, interest rates, oil prices, weather, stock markets etc.

• need to reduce risk and increase competitive advantage by sensing earlier and reacting with instantaneous agility

• need to increase customer satisfaction through improved service availability and responsiveness

4 CONFIDENTIAL

Given these imperatives, it is clear that driving down time to respond is as important as (or possibly more important than) reducing costs. In a very strong sense, we see that today “time really is money”.

MOVING TO A REAL-TIME INFRASTRUCTURE

What do businesses want from IT? Numerous recent surveys show clearly that they want

• Increased efficiency. Hardware utilization and efficiency levels need to be dramatically increased. For years now, companies have been forced to massively overprovision their IT infrastructure in order to be able to accommodate unexpected and unpredictable spikes in demand. This has resulted in average levels of utilization of around 15%, a figure that is now no longer acceptable. Businesses want utilization levels of 70%-90%, but want it achieved in a way that ensures that high priority tasks continue to get the resources they need.

• Improved alignment. Infrastructure needs to be directly aligned with business priorities and policies. In today’s fiercely competitive environment, IT has to deliver in clear business terms. There is no interest in deploying a technology simply because it is new, and claimed to be superior. IT allocations, like any other part of the business, have to be driven, increasingly in real-time, by rapidly changing fiscal/economic considerations, with clear and accurate fine-grain reporting to enable appropriate billback and reconciliation.

• Increased availability and responsiveness. Service availability and service level priorities need to be guaranteed. Downtime is unacceptable. Service level agreements need to be enforced.

Moreover, they want all these capabilities to be delivered at all times, and in all conditions, including periods of rapidly fluctuating and unpredictable changes in demand and in priority. To achieve this we need to move to a real-time infrastructure, to get our IT in sync with the business.

ORCHESTRATING IT FOR BUSINESS

As the economy rebounds, it is becoming clear that the next massive megatrend in enterprise computing is going to be “Orchestrated IT” – the architecting and deployment of automated, dynamic IT infrastructures that second by second, minute by minute, are always exactly aligned with the rapidly changing business priorities and requirements within a real-time enterprise, always delivering exactly the right IT resources to each application and service as demand on each of those fluctuates rapidly and unpredictably over time.

Over the past few years, a wide range of basic enabling technologies for Orchestrated IT have been developed and deployed within data centers. The list includes tools and technologies for operating system virtualization, application installation and configuration, server provisioning, application monitoring, clustering and checkpointing, load balancing, single-server resource fencing, root cause analysis, grid scheduling. This patchwork of manual and (semi-)automated sensor and effector tools is about to be integrated and supercharged by the release of Sychron™, the first product designed to leverage, integrate and enhance these point solutions into what Forrester has called a “Fabric Operating System”, an intelligent and automated command-and-control system for the real-time data center.

5 CONFIDENTIAL

The coming Orchestrated IT tidal wave will be driven by major technology breakthroughs in three separate and vital aspects of data center automation: Speed, Scale and Policy. A real-time data center infrastructure needs to be able to automatically execute a continuous feedback control loop, consisting of many complex global (whole data center level) operations. Moreover it has to execute this loop in a matter of seconds, across a large heterogeneous pool of Windows, Linux and Unix servers, perhaps containing hundreds or thousands of servers/blades. At its most basic, the fundamental “Real-Time Orchestrated IT Loop” involves five steps:

1. Aggregation. Dynamic detection, aggregation and virtualization of all hardware and software components through various sensors

2. Awareness. Demand-driven global monitoring of the health and performance of all hardware and software elements via those sensors

3. Triggering. Dynamic evaluation of policy conditions and triggering of requests for action.

4. Decision. Optimized resolution of conflicts and competition amongst requests, based on current business priorities.

5. Action. Initiation and global control of multiple distributed actions throughout the data center using multiple local “effector tools and technologies”, e.g. auto-provisioning, load balancing, server resource fencing.

Each one of these five steps presents formidable technological challenges, given the incredibly short timescales (seconds) and the potentially very large server pool sizes (hundreds or thousands of servers) involved. Taken together, the five steps constitute a combined set of requirements that can only be achieved by a radically new kind of software architecture for data center control, one that is ultra-fast, massively scalable and policy-driven. Moreover, to achieve the speed and scale required we need a new kind of in-band design in order to achieve the necessary performance levels, in the same way as on any PC or server we need the operating system to be in-band to avoid any latency or inefficiency, i.e. we need the operating system on the PC or server, not sitting outside.

In this white paper we describe the main features of the Sychron product, showing how its innovative, massively-scalable in-band design delivers, for the first time, the key requirements of intelligent, policy-driven sensor and effector orchestration, and thus enables, for the first time, the real-time orchestration of IT for business.

THE SYCHRON POLICY-DRIVEN ORCHESTRATOR

Over the last few years, a number of new tools and technologies have emerged that provide important solutions to various static, non-real-time aspects of data center automation. Some examples of companies in these areas include the following:

6 CONFIDENTIAL

1. Operating system virtualization and protected domains for single-servers, SMPs and mainframes. HP, IBM, Sun, VMware, Xen, Revario, Microsoft Virtual Server (Connectix), Intel. These effector technologies allow multiple operating systems to reside on a single server.

2. Tools for semi-automated installation, configuration, provisioning and deployment of system software, applications, networks, servers, blades, storage systems. HP, IBM (Think Dynamics), Sun (CenterRun), Veritas (Jareva), Microsoft Systems Management Server, CA, Bladelogic, Moonlight, Altiris, Verbatix, Opsware, Ejasent, Egenera. These effector tools can be used to simplify many of the activities that need to be carried out as part of the whole administrative/management lifecycle for hardware and software systems.

In addition to these non-real-time elements, there are also a number of real-time building blocks available today in various areas:

3. Monitoring of single-server application instances. [Sensors] HP, IBM, CA, Veritas (Precise), Mercury, Wily.

4. High-availability clustering, checkpointing, migration of live sessions. [Effectors] HP, IBM, Sun, Veritas, Microsoft, VMware, Ejasent, Eternal.

5. Load balancing and traffic management. [Effectors] F5, Cisco, Veritas, Linux Virtual Server.

6. Single-server CPU and memory fencing. [Effectors] HP, IBM, Sun SRM, Microsoft WSRM, Aurema.

7. Single-server bandwidth fencing. [Effectors] Sun, Microsoft.

8. Single-server power management. [Effectors] Intel.

9. End-to-end monitoring, event correlation and root cause analysis. [Sensors] Veritas (Precise), CA Sonar (Silent Runner), Altaworks/Resonate, Vieo.

10. Grid scheduling and resource management. [Effectors] Platform, DataSynapse, United Devices, Global Grid Forum.

Given this array of productivity tools and basic building blocks, what is needed now is a policy engine that can coordinate and control these various elements in order to realize an integrated, orchestrated real-time infrastructure. Sychron provides that orchestration, offering the opportunity to achieve real-time optimization of application resources across a large-scale data center infrastructure. More specifically, it provides the following critical elements of real-time orchestration:

7 CONFIDENTIAL

11. Global aggregation and virtualized orchestration of all servers and load balancers.

12. Global aggregation and virtualized orchestration of all external bandwidth connections.

13. Global aggregation and virtualized orchestration of all application software components (processes, sessions, EARs, threads etc.) across all servers.

14. Global real-time monitoring of health and performance for all hardware and software components.

15. Policy language and fully distributed, massively scalable run-time system.

16. Real-time condition-action triggering of requests to instantaneously burst or constrain resources in line with fluctuating demand on applications.

17. Real-time decision: intelligent analysis and optimized resolution of conflicts and competition between resource adjustment requests.

The range of effector actions that may be incorporated into policies is broad and fully extensible.

They include:

• Increasing or decreasing the number of servers, blades or virtual machine instances allocated to an application. This might, in some cases, involve real-time orchestration of the use of a server provisioning and configuration tool.

• Increasing or decreasing the CPU or bandwidth shares allocated to an application on a particular server. This will involve real-time orchestration of he associated single-server building block technology, e.g. Windows System Resource Manager.

• Orchestrating the turning on power saving mechanisms on a particular server or blade, or turning it off again through a Wake-On-LAN capability.

• Orchestrating the changing of a load balancer method. For example, changing an F5 BigIP hardware load balancer from operating in Round Robin mode to Least Connections mode.

Other changes may be automatically required as a consequence of an action. For example, a web-facing load-balanced application that has its server pool increased or decreased will require that the load balancer be automatically adjusted accordingly.

The following diagram outlines the flow of information between the various components of a real-time orchestrated IT loop.

In the folloptimizatmechanisof these rachieved

Althoughnoted thaa key parfiscal recfrom the

Policy based triggering and goal

oriented orchestration

EEffffeeccttoorrss

8

owing sections we describe the core technologies that pion” that lies at the heart of the real-time policy engine: thm and the massively scalable parallel algorithms for optiequire unprecedented levels of real-time global awarenes through a combination of randomization and high-perfor

our emphasis here is on describing the core policy engint a number of other valuable benefits result from the realt of the engine. In particular, the data gathered permits hionciliation to be achieved, based on the detailed usage mengine.

Clustering r

L

Load Balancers

Provisioning

Continuous aggregation of

variable conditions

SSeennssoorrss

Utilization Monitors

DDaattaa CCeenntteer

App. Perf. Monitors

Systems

Virtual OS’s

Mgmt.

SRM’s

CONFIDENTIAL

rovide the “distributed intelligent e distributed parallel triggering

mized contention resolution. Both s, and we outline how that is

mance low-latency messaging.

e technologies, it should be -time fine-grain monitoring that is ghly accurate chargeback and etering data that is available

Service evel Mgmt.

9 CONFIDENTIAL

POLICY CONDITIONS AND TRIGGERING

At the heart of the Sychron policy engine is a highly scalable, distributed policy language based on real-time parallel evaluation of local and global Excel-like policy expressions across the whole server pool. This unique and very powerful triggering mechanism uses built-in and user-defined variables which capture instantaneous values and histories of resource consumption, health and performance metrics from hardware elements and application software components, environment details etc. providing a complete real-time view of the present state and histories of all aspects of the hardware and applications. This real-time global awareness is the background against which policy conditions are evaluated and triggered in real-time.

AGGREGATION AND VIRTUALIZATION

A key enabling technology that allows real-time global awareness to be achieved is Sychron’s unique technology for the dynamic detection, aggregation and virtualization not only of the various hardware elements in the data center, but also of the multiple processes etc., among the multiple application instances, that aggregated together constitute a single distributed (virtual) application to which the appropriate policy should be applied across that virtualized hardware infrastructure. Having a virtual, combined view of application software components is as important as having a unified view of the underlying hardware. That capability has been achieved for the first time in the Sychron policy engine, and enables policy-based orchestration of distributed applications to be performed at many levels

The policy engine automatically creates an inventory of all running applications across the server pool, utilizing defined application detection rules. The rule set may be modified at any time, which allows the user to dynamically alter the way processes are organized into applications. The product identifies and logically classifies processes spawned by an application, using attributes of the operating system process hierarchy and process execution environment. Default rules are included with the product, and custom rules may be easily created and edited through the Sychron GUI. The application rules are described as an XML document.

The following brief description summarizes the main features of the application aggregation capabilities of the policy engine. A set of process rules and flow rules are used to specify the operating system processes and the network traffic that belongs to an application. The policy engine uses these rules to identify the various components that together constitute a single application, for the purpose of real-time policy-based orchestration. All components of an application are managed and have their resource utilization monitored as a unified entity, with individual server application instance breakdowns available for most functionality.

The definition section of an application rule comprises a non-empty set of process rules and flow rules. Each process rule specifies that the operating system processes that match a certain specified combination of some subset of:

process name, process ID, user ID, group ID, session ID, command line, environment variables, parent process name, parent process ID, parent user ID, parent group ID, parent session ID, running on a specific server

belong to the application. Optionally, all their child processes may be declared to belong to the same application. Each flow rule specifies that the network traffic associated with a certain local IP address/mask and/or local port belongs to the application.

10 CONFIDENTIAL

DEMAND-DRIVEN HEALTH AND PERFORMANCE MONITORING

Driven by the demands of the policy conditions, the Sychron product continuously gathers exactly the right detailed real-time information on the health and performance of all hardware elements and applications across the server pool. Information is gathered on the availability/status of individual servers, use of resources by each application, number of flow connections made to each server, configured IP addresses etc. Recent data (last few seconds, minutes), along with the latest policy information, is held in a real-time distributed fault-tolerant data cache that is maintained by the Sychron agents on each server across the server pool. Older data is held in a conventional archive database that can also be rapidly queried by the policy engine to obtain critical historical summary information on past performance.

One of the fundamental and unique strengths of the Sychron approach to resource orchestration is that not only is it policy-driven, but it is also demand-driven in the way it gathers live performance data from the data center. A more simplistic approach (and the one that is used in most existing attempts to provide data center automation) would simply operate in a data-driven style, where high volume streams of monitoring data are generated and then filtered to produce a summary picture that is then analyzed either manually or using some tools or scripts. The time and resources required to generate, store, filter etc. these huge data streams means that it is impossible to get even close to the level of performance required to be able to trigger policy conditions and actions in real time. With Sychron, monitoring data need only be gathered if it is required as part of a policy condition evaluation. This is a key element in being able to achieve the necessary scalability and performance.

The demand-driven health and performance monitoring performed by Sychron is completely non-intrusive, in the sense that it requires no changes to application software. The policy engine can also leverage Application Performance Management products (e.g. Veritas i3, Wily IntroScope, Mercury Optane or Topaz, etc.) that monitor performance of server-side Java and J2EE applications. These solutions can provide detailed real-time application performance data generated from inside an application server environment, such as response times, queue lengths from various Java/J2EE components (servlets, EJBs, JMS, JNDI, JDBC, etc.), all of which can be automatically exploited within the policy engine. Similarly, the policy engine can exploit information from tools that capture deep end-to-end knowledge, and perform event correlation, across all hardware and software layers in the data center: network, servers, storage (from the internet to the disk and back again).

Example: A BEA WebLogic server periodically logs its average transaction response time to a file. An application performance policy has been created which queries this file and, through the Sychron policy engine, instructs more server power to be provided whenever the application’s transaction response time increases above 500ms. This same type of performance metric could also be obtained from a product such as Mercury’s Topaz or Wily’s Introscope and used by the policy engine.

Sychron monitors the behavior of J2EE application servers such as Weblogic, WebSphere or Oracle 8i AS, using their MBean interface so that predefined actions can be taken by Sychron when certain conditions are met. For example, Sychron can receive events from WebLogic to inform the product of WebLogic’s status, i.e. whether it is up or down, or what the average number of transactions per thread is. These metrics can then be matched against actions, defined by the user in the application’s rule, to affect appropriate environmental changes with the aim of improving the execution of the application. The actions that may be taken are to modify the application’s policy, issue an Explicit Congestion Notification to inform network devices and load balancers to delay or reroute new requests, or to execute a local script. A summary of the interaction between Sychron and the MBean is shown below:

SyapSycothe

PO

Sytimanlanvarutibares

e Registration

Perform action associated with vent notification

11 CONFIDENTIAL

chron establishes a connection to the application server in order to request its MBean. The plication server validates the security of the request and then sends the MBean structure to chron. Sychron then registers with the application server to get a notification whenever certain nditions occur within the application server. The user configures the events to register for, and consequential actions to take, in the application’s detection rule.

LICY VARIABLES

chron’s unique policy language technology provides a general framework for guaranteeing real-e service level agreements in the presence of rapid and unpredictable fluctuations in demand

d in priority. This post-scripting framework is based on an abstraction that has an expression guage that contains variables that are either user-defined or built-in. Each of the built-in iables identifies some characteristic of the running app instance, for example, the CPU lization. The scope and flexibility of the policy language allows real-time policy conditions to be sed on a huge range of possible factors – time of day, resource consumption, transaction ponse times, queue lengths, other custom conditions.

J2EE Application Server (Weblogic,

Websphere) MBean

Notification

12 CONFIDENTIAL

POLICY EXPRESSIONS

The policy language allows policy condition expressions to be constructed from the values of the built-in variables and the values returned by user-defined scripts. The Excel-like language has the standard set of arithmetic and relational operators, plus a series of functions. The arithmetic and relational operators include: NOT, IF, AND, OR, EQ (Equal), NE (Not equal), GT (Greater than), GE Greater than or equal), LT (Less than), LE (Less than or equal). The functions are split into two classes:

• Local functions that are applied to scalar values such as literals, or the value of a variable on a single server. For example, ISSET (Is variable set on this server), NOW (Current date/time), DAY (1-31), HOUR (0-23), MONTH (1-12), WEEKDAY (1-7), SYSTEM (Exit code from script), SYSTEMVAL (Number returned from script).

• Global functions that read all the defined instances of a variable on all servers, and summarize this data using a grouping operator such as SUM, PRODUCT, MAX, MIN, COUNT, AVERAGE. If the variable is not defined on any of the servers, then an error is raised. If the optional from and to parameters are used, then the period of interest for the variable is the current time minus from seconds, to the current time minus to seconds. For example, AVERAGE(PercCpuUtilPool,4200,3600) is a rolling ten minutes average CPU utilization from an hour ago.

GOAL-ORIENTED POLICY-DRIVEN ORCHESTRATION

A major, and very powerful, feature of Sychron is the fine-grain monitoring and recording of all aspects of the health and performance of the hardware and applications over time. With this information held in the real-time distributed data cache and in the on-line archive, the policy variables and expressions in the triggering mechanism are able to capture and use not only the instantaneous current values but also summaries of the historical sequence of values.

A central issue in any feedback control system and in any policy-based triggering mechanism is the problem of rapid and inefficient oscillation between states, as trigger conditions fire, adjustments are made, those adjustments cause triggers in the opposite direction to go off, adjustments are again made this time in the opposite direction, and so on. By basing the Sychron policy language and engine on variables which capture history rather than just instantaneous values, we provide a direct, simple and easy-to-use means of achieving the hysteresis necessary to avoid (a) triggering a change unnecessarily due to an unimportant and unrepresentative spike, and (b) oscillating inefficiently between states.

Another powerful advantage of having both a real-time capacity to react and adjust, and real-time access to detailed historical information on previous situations (e.g. demand applied, resources allocated, performance achieved, for a particular app) is that we can achieve not only direct policy-driven orchestration, but also goal-oriented policy-driven orchestration that is guided by past experience. For example, a policy that regulates resource allocation in order to achieve a guaranteed transaction response time under rapidly varying levels of demand can not only be based on a simple, predictive model but also on (or alternatively on) knowledge of past situations. As in many other areas of computing where there is a high degree of unpredictability, speed, feedback, and a data store of past knowledge can be far more effective than complex rule-sets, complex models and artificial intelligence methods of symbolic inference.

13 CONFIDENTIAL

INTELLIGENT OPTIMIZED DECISION MAKING

The Sychron policy trigger mechanism generates a large number of simultaneous real-time requests to adjust resources in various ways. These requests will in many cases be conflicting and competing. What is needed then is a real-time intelligent decision making mechanism that can resolve these conflicting and competing requests in a way that results in the best possible business-aligned orchestration across the server pool or data center. To achieve this, the decision making mechanism needs to be driven not only by clearly defined business priorities (which may, of course, themselves be dynamically changing over time) but also by a clear real-time global awareness of the current constraints and potential conflicts.

In general, such a problem will be a complex combinatorial optimization problem for which an optimal solution would require exponential time for the computation (perhaps hours, days, or weeks). Given the infeasibility of determining the optimal solution in any reasonable period of time, our more pragmatic goal is instead to determine a very good solution, but to determine it very rapidly – in real-time (seconds). Eliminating a number of detailed issues, the core problem (bulk real-time resolution) is essentially as follows. Given:

• A heterogeneous server pool comprising p servers (or blades or virtual server instances)

• A set of n applications running on those p servers

• A set of triggered requests to increase (or decrease) the resource allocations to some of those applications

• A set of constraints determining which subset of servers in the pool a given application can be run on

• A set of current business priorities which defines the relative priority of any two competing requests

• A set of architectural constraints defining ideal hardware requirements of a particular application instance, load balancer coverage, low bandwidth network points, geographical considerations etc.

• A set of server dependencies and mandatory minimal requirements that are dictated by the internal characteristics of an application. For example, if running on server 7 it will automatically copy information to server 14. Therefore, servers 7 and 14 are pinned and have to be allocated and de-allocated together.

What is required is a policy-driven orchestrator that can take account of this array of considerations and (in real-time) produce a selection of real-time actions to be performed which resolves the various conflicts and competition within the set of triggered requests in a way that is best aligned with the business priorities.

The Sychron policy language allows business priorities and various kinds of server dependencies and minimal app requirements to be defined and adjusted over time. Based on this information and on the detailed inventory of information concerning where applications can be run, the policy engine uses an ultra-fast scalable parallel algorithm to resolve contention and competition in real-time.

14 CONFIDENTIAL

HIGHLY SCALABLE POLICY ARCHITECTURE

We have seen that we can achieve real-time policy-driven orchestration of application resources, in direct alignment with business priorities, as long as we are able to perform the following, in real-time, across very large scale server pools:

• Aggregation and virtualization of all hardware and software components (servers, blades, external bandwidth connections, application processes)

• Fine-grain monitoring of the health and performance of all hardware and applications

• Setting of policy variables and evaluation of policy expressions to trigger requests for action

• Resolution of conflicts and competition via intelligent optimized decision making

• Initiation of required actions across the server pool, to adjust resources and interoperate with other tools

To achieve all of this in a matter of seconds, across a server pool that may contain hundreds or thousands of servers, requires an in-band and ultra-efficient, scalable policy architecture.

Sychron is based on an in-band agent architecture, in which a lightweight, low-overhead software agent is situated on each server, on top of the operating system (Linux, Windows, Unix) on that node. The in-band positioning and architecture of the set of distributed agents enables them to act together as a kind of “meta operating system”, handling all aspects of the real-time orchestration of the server pool, in a way that is analogous to the way in which an operating system sits on a single server, orchestrating the various applications that are running on that server.

The agent architecture is completely non-intrusive to applications, and can be deployed on existing data centers with no changes to the physical architecture, and with no new hardware required. For example, it can be deployed

• Within an existing heterogeneous data center to provide policy-driven orchestration of a collection of Linux, Windows and Unix servers.

• As a shrink-wrapped policy-driven blade farm, delivering an integrated, low cost scale-out solution for a particular market segment

The core of the distributed agent architecture is an ultra-fast highly parallel data cache which stores all of the “live” information required by the policy engine. The data cache provides the required real-time and fault-tolerant global awareness through a combination of randomization, low-latency messaging, and parallel processing technologies.

15 CONFIDENTIAL

A COMPLETE DATA CENTER AUTOMATION SOLUTION

We noted above that various sensor tools and effector technologies in the data center automation space already exist. In each of the ten areas mentioned, the Sychron product integrates with, leverages or enhances what they provide, to provide a complete orchestrated solution.

1. Operating system virtualization and protected domains for single-servers, SMPs and mainframes. Sychron leverages resource isolation to improve server consolidation and gain further utilization efficiencies. [Out-of-the-box support for VMware, Microsoft Virtual Server]

2. Tools for semi-automated installation, configuration, provisioning and deployment of system software, applications, networks, servers, blades, storage systems. Sychron further automates provisioning functionality based on application level business requirements. [Customer-driven or configured solutions]

3. Monitoring of single-server application instances. Sychron leverages monitoring tools to enhance application-specific or business-specific resource awareness and actions. [OpenView support planned for 1Q04, others to follow through partnerships]

4. High-availability clustering, checkpointing, migration of live sessions. Sychron co-exists with, and complements, HA clustering and leverages migration of live sessions. [Initial support of VMware VirtualCenter, VMotion technology]

5. Load balancing and traffic management. Sychron enhances load balancing with more business-aware, application-aware resource policies. [Out-of-the-box support for F5, Cisco, Linux Virtual Server]

6. Single-server CPU and memory fencing. Sychron leverages for enhanced server utilization and application performance. [Integration with Microsoft WSRM]

7. Single-server bandwidth fencing. Sychron leverages for enhanced server utilization and application performance.

8. Single-server power management. Sychron provides policy-based orchestration of power management mechanisms to achieve peak energy efficiency.

9. End-to-end monitoring, event correlation and root cause analysis. Sychron leverages to provide advanced and powerful resource policies and actions.

10. Grid scheduling and resource management. Sychron leverages to enable policy engine to drive resource allocation to compute-intensive batch applications during off-hours.

COMPONENT MATRIX

The following table provides a brief summary of the offerings that are available today (February 2004) in each of the 17 key areas of data center automation listed earlier. In areas 2-5 there are many additional companies with products in the market. In those cases we have only listed some of the more prominent vendors, or those that have significant technologies in some of the other areas too.

16 CONFIDENTIAL

Component Matrix

DATA CENTER AUTOMATION CAPABILITIES

IBM

SU

N

HP

VE

RIT

AS

CA

MIC

RO

SO

FT

EM

C

INT

EL

OP

SW

AR

E

ALT

IRIS

AU

RE

MA

F5

CIS

CO

EG

EN

ER

A

VIE

O

ALT

AW

OR

KS

PLA

TFO

RM

SY

CH

RO

N

OS Virtualization & protected domains

√√√√ √√√√ √√√√ √√√√ h

√√√√ i

√√√√

Semi-automated config, install, & provisioning tools

√√√√ a

√√√√ b

√√√√ √√√√ d,f

√√√√ √√√√ √√√√ √√√√ √√√√

Singl Srvrs App. instance monitors

√√√√ √√√√ √√√√ e,f

√√√√ √√√√ √√√√ √√√√

H.A. clustering, checkpointing, session migration

√√√√ √√√√ √√√√ √√√√ f

√√√√ √√√√ √√√√ i

√√√√

Load balancing & traffic management

√√√√ √√√√ √√√√ √√√√

Single Svr. CPU & Mem. fencing

√√√√ √√√√ √√√√ √√√√ √√√√ √√√√

Single Svr. Bandwidth fencing

√√√√ √√√√ √√√√

Single Svr. Power management

√√√√ √√√√

Monitoring, event correlation & RCA

√√√√ e

√√√√ g

√√√√ √√√√

Grid sched. & resource mgmt.

√√√√ √√√√ √√√√ √√√√

Global aggregation & virt. ochstrn. svrs. & load bal.

√√√√ √√√√ c

√√√√ √√√√

Global aggregation & virt. orchstrn. bandwidth cxn..

√√√√

Global aggregation & virt. orchstrn. App s.w. comp.

√√√√

Global health & perf monitoring

√√√√

Policy language and RTE

√√√√

Real time condition action triggering

√√√√

Real time ordering resource requests

√√√√

Recent Acquisitions: a: Think Dynamics, b: CenterRun, c: Terraspring, d: jareva, e: Precise, f: Ejasent, g: Raytheon, h: Connectix, I: VMWare

17 CONFIDENTIAL

ALTERNATIVE APPROACHES

As we have seen, the Sychron product provides a real-time, policy-language-based, in-band solution to the challenge of orchestrating sensors and effectors across a large scale server pool. Real-time responsiveness is the key to achieving the following three critical requirements:

• Peak levels of utilization

• Continuous business alignment

• Automatic self-adjustment to achieve high level goals through on-line real-time feedback, rather than by static off-line algorithmic analysis of complex rule-sets using artificial intelligence technologies

Similarly, having an executable general purpose policy language is the key to achieving:

• A direct on-line connection between high level goals and the myriad of low level hardware and software controls that need to be continuously adjusted in real-time to meet those goals

• Direct policy-based guidance on the tradeoffs that should be made in situations of rapid, unexpected and unpredictable changes in demand, in resource availability, and in priority

Finally, having an in-band policy architecture is key to achieving:

• Massive scalability and real-time performance

• Robustness and fault tolerance through decentralized control

In contrast to this approach, all recent attempts at providing an initial degree of data center “orchestration” have taken a much more non-real-time, scripting-based, out-of-band approach. The reason for this is that these technologies have been aiming to essentially mimic and script the manual processes and workflows that are currently performed by data center operations staff. The Sychron policy-driven orchestrator, on the other hand, is providing something more like the equivalent of a meta-level operating system for the data center. Just as an operating system provides real-time in-band orchestration of competing application processes on a single server, Sychron provides the same type of orchestration at the whole server pool level. Just as it would not make much sense to run a server operating system outside the server, although that is possible, positioning a policy engine outside the server pool (out-of-band) gives rise to the same kinds of inefficiencies and latencies, and reduces the opportunity of achieving the benefits of real-time reaction to change.

18 CONFIDENTIAL

SCRIPTING BEST PRACTICES: IBM INTELLIGENT ORCHESTRATOR

A good example of the alternative first-generation scripting approach is provided by IBM’s Tivoli Intelligent ThinkDynamic Orchestrator (TIO). Quoting from the product webpage “Using IBM Tivoli Intelligent ThinkDynamic Orchestrator, the data center’s most experienced personnel model their knowledge and experience into a repository through a simple graphical user interface (GUI). Then these automated workflows can be executed manually, semi-automatically (via alerts) or automatically” Comparing the product to the Sychron policy engine, we see that it is aimed more at modeling and scripting data center best practices, rather than at providing real-time policy-driven control.

As we now examine the functionality provided by TIO in more detail, we will see that it is in fact highly complementary to Sychron, and that together the two products could offer a number of powerful, synergistic opportunities within the data center automation space.

Our brief outline of the main features of TIO, Version 1.1.0, is based on the Overview Guide (First edition, August 2003, 27 pages) and the Operator’s Guide (First edition, August 2003, 209 pages). The main features of TIO are as follows:

• Installation and Configuration. Configures users and inventory resources among applications in a multi-application environment. The range of resources is broad, including subnets, VLANs, switches, load balancers, routers, firewalls virtual IPs, servers, boot servers, power units, license pools, software products, software patches,…

• Monitoring and Prediction. Gathers information about applications and builds a workload model that is used to predict future resource requirements, and to calculate the probability of future potential breaches in service delivery. Monitoring is performed through a subscription mechanism that is used to selectively capture particular types of data from applications and operating systems, and to apply filtering algorithms to the resulting raw signals. A separate prediction and “workload modeling” component is created for each application under management. The prediction component receives monitored demand information, e.g. hits/sec for a web site, and analyzes the information to determine whether the pattern is random or periodic in some way, e.g. with spikes at a particular time of day, or day of the week. The resulting information is used to provide a probabilistic prediction of future demand levels, and from that a prediction of the likelihood of a breach in service levels if there is no change in resource allocation. Based on this probabilistic analysis, TIO estimates (a) the number of additional servers required to maintain the desired service level, and (b) the effect on application performance of not implementing the adjustment.

• Workflows and Provisioning. Allows users to create, customize, store, reuse and execute scripts from a library of “workflows” for various data center operations such as configuring and allocating servers, installing, configuring and patching software.

Some other points about TIO:

• The product uses a centralized database to store and access (via JDBC) details of all the assets in the data center, e.g. servers, switches, load balancers, application software, VLANs, security policies, server groups allocated and unallocated, service level agreements.

19 CONFIDENTIAL

• The product can be run in one of three modes: manual (generate recommendations), semi-automatic (adjust once authorized), automatic (adjust as soon as all other pending adjustments have been completed).

• The product has various optimization and stabilization (hysteresis) mechanisms. From the brief outline of the approach to optimization in the Product Overview, it appears that the goal is to analyze not only the current dynamic situation (complex enough though that is in large scale data centers) but also to explore the myriad possible permutations and combinations of allocations that are possible throughout the whole future time horizon for prediction. In this respect it seems to be similar in many respects to a conventional off-line batch scheduler or other capacity planning system.

• Although the Product Overview claims that TIO allows “automatic policy-based management”, there is no mention anywhere of any policy language or similar framework. Instead, it appears that TIO is actually “prediction-driven” rather than policy-driven.

• In contrast to Sychron, TIO is not designed to offer real-time responsiveness (resource changes in seconds) and not designed to be applicable to large scale data centers (hundreds or thousands of servers). For example, in TIO an environment is regarded as “volatile” if the infrastructure has to be changed more than ten times in an hour.

In summary, TIO provides a very comprehensive set of configuration and workflow/provisioning tools for simplifying data center administration through the capture and reuse of best practices. For environments where change takes place either slowly or predictably, it also offers a variety of useful capacity planning, modeling and prediction tools.

MONITORING AND ROOT CAUSE ANALYSIS: VIEO

Another example of an alternative approach is the Vieo hardware appliance for “Adaptive Application Infrastructure Management”. Leaving aside the fact that it is centralized, non-scalable and non-fault-tolerant, the major difference between the Vieo approach and that of Sychron is that what Vieo really provides is an out-of-band monitoring, event correlation and root cause analysis capability that can be used to generate alerts as to areas where there may be bottlenecks or other health / performance problems. As such it competes more directly with the various network and system monitoring and root cause analysis tools that already exist in the marketplace (such as Veritas i3, CA Sonar, Altaworks/Resonate) rather than with the policy-driven resource orchestration technology of Sychron.

As noted earlier, a major problem with any data-driven approach to the problem of real-time data center control is the sheer volume of streaming data that any simple server monitoring tool can generate, for example the Ganglia ( http://ganglia.sourceforge.net ) distributed monitoring and execution system used in high performance computing and grids. Since most (almost all?) of this data is unlikely to be actively used in control decisions, the costs of gathering and filtering are likely to be excessive and, most of the time, unnecessary. Sychron’s top-down policy-driven orchestration, with its associated demand-driven monitoring, provides a major breakthrough in efficiency that delivers unprecedented levels of scalability, performance and responsiveness to data center automation, taking it far beyond today’s approaches based on the inefficient bottom-up streaming and filtering of monitoring data.

20 CONFIDENTIAL

GRID SCHEDULING: PLATFORM COMPUTING

A third example of an alternative approach is Platform Computing’s Symphony product for “grid-enabling” applications. Platform’s LSF product has, for many years, been one of the leading schedulers for distributed batch processing in the supercomputing area. Symphony, developed initially for JP Morgan Chase, provides a set of tools that enable organizations to distribute compute-intensive portions of their workload across networked data center resources that may be in multiple administrative domains, i.e. to access computer power as a network service. A centralized web-based interface allows data center operations staff to monitor numbers of pending/running/completed jobs, monitor numbers of idle/in-use CPUs, and to coordinate the provisioning of resources to applications at various times of day, various locations etc. The other element of the product is a set of tools that, along the lines of Globus/Web Services/OGSA/OGSI, support the development, testing and debugging of grid-enabled versions of applications, for use with the system.

For users who want to parallelize and grid-enable their applications, in order to be able to deploy them across a wider range of machines within their organization, Symphony provides a commercial version of the kind of open standards tools that are now becoming available through the efforts of the Global Grid Forum. Although billed as providing “policy-based, real-time execution”, what it actually provides is a conventional “screen watching and scripting” solution to the problem of scheduling data center resources. There is no automated policy-driven control, and hence no real-time orchestration.

Leaving aside the specifics of the Symphony product, it is perhaps appropriate to note, more generally, that the goals of grid computing and those of real-time policy-driven orchestration are quite distinct and highly complementary. Grid computing is primarily focused on the “WAN-level”, where the main challenge is to coordinate the dynamic deployment of applications across multiple administrative domains, which may be widely dispersed geographically. Relatively heavyweight technologies and protocols, such as Web Services technologies, can be used in this context since latency and bandwidth issues are much less important than the ability to tie together a diverse collection of resources. In contrast, a policy engine works mainly at the “LAN-level” and needs to operate at wire-speed levels of efficiency, with minimal latency and minimal CPU and bandwidth consumption.

In a recent and highly influential paper, Jim Gray, Head of Scalable Servers Research at Microsoft, and perhaps the world’s foremost authority on transaction processing systems, noted that because of latency and bandwidth issues there are a number of fundamental impediments that effectively rule out grid computing approaches for most performance-critical applications that need to be deployed on large numbers of servers, and therefore need highly efficient network traffic management. See J.Gray “Distributed Computing Economics” (March 2003), available at www.research.microsoft.com/~Gray/

It is therefore likely that grid computing and policy-driven resource orchestration will remain two distinct, but highly complementary technologies. Deployed together, in an integrated way, they can provide exactly the kind of resource allocation architecture that will be increasingly required to support the rapidly growing number of internet-scale business services that need real-time responsiveness.

What the various alternative approaches that we have considered here, and others, lack today is the key capability of a general purpose executable policy definition and enforcement technology that can provide the real-time command and control across large scale server pools that is required.

21 CONFIDENTIAL

CASE STUDY: GLOBAL FINANCIAL SERVICES COMPANY

The Sychron policy-driven orchestrator is currently being evaluated by a major global financial services company as a potential core component in a new large scale initiative by the company, the “Capacity-on-Demand / Auto-Provisioning Project”. The main goals of the project are to deliver major Total Cost of Ownership reductions through:

• Simplified lifecycle administration and management of servers, storage and applications.

• Real-time on-demand resource allocation across shared enterprise-wide server pools, to minimize over-provisioning and achieve peak levels of hardware utilization.

The hardware approach to be used in the initiative is (a) to deploy services on a commodity scale-out architecture comprising pools of HP ProLiant BL server blades, and (b) to consolidate storage on SANs. Through the blade approach, with integrated switches and on-board management, it is anticipated that there will be major TCO reductions through simplified administration, cabling etc., and through reduced power consumption, real estate etc.

The following quote from the company’s project blueprint (Sept 2003) summarizes the expectations “We foresee the ability to utilize this technology (blade pools) within an application farm, coupled with BigIP (F5 load balancer), SAN storage, an auto provision solution and a Capacity on Demand solution, all layered together to provide a highly scalable, manageable capacity on demand infrastructure”

The HP/Altiris ProLiant Essentials Rapid Deployment Pack will provide the auto provision tools required to move a bare metal server blade online with the required initial storage, application code and BigIP configurations, and to quickly and easily adapt software configurations to changing business demands using GUI drag and drop tools. These various tools, and associated scripting technologies, will use industry standard mechanisms such as PXE Preboot eXecution Environment and Wake-On-LAN to remotely deploy images.

As the cost of diskless blade systems falls, and data storage is increasingly consolidated and virtualized in networked storage systems, the goal of the project is to extend out from the automated on-demand provisioning of server resources, to incorporate flexible and automated on-demand storage allocation.

The following excerpts from the project blueprint summarize the core of the capacity-on-demand requirements. “In a capacity-on-demand model, these items (servers, load balancers etc.) need to be automatically manipulated to bring computing resources online………………The monitor agent is responsible for monitoring the computing resources on a continual basis to determine if it is being over taxed. Upon detection of the event it will trigger the necessary steps to add resource to the application for whatever period needed. Upon resource requirements dropping below a predefined threshold it will trigger the steps necessary to remove the excess added capacity from the application……………………Triggered by the monitor agent, the load balancer’s configuration will be modified to add the provisioned server resource to its pool.”

22 CONFIDENTIAL

The high-level capacity-on-demand capabilities that the project blueprint specifies, include the following:

• Proactively manage resources intelligently and automatically

• Eliminate over-provisioning

• Lower data center TCO while improving ROI on hardware investments

• Be able to offer differentiated levels of service tailored to the needs of diverse internal requirements

• Automate labor intensive tasks, saving personnel time while concurrently reducing downtime

• Implement a utility model enabling data centers to charge customers based on resource usage

• Create service level objectives and protect them from violation

• Integrate seamlessly into existing data centers

These are then refined into a series of more specific functionality requirements, including:

• Recognize various aggregated utilization levels and proactively take action to allocate the appropriate resource at the right time

• Provide differentiated levels of service, supporting business-driven IT objectives

• Adjust the amount of resources available to any application based upon predefined SLA requirements

• Trigger all Altiris provisioning events

• Interoperate with the OpenView monitoring solution by passing event information to he console

• Fully interoperate with F5 BigIP load balancers

• Provide usage based reporting

The fact that this set of requirements, and the design of the Sychron orchestrator, was generated entirely independently is remarkable, as the requirements capture almost all of the key and distinctive solutions that the Sychron product uniquely provides today.

23 CONFIDENTIAL

CONCLUSION

Sychron’s policy language and highly scalable policy-driven orchestrator provide a major leap forward in delivering an agile, real-time infrastructure. Going far beyond today’s complex and time consuming scripting-based approaches to data center automation, they allow both simple and very complex policy conditions to be easily defined and automatically enforced.

As this powerful and completely new technology is deployed in a variety of industries and a variety of environments, we will begin to gain much deeper insights into how people learn to use high-level policy-based orchestration most effectively. As we do so, these insights will be incorporated into a whole new generation of policy wizards, templates and modeling tools that will bring even the most complex forms of goal-oriented policy-driven automatic resource orchestration within reach of all data center operations staff, enabling them to get their IT in sync with business at all times.

Copyright © 2004 Sychron Inc. All rights reserved. Sychron, Sychron™ and the Sychron logo are trademarks of Sychron Inc. Any other company and product names used herein may be the trademarks or registered trademarks of their respective companies.