rpath wp visible ops

8/6/2019 rPath WP Visible Ops

1/15

Achieving Visible Ops

with Intelligent System Automation


2/15


I n t r o d u c t i o nModern IT is in crisis. Compounding growth in system scale, accelerated change,

and the expectation for zero-latency responsiveness create the perfect storm

particularly with the added backdrop of contracting budgets and lean, do-more-

with-less mandates.

This calls for improved IT processes and reliable automation, where the goal is

massively leveraged human capital and fast, predicable IT operations.

Most IT organizations have begun the journey to process improvement and IT

automation, and many have achieved meaningful resultsbut processes remain

flawed, and the inherent limitations in automation technology have prevented areal transformation. Many approaches to automation cause the wrong things to

happenfaster.

Process frameworks like ITIL and CobiT provide best practices for defining IT

processes, but they lack prescriptive guidance for how to implement them.

Manifestos such as The Visible Ops Handbookgo a step further to offer practical

guidance and a prescriptive roadmap for implementing release management and

change processes, but stop short of discussing automation techniques.

This whitepaper focuses on the last lap of the process improvement journey

intelligently automating release and change processes to transform and accelerate

IT, reduce cost and mitigate risk.

More specifically, this whitepaper focuses on intelligently automating Visible Ops.

It briefly summarizes the Visible Ops methodology, and explains how rPath can

automate key elements of the Visible Ops approach.

The Visible Ops Handbookanswers a fundamental questionhow do I implement

ITIL and where do I start? rPath answers an equally important and fundamental

questionhow do I implement ITIL principles without the bloat and bureaucracy

often associated with this approach? How do I automate for speed and c ontrol?

For many, ITIL brings to mind slow and ponderous review boards and layers upon

layers of process and approval. That doesnt need to be the case.


3/15


F i g u r e 1 : The IT process maturity model.

I n t r o d u c i n g V i s i b l e O p sThe IT Infrastructure Library (ITIL) is a framework for implementing processes and

best practices into IT operations; Visible Ops is a methodology for adopting ITIL.

The Visible Ops team met with hundreds of IT organizations and identified eight

high-performing IT operations groups with the highest service levels, best security

and best efficiencies. These organizations shared a c ulture of change

management, a culture of causality and a culture that valued effective and

auditable controls, promoting fact-based management.

Published by the IT Process Institute, The Visible Ops Handbook summarizes how

these organizations operate and provides step-by-step guidance and a roadmap for

implementing ITIL practices. It describes a control-based on-ramp to ITIL that

others can leverage to springboard their own process improvement efforts.

The findings are distilled into four practical and auditable steps:

1. S t a b i l i z e P a t i e n t , M o d i f y F i r s t R e s p o n s e Since almost 80% ofoutages are self-inflicted, the first step is to control r isky changes and

reduce mean time to repair (MTTR) by addressing how changes are

managed and how problems are resolved.

2. C a t c h a n d R e l e a s e , F i n d F r a g i l e A r t i f a c t s Infrastructure is highlydiverse and cannot be repeatedly replicated. This step focuses on creating

an inventory of assets, configurations and services, to identify those with

the lowest change success rates, highest MTTR and highest business

downtime costs.

T h e I T p r o c e s s m a t u r i t y m o d e l :A w a r e ITIL-aware, beginning to define

repeatable and verifiable IT processes

O p t i m i z e d Leveraging Visible Ops toimplement and optimize ITIL release,control and resolution processes

A u t o m a t e d Intelligently automatingkey elements of Visible Ops for fast,predictable and scalable IT operations


4/15


3. E s t a b l i s h R e p e a t a b l e B u i l d L i b r a r y The highest return oninvestment is implementing effective release management processes. Thisstep creates repeatable builds for the most critical assets and services, to

make it cheaper to rebuild than to repair.

4. E n a b l e C o n t i n u o u s I m p r o v e m e n t The previous steps haveprogressively built a closed-loop between the Release, Control and

Resolution processes. This step implements metrics to allow continuous

improvement of all of these process areas, to best ensure that business

objectives are met.

For more information on Visible Ops, visit: http://www.itpi.org/home/visibleops.php

I n t e l l i g e n t S y s t e m A u t o m a t i o n M a k i n g V i s i b l e O p s a R e a l i t yIntelligent system automation can be used to codify these principles as automated

steps in your release, change and resolution processes.

Today, provisioning and updating software systems is more challenging than ever.

Software is increasingly diverse, change is accelerating, and the volume of

systems is skyrocketing. This means that provisioning a new system takes weeks

or longer, change is avoided because updates and patches cause outages, and

troubleshooting systems and reversing changes is manual and time-consuming.

Visible Ops is a way to start taking the time, cost and risk out of system

provisioning and change, but the linchpin is automationits the only way to make

Visible Ops a reality in the age of massive scale and low-latency IT, where adding

resources is no longer an option.

Recognized as a leader and innovator in next-generation system automation, rPath

automates the packaging, provisioning and maintenance of software systems

across physical, virtual and c loud-based environments. This helps IT organizations

to :

Accelerate deployment cycles from months or weeks to minutes Reduce costs and improve system-to-administrator ratios by 6-10X Eliminate the pain of change with conflict-free and fully reversible updates


5/15


There are two key aspects of rPath that make it unique.

D e e p S y s t e m M o d e l i n gThe system model is about creating a blueprint for how systems should look and

using that as the basis for constructing and maintaining systems over time. rPath

automatically analyzes and deeply models entire software stacksfrom the

application through the OSand every layer in between.

The result is a deeply modeled system inventory, which describes the desired

state of every file, binary, application component, and software stack on every

production system, including information about policies that must be adhered to,

the entire dependency chainincluding OS components, middleware andlibrariesand the impact of change.

These models are stored as version-controlled system manifeststhey provide

deep system transparency, and serve as a basis for conflict-free deployments,

updates and managing the complete lifecycle of deployed systems.

F i g u r e 2 : The version-controlled system model provides deep system transparency andserves as the basis for managing change.


6/15


S y s t e m V e r s i o n C o n t r o lSystem version control is about describing systems over timeeasily roll the

system forward, back or reproduce itexactlyacross release lifecycle stages.

rPath is not a source code management systemits an operational management

platform that applies the principles and disciplines of source code control to the

management of deployable software systemssystem manifests, packages,

binaries, policies and configurations.

System version control is the technological foundation that allows a complex

system to be defined by a single version number, ensuring systems can be quickly

reproduced, patched and updated, rolled back and r eported on.

F i g u r e 3 : rPath provides a platform for modeling and versioning entire software stacks,including policies, dependencies, configurations, etc.

rPath provides the perfect foundation for automating Visible Ops.

What follows is a brief synopsis of each of the four Visible Ops steps, and a

description of how rPath can enable and automate those steps.


7/15


P h a s e O n e : S t a b i l i z e P a t i e n t , M o d i f y F i r s t R e s p o n s eFor enterprise IT, change is the only constant. And today, change is more disruptive

than ever. There are more sources of change, more systems to maintain, and faster

rates of change. According to industry analysts, the average IT enterprise suffers

multiple major system outages per year. 80% of the time, the outage is self-

inflicted1a direct result of change gone awry.

System outages are crippling and expensive. They lead to lost business, missed

service-level agreement (SLA) and compliance risks, as well as excessive labor

costs due to all-nighters and expensive overtime.

When systems are down, 80% of the effort is spent detec ting what changed, andonly 20% of the recovery time is spent act ually repairing the infrastructure. Blame

is readily assigned, fingers are freely pointed, and tension escalates, because no

one takes accountability for changes. When changes are finally detected, the

source of the change and the reason for the change are not always known.

When system failures happen during critical, peak and high-demand periods, the

consequences can be catastrophic. Many retailers have suffered major outages

just days before the year-end holiday, or during a peak month.

One notable example of the cost of IT change is Canada Revenue Agencythey

applied a certified patch that resulted in an eleven-day outage. All employees were

sent home, it took ten days to manually troubleshoot and restore the system, and

the associated cost was $2.4 million!

Sometimes changes wreak havoc later, long after the change was made. And

sometimes, a change can undo a previous change or even a whole series of

changes. For many IT organizations, only 70% of the changes made work the first

time without generating a firefighting episode.

All of this points to one overarching symptom: Todays IT organizations are

grappling with an inability to deal with change at scale.

rPath provides a next-generation approach to dealing with IT change. Based on a

version-controlled system model as the basis for managing change, rPath allows IT

to consume more rapid and frequent change in a way that is scalable and non-

disruptive.

1 Source: Stephen Elliot, Senior Analyst Network and Service Management, IDC, 2004.


8/15


The goal of the first Visible Ops phase is two-fold: 1) reduce the number of outages

by freezing change outside of scheduled maintenance windows, and 2) modify thefirst response process by ensuring that problem managers have all change

information at hand about what could have caused the outage.

rPath can enable and automate the specific steps outlined in Phase One:

E l i m i n a t e u n a u t h o r i z e d c h a n g eThe first phase is devoted tobuilding and reinforcing a functional change management and

authorization process. This calls for change automation. rPath automates

and controls change, helping to identify, isolate, and ultimately eliminate

unauthorized change.

E s t a b l i s h a c u l t u r e o f c a u s a l i t y When service outages occur, VisibleOps recommends examining all approved and detected changes first before

making a diagnosis. In addition to tracking changes to software artifacts

managed in the rPath repository, rPath also tracks all change on a system-

by-system basis. This allows for a complete change timeline for any

individual system, making it very easy to identify recent changes that c ould

have caused the outage.

E n s u r e b a c k o u t p l a n s f o r c h a n g e s System version control makesit very easy to roll back changes. Reversing a change is more than simply

reverting to the previous version of the system. rPath maintains a record of

all overwritten files, so when a change is reversed, previous files are

restored, even if those files were not modeled in rPath (e.g., locally-edited

configuration files).

S t r i v e f o r c h a n g e s u c c e s s r a t e s h i g h e r t h a n 9 8 %A version-controlled system model makes change predictable and scalable. All

changes are pre-validatedrPath analyzes imported software artifacts and

understands the complete dependency chain. This ensures guaranteed,

conflict-free change, and enables 100% change success rates.

C r e a t e a C h a n g e A d v i s o r y B o a r d ( C A B ) rPath can help answerquestions before and during the CAB meeting, including What questions,

such as, what are the expected results of the c hange? and, What If

questions, such as, what is the rollback plan should the change fail?

Integrations can auto-pull information from rPath into CAB tickets.


9/15


R e s o l v e l a t e n t p r o b l e m s rPaths complete and versioned changehistory is ideal for troubleshooting latent problems. System version controlcan precisely reproduce any previous version of an entire system t o help

isolate the exact change that caused the problem.

P r e v e n t p r e v i o u s c h a n g e s f r o m b e i n g u n d o n eA change can oftencause problems by inadvertently erasing previous changes. rPaths model-

driven change management prevents this. All changes are pre-validated,

and the impact of change is deeply understood.

P h a s e T w o : C a t c h & R e l e a s e a n d F i n d F r a g i l e A r t i f a c t sUncontrolled changes on systems cause them to deviate from known and trustedstates. Instead of parity, variance creeps in. A thousand servers may be identical at

deployment, but when their configurations drift over time, t hey become like

snowflakeseach unique and impossible to reproduce.

The goal of the second Visible Ops phase is to identify and analyze the critical

elements of production infrastructure and the interdependencies between

components that make up that infrastructure. This insight will be used to reduc e

configuration variance.

rPath automates this process by creating a system model. Software artifacts are

imported into the rPath repository and deeply modeled. Dependencies are

automatically discovered and resolved.

rPath is essentially a modeled system inventory that describes all of the

operational software assets within a running stack. This includes files, binaries, OS,

middleware and application components, configurations, policies and the entire

dependency chain. All of this content is managed under strong version control.

rPath can enable and automate the specific steps outlined in Phase Two:

I n v e n t o r y a l l m a n a g e d a s s e t s rPath automates the inventoryconstruction process by examining existing software and constructing

initial system models. This is the first step in moving an existing

environment along the path to standardization.

M i n i m i z e c o n f i g u r a t i o n v a r i a n c e a n d n o r m a l i z e t h ei n f r a s t r u c t u r erPaths whole-system differencing helps to normalizeunique, snowflake systems. Sometimes the differences between

systems are arbitrary and incorrectin this case, rPath can b e used to


10/15


automatically correct them. Sometimes the differences are correct, but

theyre not modeled and therefore not reproduciblein t his case, rPathcan be used to explicitly describe the desired differences between a base

system model and particular system instances.

P h a s e T h r e e : E s t a b l i s h R e p e a t a b l e B u i l d L i b r a r yModern software systems are complex, interdependent and dynamic, comprising

tens of thousands of files, thousands of configuration options, and countless file

versions. These systems are often hand-assembled in a manual, haphazard

manner, without a system model or blueprint.

This lack of transparency makes it very difficult to ensure that all system artifactsare consistently synchronized, and that all stakeholders across the release

lifecycle are drawing from the same definitive source, and working with the most

current software.

A common scenario may look something like this: Developers build an application

on an IT-issued platform. They hand it off to QA, who recreates the system based on

a different version of the platformthey resolve dependencies, and tweak and

tune the system until it works. They then pass the cert ified system to the

production teams who recreate the system, again, based on the latest version of

the platform.

During each of these handoffs, software versions and system configurations drift.

Dependencies are missed, found, resolved, poorly documented and, consequently,

missed again. Deployments fail, and the system goes back to development and

test for hot fixes and recertification. Once the system is finally in product ion again,

the software usedsources, versions and dependenciesremains unclear.

Updating and patching the system becomes an ordeal.

This problem is so acute that it has inspired a movement called D evOps. Its mission

is to bridge the gap between application development and IT operations, and tocreate an environment in which development and IT are collaborative, agile and

high functioning.

The goal of Phase Three of Visible Ops is to create and maintain a versioned

definitive software library (DSL) for all software and patches, and to ensure per fect

configuration synchronization between pre-production and production systems.


11/15


rPath combines deep system modeling and version control as a basis for managing

a definitive software librarya canonical source of record for operational softwareassets.

This enables a very differentand improvedscenario: Development builds a new

application using the latest platform. They check in a complete, dependency

resolved systemapplication, OS and middleware componentswhich is identified

by its own version number. QA is alerted and uses the same version to run their test

suite. Production then takes the certified version and successfully provisions the

system.

Development, test and production are all accessing the same exact version,

eliminating drift and confusion. And once a system is deployed, a version manifest

becomes the basis for seamless change. Updates can be quickly matched to

system inventories, change impact is clearly document, and patches and updates

can be implemented incrementally.

rPath can enable and automate the specific steps outlined in Phase Three:

R e b u i l d i n f r a s t r u c t u r e r a t h e r t h a n r e p a i r Repeated break/fixcycles tend to add variance, whereas rebuilding infrastructure minimizes

configuration drift. rPath uses the same version-controlled system

model to drive provisioning, image creation, and incremental updates. This

allows for flexibility in deciding whether to build or updatein the case of

build, the result is immediately up-to-date with ev ery change across the

stack. Traditionally provisioned and imaged systems are instantly out-of-

date because change is constant. And since rPath rebuilds a system from

its constituent components, a build can be retargeted t o a different

physical, virtual, or cloud environment without re-developing the system.

C r e a t e a n d m a i n t a i n t h e d e f i n i t i v e s o f t w a r e l i b ra r y ( D S L )rPathprovides a solid foundation for managing a versioned software library for IT

operations. rPath extends the DSL concept by modeling and versioning

entire stacks, including OS components and configuration files.

M o v e f r o m p r o d u c t i o n a c c e p t a n c e t o d e p l o y m e n t rPath enablesrapid, on-demand, conflict-free provisioning of complete, dependency-

resolved systems for physical, virtual and cloud targets. Deployment is

fast and conflict-free.


12/15


S t o r e n o n - r e p e a t a b l e s n o w f l a k e b u i ld s i n t h e D S L a n d s l o w l ym i g r a t e Standardization is a gradual process of adoption, and bothVisible Ops and rPath accommodate this reality. rPath supports the process

by automatically converting existing systems into initial system models.

The systems can then be standardized incrementally in successive

versions.

C l o s e t h e l o o p b e t w e e n p r o d u c t i o n a n d p r e - p r o d u c t i o nTherPath approach ensures a controlled software release lifecycle with clean

separation of duties and consistent, access-controlled handoffs between

lifecycle phases. This includes explicit functionality for promoting content

from one stage to another, and enforcing access or lack of access todifferent user populations at different stages.

F i g u r e 4 : rPath enables consistent and controlled handoff between lifecycle stages, includingexplicit functionality for promotion and access-control.


13/15


P h a s e F o u r : E n a b l e C o n t i n u o u s I m p r o v e m e n tThe previous Visible Ops steps focused on building a closed loop between the

release, control and resolution processes. This step implements metrics to allow

continuous improvement of all of these process areas.

ReleaseHow efficiently and effectively do we generate and provisioninfrastructure?

ControlsHow effectively do we make good change decisions that keepproduction infrastructure available, predictable and secure?

ResolutionWhen things go wrong, how effectively do we diagnose andresolve issues?

rPaths unique approach to system modeling creates the level of transparency and

insight needed to answer these fundamental questions, to enable ongoing process

improvement, and to satisfy audit and reporting requirements.

rPath can inform and automate the specific metrics and improvement points

outlined in Phase Four:

R e l e a s e m e t r i c s rPath provides deep system transparency andcontrol, and can inform key release metrics, including time to provisionknown good builds, number of turns to a known good build, shelf life of

builds, percent of systems that match known good builds, percent of

builds that have a security sign-off, and number of fast-tracked builds.

C o n t r o l m e t r i c s rPath can also inform key control metrics such asnumber of changes authorized per week, number of actual changes made

per week, number of unauthorized changes, change success rate, number

of emergency changes, number of special changes, and number of

business as usual changes.

R e l e a s e i m p r o v e m e n t p o i n t s rPath can automate suggested releaseprocess improvements, including tracking all configurations in use across

the release lifecycle, segregating development/test/production systems,

creating a library of automated build systems, and confirming that all

deployed system images are under version management.

R e s o l u t i o n i m p r o v e m e n t p o i n t s rPaths complete visibility intochange enables ongoing improvement of diagnosis processes. rPath also


14/15


helps define bulletproof back-out processes to recover from failed or

unauthorized changes.

A v o i d a u t o m a t i n g b r o k e n p r o c e s s e s Visible Ops cautions againstautomating a flawed process. This only makes bad things happen faster.

Rather than simply using custom scripts to automate manual tasks, rPath

uses intelligent automation to make release management and change

processes predictable and scalable.

P r e p a r i n g f o r A u d i t s w i t h r P a t hVisible Ops provides a number of helpful tips when preparing for an audit. rPath

provides an automated model for creating and deploying software systems that aretransparent, easily controlled and easily audited. rPaths reporting capabilities help

answer key questions about the state of deployed systems, helping to take the

time, cost and risk out of audit preparations.

For example, rPath can help document and report on systems and processes used

to detect changes. rPath can also help with documenting and proving that an

accurate system inventory is being maintained. Finally, rPath can help generate

reports on change success rates, deployed systems and whether or not t hey match

golden builds, and integration with change management processes.

But compliance is more than something rPath checks after the factcompliance is

inherent in the rPath model. The rPath approach to compliance is simple, powerful

and uniquea system is compliant if it matches its system model. The model is

powerful enough to capture all automatable aspects of audit compliance.

This is a much deeper compliance model than usual. It puts an end to the endless

drift/audit/repair cycle, since compliance rules are applied for every incremental

change. And because no change is allowed to get the system out of compliance,

theres the assurance that an audit will pass on the first t ry.


15/15


C o n c l u s i o nGrowth in system scale, accelerated change, and the expectation for zero-latency

IT are forcing a transformation in enterprise IT. This creates the need for improved

release management and change processes, and next-generation automation

technologies.

The ITIL release management process is fundamentally about controlling and

monitoring the flow of change into an IT infrastructure. The Visible Ops Handbook

offers practical guidance and a prescriptive roadmap for implementing the release

management, control and resolution processes. But in todays cost-conscious and

constrained IT climate, the only way to make Visible Ops a reality is with reliable,

intelligent automation.

rPath delivers an automated, next-generation model for creating, deploying and

changing software systems. The result is dramatic improvements in

responsiveness and a way to deal with exploding scale without adding cost.

This makes ITIL and Visible Ops achievablewhile remaining responsive, agile and

lean.

A b o u t r P a t h rPath automates system provisioning and maintenance across physical, virtual and

cloud environments. rPaths innovative release automation platform is based on the

industrys only commercial version control repository for managing deployed

software systems. The result is an easy-to-deploy and cost-effective automation

solution for rapid, low-risk and low-overhead deployment and maintenance of

complex software systems. rPath dramatically improves responsiveness to

business lines, reduces compliance risks, and allows resource-constrained IT

organizations to significantly reduce operating costs and do more with less.

Headquartered in Raleigh, NC, rPath has 80+ customers including some of the

worlds largest enterprises and ISVs. Visit www.rpath.com.

Corporate Headquarters:

701 Corporate Center Drive, Suite 450

Raleigh, NC 27607

+1 919.851.3984 Main

+1 866.508.6200 Sales

+1 919.851.3985 Fax

[email protected]

www.rpath.com

Copyright 2010 rPath, Inc. All rights reserved. rPath, rBuilder, rPath Lifecycle Management Platform and the rPath logo are registered trademarks or trademarks of rPath, Inc. All other brands and product names are trademarks or registered

rpath wp visible ops

Documents