it automation principles, best practices, and lessons learned
DESCRIPTION
Technological and structural changes occurring in the industry such as virtualization, outsourcing, acquisitions and off-shoring make the maintenance of the information required to effectively manage and maintain the link between high level business processes and the underlying infrastructure that supports them increasingly complex particularly in environments that have a high rate of change. Level 3, a leading provider of global voice, video, and data networking services is solving this problem through the application of state-of-the art process automation and auto-discovery technologies. In this discussion, key technologies being applied to the problem of IT automation and lessons learned from Level 3’s experience will be presented.TRANSCRIPT
1 ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
IT automation principles, best practices, and lessons learned
Ron Cotten – Sr. Manager of OSS Engineering for Global Technology Operations, Level-3 Communications
Kristin Brennan – Product Marketing, HP Software & Solutions
22 3 June 2010
BSM
Monitoring events & alerts
Client Automation
NetworkAutomation
ServerAutomation
Storage Essentials
ITSM
Help desk/ change requests
CMDB
Operations Orchestration
BSA Essentials Reporting
Community Network Subscription Services
Service-centric, integrated automation suite
HP Business Service Automation
©2009 HP Confidential3
Polling question
–What’s driving you to consider automation:• Improving labor efficiency
• Reduce compliance risk
• Improved application release management
• Automated incident remediation
• Improving change and configuration management
©2009 HP Confidential4
Polling question
–Where do you plan to start? • Servers
• Network devices
• IT Processes
• Monitoring + Incident Management
• Servers + Process Automation
• Compliance management across servers + network devices
© Level 3 Communications, LLC. All Rights Reserved. Level 3, Level 3 Communications and the Level 3 Communications Logo are either registered service marks or service marks of Level 3 Communications, LLC and/or one of its Affiliates in the United States and/or other
countries. Level 3 services are provided by wholly owned subsidiaries of Level 3 Communications, Inc. Any other service names, product names, company names or logos included herein are the trademarks or service marks of their respective owners.
IT Automation Principles, Best
Practices and Lessons Learned
Ron Cotten
Senior Manager IT OSS Engineering
Global Technology Operations
Level 3 Communications
Office: 720-888-5842
Email: [email protected]
April 6, 2010
66Level 3 Communications, LLC. All Rights Reserved.
Agenda
About Level 3
Large Scale IT Operational Environments
Common Operational Problems in Many Large Scale IT Environments
The Level 3 IT OSS Engineering Charter
Principles of IT Automation (How it Works)
Best Practices and Lessons Learned at Level 3
Q & A
77Level 3 Communications, LLC. All Rights Reserved.
About Level 3
As a leading international provider of fiber-based
communications services, Level 3 is dedicated to helping our
customers keep pace with the demands of an increasingly
networked world.
Level 3 couples a broad service portfolio with one of the
world’s most scalable end-to-end networks to deliver a set of
solutions built for the 21st century. Our network offerings
include Internet Protocol (IP) services, metro and longhaul
transport, content and video delivery as well as data and
voice services.
Level 3’s customers include:
18 of the world’s top 20 telecommunications service providers
9 of the top ten largest telecommunications service providers in Europe
3 of the top 4 telecommunications service providers in Asia
9 of the top 10 U.S. Internet Service Providers (ISPs)
9 of the top 10 U.S. cable television companies
The top 5 U.S. wireless service providers
88Level 3 Communications, LLC. All Rights Reserved.
Large Scale IT Operational Environments
Geographically distributed data centers
Thousands of physical servers from multiple vendors
Multiple OS variants and versions deployed
Increasing use of virtualization to maximize infrastructure
utilization and minimize cost
Thousands of COTS and custom applications deployed
Hundreds of mission critical business services supported
Rapid set up, provisioning and tear down of environments
required to keep pace with business
Legacy infrastructure and applications inherited through
acquisition
99Level 3 Communications, LLC. All Rights Reserved.
Common Operational Problems in Many Large Scale IT Environments
Large numbers of applications and business services deployed with complex or unknown interdependencies
Critical information about IT infrastructure and applications historically maintained manually in multiple locations
and formats by many different people and organizations
High level of change in some environments makes maintenance of current state of operational environment
difficult to manage
Critical information required for operational support not always updated or accessible resulting in higher MTTR
and cost to business
Critical knowledge about applications and systems historically owned by people (SMEs)
Reductions in IT staff and off-shoring accelerates loss of knowledge about business critical applications and
systems exposing companies to significant risk
Lack of visibility into current application topologies and dependencies prevents effective implementation of
service impact management and service level management solutions for mission critical applications
1010Level 3 Communications, LLC. All Rights Reserved.
The Level 3 IT OSS Engineering Charter
To continually seek to improve operational efficiency
and integrity and to drive down Level 3 operating and
capital expenditures through IT process automation
To deliver technologies and solutions that promote
consistency throughout the Level 3 IT operational
environment
To integrate, consolidate and retire existing legacy IT
OSS and management systems using state-of-the-art
technologies culminating in a world class IT
configuration and service level management system
that will carry Level 3 IT forward into 2010 and beyond
1111Level 3 Communications, LLC. All Rights Reserved.
Principles of IT Automation
You can’t automate if you
don’t have the data.
A solid configuration
management system
provides information which
is essential for effective IT
automation.
Auto-discovery is a critical
component of configuration
management and IT
automation in large scale IT
environments.
1212Level 3 Communications, LLC. All Rights Reserved.
Principles of IT Automation
The maturity of your
configuration management
and auto-discovery
systems determines the
extent to which IT
automation can be
implemented and is the
critical limiting factor in
business value and ROI
that can be achieved
through automation.
A mature configuration
management system
based on auto-discovery
provides a solid
foundation for IT
operations automation.
1313Level 3 Communications, LLC. All Rights Reserved.
Principles of IT Automation
Automations that provide
the highest return on
investment to the business
require higher levels of
configuration management
data quality and
increasingly sophisticated
auto-discovery systems.
1414Level 3 Communications, LLC. All Rights Reserved.
Principles of Auto-Discovery (How It Works)
There are multiple levels
of auto-discovery maturity
each with differing levels
of complexity and value
add to the business.
Business ServicesComplex Application
Signatures
Process Connection Maps
Service Topologies
ApplicationsInstalled Applications
Installed Packages
Installed Patches
Running Processes and
Services
InfrastructureChassis Model
Processor Cores
Installed Memory
Attached Storage
Network Interfaces
Co
mp
lexity a
nd
Va
lue
Ad
dSignature for HP DDM OSS Service at Level 3
1515Level 3 Communications, LLC. All Rights Reserved.
Principles of Auto-Discovery (How It Works)
Signature for HP DDM OSS Service at Level 3 Applied to Enterprise Wide Scan of all Level 3 IT Infrastructure
DDM Probe
DDM Core
DDM Oracle DB
Client connections between DDM core and DDM Oracle DB instance
Client connections between DDM Probes and DDM Core
1616Level 3 Communications, LLC. All Rights Reserved.
Principles of Auto-Discovery (How It Works)
The list of systems comprising this service has changed. New items are in green, items that are expiring soon are shown in yellow,
and items which recently aged off are shown with strikeout:
Service: DDM
Environment: PROD
Discovery Mode: AUTO
App NameApp Version Server Appeared Last Seen
DDM Core (ddmcore02.prod.idc1.level3.com) ddmcore02.prod 9/17/2009 9:01:00 AM 10/5/2009 9:02:00 AM
DDM Probe(cdc1ddm0001) cdc1ddm0001 9/17/2009 9:01:00 AM 10/5/2009 9:02:00 AM
DDM Probe(vadcddm0001) vadcddm0001 10/5/2009 9:01:00 AM 10/5/2009 9:01:00 AM
DDM Probe(vausddm0001.brw.corvis) vausddm0001 9/17/2009 9:01:00 AM 10/5/2009 9:02:00 AM
DDM Probe(vidcddm0001) vidcddm0001 9/17/2009 9:01:00 AM 10/5/2009 9:02:00 AM
DDM Probe(vidcddm0002) vidcddm0002 9/17/2009 9:01:00 AM 10/5/2009 9:02:00 AM
DDM Probe(vldcddm0001) vldcddm0001 9/17/2009 9:01:00 AM 10/5/2009 9:02:00 AM
Oracle (ddmprd2) 9.2.0.5.0 f11v33-06 8/19/2009 8:10:00 AM 10/5/2009 9:02:00 AM
Please consult the CMDB or DDM for current service topology.
New DDM probe detected - Provisioned by Level 3 OSS Engineering on 10/4/2009
Topology Change Notification for HP DDM OSS Service from 10/5/2009
1717Level 3 Communications, LLC. All Rights Reserved.
Principles of IT Process Automation (How It Works)
HP Operations Orchestrator iterates through defined standards
and policies.
For each standard, a generic automation handler process
retrieves the list of CIs that the standard is to be applied to,
filtered by exclusions, exemptions and exceptions.
The automation handler then iterates through each CI in the
resulting list of CIs to be acted upon and:
1. Invokes a validation script through HP SA to determine if any action needs to
be taken.
2. Systems that pass validation are marked as compliant.
3. For systems that fail validation, correction scripts are invoked via HP SA to
remediate the exception.
4. Validation is re-run following corrective action and systems that pass are
marked as compliant.
5. Tickets are automatically opened by HP OO for systems that continue to fail
validation after remediation is attempted.
6. Once the ticket is closed, validation will be run again and passing systems
marked as compliant.
1818Level 3 Communications, LLC. All Rights Reserved.
Principles of IT Process Automation (How It Works)
1919Level 3 Communications, LLC. All Rights Reserved.
IT Process Automation Use Cases
OSS Credentials Provisioning
Administrative Credentials
Application Specific Credentials
Auto-Discovery Credentials
OSS Agent Provisioning
Opsware SA Agent Deployment
OVO Fault Monitoring Agent Deployment
Backup Agent Deployment
Performance Data Collection Agent Deployment
Configuration Management Data Integrity
New Chassis Detect Locate
Chassis Decom
New Virtual Detect
Service Topology Change Detect
2020Level 3 Communications, LLC. All Rights Reserved.
Best Practices and Lessons Learned at Level 3
Expect to invest 1 year of sustained effort deploying any given product in any OSS category
(configuration management, auto-discovery, process automation, etc)
Architecture
Deployment
Configuration
Integration
Patching
Upgrades
Optimization
Data Cleanup
Credentials
Expect to invest 1 year of sustained effort following deployment before realizing measurable gains in
operational efficiency
Content Development
Process Re-engineering
Staff Training
Adoption
2121Level 3 Communications, LLC. All Rights Reserved.
Best Practices and Lessons Learned at Level 3
Expect non-linear accelerating benefits to the business in year 3Operational Integrity
Reduced Ticket Volume
Reductions in MTTR
Optimization of Infrastructure
Improved Reporting
Improved Business Intelligence and Planning
Reduced OPEX and CAPEX
Return on Investment
OSS systems integration results in non-linear value add to business (i.e. the sum is greater than the
parts)
Level of difficulty is high and most programs fail to achieve expected results for many reasons including:Unrealistic expectations about time frames and ROI
Failure to view problems at a systems level (system of systems)
Failure to invest in data quality improvement
Failure to understand the need for auto-discovery
Failure to understand dependencies
No end state vision
Inability to garner sustained commitment at executive level
22
Questions
Thank you
– Get automated lifecycle management for your enterprise with
HP Business Service Automation software solutions
Learn more at: www.hp.com/go/BSA
– Learn more about how HP Business Service Automation can help you
manage virtualization: www.hp.com/go/BTOVirtualization
– Visit the HP Software Web Events portal to see all live
and on-demand web events: www.hp.com/go/hpsoftwarewebevents
24 ©2010 Hewlett-Packard Development Company, L.P.
To learn more on this topic, and to connect with your peers after
the conference, visit the HP Software Solutions Community:
www.hp.com/go/swcommunity
25