lean data center, a telco experience by andrea pinnola, telecom italia

Copyright © Institut Lean France 2012

3 & 4 October, 2013 Paris, France

Lean Data Center: A Telco Experience

Andrea Pinnola - Telecom Italia

European Lean IT Summit 2013


• TI Background

• Bologna Data Center Operations Pilot

• Extending to the Delivery Value Streams

• Lessons Learned

Agenda

Lean Data Center: A Telco Experience - Andrea Pinnola 2


Telecom Italia is the leading Italian Telecommunications and ICT

provider, one of the most important Telcos players in LATIN

AMERICA and one of the Top Ten global telcos players

As of March 31th, 2013

Telecom Italia Group Figures and

Organization (*)



• 7 National Data Center Sites

• 4 Local Service Centers

• 24.000 Servers

• 21.000 TB Storage

• 20.000 TB Backup

• 12.000 TB Backup

Transferred/month

Domestic IT Infrastructure

4 Lean Data Center: A Telco Experience - Andrea Pinnola

Organization

Engineering & Delivery coordination

plus 3 Data Centers Areas (North

East, North West & Center South) Local Service

Centers

National Data

Center Sites

IDC Cesano Maderno

IDC Rozzano

DC Bologna

DC Padova

DC Bari

IDC Pomezia

Palermo

Firenze

Torino

Napoli

DC Oriolo Romano

Milano

(Virtual

Data Center)

2,5 Gbps

155 Mbps

VDCN Network


The case

• Difficulty to answer to all the incoming requests

• Feeling overwhelmed

• Not clear work loads

• Customers complains

Lean DC Bologna Pilot


Focus on Operations activities: Reducing

Problems and Improve Effectiveness


• Data Management: Storage

Backup

• Systems: Windows e Linux

UNIX

• Networking

• Plant & Logistics

Bologna Data Center Delivery &

Operations Teams


Sponsorship from the Head of Infrastructure

Head of DC Bologna willing to improve


The Improvement Cycle

Process

Identification

Activity

Tracing

Data

Analysis

Improvement

Identification

and Proposal

Improvement

Execution and

Check


Make Process & Problems Visible:

e.g. Management of Backup Policies


Over 1200 Backup Policies for Systems and Applications

Problems Request

Documentation

Rework

Requests to know

Policies in place

No access to the

Policies DB

Difficult Password

Retrieval

OOPS … No Data ? Problem!

Make process visible: tracking requests

Operation requests Tracking


Track every Backup Policy

Request received by Mail, IT

System, Phone call, coffee chat …

From 7-8 main activities / month

to 31 activities tracked / week !

0

2

4

6

8

10

12

14

Fault

Modific

a

SW

Info

rmazio

ni

Impl. P

olicy

Nu

mero

sit

à

0%

20%

40%

60%

80%

100%

120%

Perc

en

tuale

cu

mu

lata

Review of backup controls in order

to prevent faults incidence

Make process visible: tracking problems


• Backup Faults: highlighted

recurring errors on 4 drives

• New (type of) Fault not

described within the Fault

Management procedures

Disk Drivers Problems


Gemba Walk: Critical cabling


In one room:

• Longer cable plug in

• Old cables not removed

• Difficulty and risk of faults

during interventions

Why ?

• Continue escalating requests of

new connections done in a

hurry


Flow of improvements proposals


0

10

20

30

40

0

10

20

30

40

• All teams involved in a 5 months

timeframe

• Created one sharing point for

problems, countermeasures

proposal and approval

• Weekly review meeting with the

head of DC, Team Leaders and

Bologna Facilitator

106

By Team

By Type


Actions and Results (Examples)

Actions Results

• Storage size controls contemporary to storage assignment

• Effectiveness of storage assignment and reduced monthly reporting (4d to 2h)

• Semi-automated procedure to OS patching

• Reduction of OS patch time and of global delivery time of an updated server

• Simplification of systems /servers password management

• Eliminated password renewals delay • Reduced time to interventions

• Backup policy requests simplified templates

• Reliable policies update and reduction of info request

• Enriched scripts to better signaling backup failures

• Reduced errors and increased reliable operations

People feeling change and the effects of

their improvements

13


Results: Critical cabling recovery


• Clean situation restored

• New interventions simplified

• Recovered fibers for 1 year

of consumption (variations)


• Lean DCs Steering Committee

• Biweekly meetings to share improvements

• 3 Lean DCs Facilitators appointed

• Involved all the DCs Team Leaders

• … move to a wider Lean initiative and joint DCs

responsibility

Moving Forward: involving the other DCs



• Motivation

Growing number of installations & requests

Long time to deliver

Need for better delivered quality

• Three Cases on:

Infrastructure Creation

Virtual Machines Delivery

Market Infrastructure Delivery

Moving Forward: the Delivery case


Workshops on Lean Basics and VSM


Teamworking & Common Knowledge: the

actual Value in Value Stream Mapping


Infrastructure

Creation

Virtual

Infrastructure

Delivery to External

Market

Main Goals: Lead Time Reduction and reliable end date

Rome


5S in physical spaces


Before

• Time spent to look for materials

• Disordered Workspace

• No space to act

• Access forbidden to the customer

After

• Phase out of obsolete and not used

• Space to mount racks & shelfs

• Visual control of quantities

• Better Customer perception

5S on IT Shared Servers

• - 50% of the “space quota” assigned to the Shared Server

• - 83% of documents space, complete and accurate

information mantained



OPL: standardization and knowledge transfer

People able to manage the application since their first access



•Physical Flow

•Information Flow

• Organization ‘Flow’

Lead Time reduced by 15%

Effort reduced by 5%

Results: Visual Mng & Server Farm Creation


SERVICE REQUEST BACKLOG TREND SERVICE REQUEST AGING TREND

Results: Virtual Server Delivery

Improvement

Lead Time - 40%

Savings 4 hours for each activity

Quality - 50 % reworks on docs


Results: External Market


Improvement

Lead Time - 50%

Operation Tests Definition - 50% Elapsed Time

Rework on project docs - 75% Effort Spent

Organization One single person as DC reference assigned for each delivery


• Make Process and problems visible

• Adapt Lean to IT (e.g. 5S in physical and ‘virtual’ spaces)

• 80% of the knowledge already there (people are experts)

• Act readily on problems to get people involvement

• Management commitment from the start

• Customers involvement after a while

Lessons Learned



• To all the colleagues of the IT Infrastructure

Department that initiated and participated to the

Lean DC initiative

Acknowledgements




[email protected]


More Lean IT videos and presentations on www.lean-it-summit.com

lean data center, a telco experience by andrea pinnola, telecom italia

Technology