cisco unified computing and virtualization: architecture, design and deployment - best practices

Cisco Confidential © 2011 Cisco and/or its affiliates. All rights reserved. 1

Argentina, Chile, Mexico, Puerto Rico, Brasil May-Ago/2012

Cisco Confidential © 2011 Cisco and/or its affiliates. All rights reserved. 2

Top Ten Best Practices On Building a Data Center: From The Facilities Up to The Application Implementation to Cloud

Ricardo Trentin, Solutions Architect Jose Cervantes, Network Consulting Engineer Anibal Bustillo, Solutions Executive May/2012

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3

•  Experience acquired during the last 4 years deploying end-to-end complex data center projects in Latam for Cisco

•  Most critical pain points Unfortunately there is only one black terminal screen on this session Dozens of devices and only 60 min!

•  But you definitely can engage

us for your next project!


1.  Engage Everybody!

2.  Top Down Approach

3.  Use “Our” Technology in Your Favor

4.  Block to Conquer!

5.  The New Kid on the Block: DCI

6.  Your Apps Run Smoothly here!

7.  Align Logical and Physical Designs

8.  Check for Bugs and Other Insects

9.  Plan Your Data Center Implementation

10.  Test Your Data Center Implementation


•  Identify the Stakeholders Early On

•  Engage Vendors and Let Them Talk to Each Other

•  Create an Effective Communication Plan


•  Data Center projects should follow a direction: from bigger pay checks down to the technical pay checks

•  Jokes apart, a correct understand of business requirements is mandatory for a successful deployment (on time, on budget)

•  Those business requirements needs to be translated to a technical solution, a data center architecture (High Level Design)

•  Once a high level architecture is created, a list of material could be carved and shared with your company procurement staff

•  Experience shows higher costs when the approach is bottom-up


What does High Availability mean to your Data Center?

Tier 4 DC – Fault Tolerant: 99.995% §  Planned Activity does not disrupt critical load

and DC can sustain at least one worst-case unplanned event with no critical load impact

§  Multiple power and cooling distribution paths includes redundant

§  Annual Downtime = 24 minutes

Tier 3 DC – Fault Tolerant: 99.982% §  Enables planned activity without disrupting

computer hardware operation, but unplanned events will still cause disruption

§  Multiple power and cooling distribution paths but with only one path active, includes redundant components

§  Annual Downtime = 1 hour 36 minutes

Tier 2 DC – Fault Tolerant: 99.741% §  Less susceptible to disruption from both

planned and unplanned activity

§  Single path for power and cooling disruption

§  Maintenance of power path and other parts of the infrastructure require a processing shutdown

§  Annual downtime = 22.0 hours

Tier 1 DC – Fault Tolerant: 99.671% §  Susceptible to disruptions from both planned

and unplanned activity

§  Single path for power and cooling distribution, no redundant components

§  Must be shut down completely to perform preventive maintenance

§  Annual downtime = 1 day 4 hours


Sismos 2006-2012 Mayores de 4.5


•  Cisco developed several technologies to maximize the investment and mitigate the risk of complex deployments as a green field data center build out

•  Nexus 7000 (VDC, OTV, FCoE, VPC)

•  Nexus 5000 (FabricPath, VPC, FCoE,

Unified Port, NPV/NPIV)

•  Nexus 2000 (FEX, FCoE)

•  Nexus 1000v

•  MDS 9500 (VSAN, FCIP, FCOE,

NPV/NPIV)

•  UCS (UCSM, FI, NPV/NPIV, Service Profiles, FEX, FCoE, VICs)


•  Hierarchical Model

•  Domain Failures

•  Boundaries


•  Vblock Series 700

Storage: EMC Symmetrix Vmax

Compute: Cisco UCS

Virtualization: VMware

Orchestration: Unified Infrastructure Manager (UIM)

Vblock Series 700 model MX

•  Vblock Series 300

Storage: EMC VNX

Compute: Cisco UCS

Virtualization: VMware

Orchestration: Unified Infrastructure Manager (UIM)

Models

Vblock Series 300 model EX

Vblock Series 300 model FX

Vblock Series 300 model GX

Vblock Series 300 model HX

Pre-Engineered, Pre-Integrated and Validated


Receive Components

Staging Rack & Cable

Traditional Data Center Experience 90-120 Days From Order to Production

Program & Provision

VCE Data Center Experience ~30 Days From Order to Production


Cisco® UCS B-Series Blade Servers and UCS Manager

Cisco Nexus® Family Switches

NetApp® FAS

¡  Standard, prevalidated, best-in-class infrastructure building blocks

¡  Flexible: One platform scales to fit many environments and mixed workloads –  Add applications and workload –  Scale up and out

¡  Simplified management and repeatable deployments

¡  Design and sizing guides

¡  Services: Facilitate deployment of different environments


VM #4

VM #3

VM #2

Host Facing vPC

Nexus 1000v & vPATH VSG, vWAAS, vASA

Virtualized Interfaces Adapter FEX, VM-FEX & FCoE

VDC: Virtual Device Contexts

NX-OS – Modular Operating System common across the DC ISSU – True non-stop operations vPC – Between Nexus layers for bi-sectional bandwidht use (no STP loops) DCNM – Consolidated Configuration and Management

Unified Fabric: Multi-Hop FCoE Unified Ports

FCoE

FC

FEX Architecture

FET + FEX: Cabling cost efficiencies

Leaf

Lay

er

Spin

e La

yer

Converged FCoE link Dedicated FCoE link

FC

1 / 10GE

DR Data Center

OTV: Layer 2 Extension

ASA 5500

ACE ACE


•  Data Center Interconnect became a common block in our deployments

•  Applications running in the VMs use non-routable traffic

•  With Virtualization, application members may be distributed across PODs/Data-centers

•  Moving and distributing application members across locations should not break the application

Hypervisor Hypervisor

Network

Application Traffic (Non Routable)

Node Discovery Heartbeats

Hypervisor Control Traffic


Routing for extended subnets Active-Active Data Centers

Distributed Clusters

IP mobility across subnets Disaster Recovery

Cloud Bursting

DCI with LAN Extension DCI without LAN Extension

DCI Scenarios LISP and OTV usage

West-‐DC East-‐DC

Non-‐LISP site

IP Network

Mapping DB

LISP-‐VM (xTR)

LAN Extension

LISP site

xTR

West-‐DC East-‐DC

LISP site

Internet or Shared WAN

xTR

Mapping DB DR LocaEon or Cloud Provider

DC

LISP-‐VM (xTR)

Application Members Distributed Live moves

Application Members in one location Cold moves


•  How to Migrate your Apps to Your Fresh New DC?


Complexity of RISC Migration Scenarios

OS Hypervisor Solaris

containers

Other Std Apps

Home grown Apps

OS Hypervisor Power

VM

Home grown Apps

OS Hypervisor

Home grown Apps

Integrity VM OS Hypervisor

Home grown Apps

OS Hypervisor

Vertical Apps

Home grown Apps

Other Std Apps

Other Std Apps

Other Std Apps


Discovery

COTS Applications

ISV support Available

Use Standard Migration Templates

SAP

Oracle DB & E-Business

PeopleSoft

Siebel ISV support not available

Custom Applications

Application Architecture

Analysis

Applications built on Standard tiers (Web, J2EE, etc.)

Websphere

WebLogic

Jboss Custom Developed

Applications

Low-risk, proven template based migration path

Proven Application migration factory (iterative) methodology for cross platform migration

COTS = Commercial Off the Shelf


RISC/UNIX Migration

Acceleration

RISC/UNIX Migration Strategy

RISC/UNIX Migration Planning

RISC/UNIX Migration Execution

You are considering RISC/

UNIX migration

You understand the benefits and

now want a migration strategy

You are ready to move and want help

with sizing and detailed planning

RISC/UNIX Migration Service

You engage Cisco’s industry leading,

cost effective Migration Factory

(COTS and Custom)


Deliverables •  High-level application inventory

•  Application framework buckets

•  Candidates for migration

•  Migration complexity rating

•  List of applications that can be migrated to Cisco UCS™

•  Agree on next steps in a joint project charter document

Workshop Identification Discovery Onsite

Workshop

Business Case

Analysis

Total Duration: 4 to 5 weeks


Discovery Candidate Qualification

Migration Strategy

Migration Roadmap

Deliverables •  Application migration strategy blueprint

•  Qualified candidates for Cisco UCS™ migration plus risk index •  Target platform strategy

•  Guidance in choosing between bare metal and hypervisor •  Guidelines for OS selection •  Comparison of vertical and horizontal scalability

•  System management framework integration strategy •  Migration roadmap



RISC to UCS Workload

Transfer Map

Detailed Migration Risk

Analysis

Low Level Design and Test Plans Creation

Detailed Migration

Plan

Deliverables •  Physical to Virtual Transfer map •  Migration Risk Analysis Report •  Low Level Design Document •  Test Plan

•  Functional and Non Functional Test cases •  Acceptance Criteria

•  Detailed Migration Plan •  Work breakdown structure of migration tasks



Platform Readiness

Application Migration

Acceptance Tests

Promote to Production

Deliverables •  Target run time environment •  Acceptance Tests summary

•  Functional Tests •  Performance Tests •  Reliability-HA Tests

•  Run Book •  Standard configurations •  As Built documentation

Total Duration: (Depends on # of Applications and Complexity)


•  Work with your facilities staff to align your logical data center design to the physical design

•  Classical ToR, EoR, MoR models

•  Cisco Live Presentations

•  Cisco Books

•  TIA-942-A


Row

Row

End of Row, Top of Rack, and Blade Switches End of Row Top of Rack Blade Switches

EoR & Blades

ToR & Blades

GE Access

DAF & 1U Servers DAF & Blades

What it used to be…

What is emerging… What influences physical layout… Primarily:

Power Cooling Cabling

Secondarily Access Model Port Density

Row

What Cisco has done… Nexus 2K+Nexus 5k Nexus 2K +Nexus 7K

UCS Fabric Extender to Fabric

Interconnect


Equip Dist Area(Rack/Cabinet)

Horiz Dist Area(LAN/SAN/KVM

Switches )

Zone Dist Area



Switches )

Horizontal cabling

Horizontal cabling

Horizontal cabling



Switches )



Switches )

Horizontal cabling Horizontal cabling



Switches )

Horizontal cabling

Intermediate Dist Area

(LAN/SAN Switches )

Intermediate Dist Area

(LAN/SAN Switches )

Main Dist Area(Routers, Backbone LAN/SAN Switches, PBX, M13 Muxes)

Entrance Room(Carrier Equip &

Demarcation)

Offices, Operations Center, Support

Rooms

Telecom Room(Office & Operations

Center LAN switches)

Entrance Room(Carrier Equip &

Demarcation)

Horizontal cabling Backbone cabling

Backbone cabling

Backbone cabling Backbone cabling Backbone cabling Backbone cabling

Backbone cabling

Backbone cabling

Backbone cabling

Computer Room

Access Providers Access Providers


Cable Transceiver Latency (link)

Power (each side) Distance

Connector (Media)

Twinax ~ 0.1µs ~ 0.1W <7m SFP+ CU* copper

MM OM2 MM OM3 ~ 0 1W 82m

300m SFP+ SR MMF,short reach

MM OM2 MM OM3 ~ 0 1W 10m

100m SFP+ USR MMF, ultra short reach

Cat6 Cat6a/7 Cat6a/7

2.5µs 2.5µs 1.5µs

~ 6W*** ~ 6W*** ~ 4W***

55m 100m 30m

RJ45 10GBASE-T copper

100Mb 1Gb 10Gb

UTP Cat 5 UTP Cat 5 MMF, SMF

10Mb

UTP Cat 3

Mid 1980’s Mid 1990’s Early 2000’s

Late 2000’s

UTP Cat6a MMF, SMF TwinAx, CX4

Standard

SFF 8431**

IEEE 802.3ae

none

IEEE 802.3an

*** As of 2008; expected to decrease over time

10G Options

* Terminated cable ** Draft 3.0, not final

Twinax 15m X2 CX4 copper IEEE 802.3ak 4W ~ 0.1µs


Main Distribution Area

(MDA)

MDF

ZDA Server Cabinets Server Cabinets EDA EDA EDA EDA EDA EDA EDA EDA



Back-End Layer Zone

Application Layer Zone

Front-End Layer Zone ZDA

ZDA

ZDA

48F 48F 48F 48F 48F 48F 48F 48F

ZDA Server Cabinets Server Cabinets EDA EDA EDA EDA EDA EDA EDA EDA Storage Zone

ZDA 3 x 72F

Core �A� Core �B�

3 x 72F

48 fibers to each cabinet (2 x 100G link capacity)


1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

45 46 47 4841 42 43 4437 38 39 4033 34 35 3629 30 31 3225 26 27 28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Server 1 Server 2 Server 3 Server xx Patch Patch


1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

Server 1 Server 2 Server 3 Server xx Network

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44

1

5

10

15

20

25

30

35

40

45

2

3

4

6

7

8

9

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

43

44


•  Ask your vendor for a software recommendation

Your particular deployment features are considered

•  Bug Scrub

•  Release Notes

•  Supported configurations

•  Cisco Validated Design (CVD)

•  Interoperability matrices

•  Configuration limits


•  Plan Your Implementation! No Place for Cowboy work!

If you want to finish on time on budget

•  Floor Plan

•  Bay Face

•  Port-Mapping

•  Device Configs


•  Create the test case scenarios that most fit your needs

•  Test the whole deployment not only devices

wear your “customer” boots!

•  Simulate traffic and failures

•  Capture and analyze the data

•  Get to know your data center!