virtual san technical walk-through.pdf

7/26/2019 Virtual SAN Technical Walk-through.pdf

1/115

2014 VMware Inc. All rights reserved.

VMware Virtual SANTechnical Walkthrough / Medium Dive

Rawlinson Rivera, VCDXSenior Architect | Storage & Availability


2/115

Virtual SAN Technical Walkthrough

1 SDS and Virtual SAN Overview

2 Use Cases

3 Hardware Requirements

4 Technical Characteristics and Architecture

5 Configuration Walkthrough

6 Virtual Machine Provisioning Operations

7 Resiliency and Failed Scenarios

8 Interoperability

9 Design and Sizing

10 Troubleshooting

2


3/115

2011-2013

2008-2010

2005-2007

vSphere 4.x Thin Provisioning

Storage I/O control Boot from SAN VAAI Linked mode

2014+

VI 3.x VMFS Snapshots Storage vMotion NAS & iSCSI

support

vSphere 5.x

Storage DRS Profile-driven Storage VASA Sphere Storage

Appliance vSphere Data Protection vSphere Replication vSphere Flash Read

VMware Storage Innovations

3

Software-definedStorage


4/115

Hypervisor-Converged Opportunities

Why the virtualizationplatform can play acritical role to solvestorage problems?

Inherent knowledge of

application Global view of

infrastructure

Hardware agnostic

Hypervisor-Converged storage solutions abstractthe plumbing to optimize storage for applications

vSphere

SAN & NAS All Flash BLOB DAS

Server Side Flash

4


5/115

Virtual SAN5

vSAN

VSAN


6/115

SAN / NAS

VVOL

VMware Software-defined Storage

6

Bringing computes operational model to storage

Abstraction and pooling

Infrastructure integration

New storage tiers

VM centric data services

Third-party servicesintegration

Common policy-based automationand orchestration

LUN LUNLUN

LUN LUN

LUN

SAN/NAS Pool

Virtual Data Plane

x86 Servers

Hypervisor-convergedStorage pool

Object Storage Pool

Cloud ObjectStorage

Virtual Data ServicesData

Protection Mobility Performance

Policy-driven Control Plane

Virtual SAN


7/115

VMware Virtual SAN

Software-defined storagesoftware solution.

Aggregates locally attachedstorage from each ESXi host ina cluster.

Flash optimized storagesolution.

VM-Centric data operations andpolicy driven managementprincipals.

Resilient design based on aDistributed RAID architecture No single points of failures

Fully integrated with vSphere.

7

vSphere + Virtual SAN

Hard disksHard disksSSD SSD Hard disks

SSD

Virtual SAN SharedDatastore

Hypervisor-Converged storage platform


8/115

8

Data Protection

Disaster Recovery

vSphere

Cloud Ops and Automation

Snapshots

Linked clones

Site Recovery Manager vCenter Operations Manager

vCloud Automation Center

Virtual DesktopVDP Advanced

vSphereReplication

Storage Policy-BasedManagement

VMware Horizon ViewvMotion

vSphere HA

DRS

Storage vMotion

IaaS

Deeply Integrated with VMware Stack

Bringing the benefit of VMwares products to make Storage Easy


9/115

Virtual SAN is NOT a Virtual Storage Appliance

9

Virtual SAN is fully integrated with vSphere (ESXi & vCenter) Drivers embedded in ESXi 5.5 contain the Virtual SAN smarts Kernel modules:

Provide the shortest path for I/O Remove unnecessary management overheads when dealing with an

appliance Do not consume resources unnecessarily

Virtual SAN Embedded into vSphereVirtual SAN Not a VSA

VSA


10/115

VMware Virtual SAN

Hybrid storage solution Magnetic disks (HDD)

Flash based disks (SSD)

Storage scale out architecture

built into the hypervisor Dynamic capacity and

performance scalability

Object based storagearchitecture

Interoperable with vSphere andenterprise features: vMotion, DRS, vSphere HA

10



SSD

Virtual SAN SharedDatastore

Radically Simple Hypervisor-Converged Storage Software


11/115

11

Installs in two clicks

Managed from vSphereClient

Policy-based management

Self-tuning and elastic

Deep integration withVMware stack

Radically Simple

Embedded in vSpherekernel

Flash-accelerated

Up to 915K IOPs from 16nodes cluster

Matches the VDI density ofall flash array

Best price/performance

High Performance Lower TCO

Eliminates large upfrontinvestments (CAPEX)

Grow-as-you-go (OPEX)

Flexible choice of industry

standard hardwareDoes not requirespecialized skills

Virtual SAN Key Benefits


12/115

Storage Policy-Based

Management

VSAN SharedDatastore

Simplifies and Automates Storage Management

12

Per VM storage service levels from a single self-tuning datastore

Capacity

Performance

Availability

Per VM StoragePolicies

Policies set based onapplication needs

vSphere + VSAN

SLAs

Software automatescontrol of service levels

No m ore LUNs!


13/115

No overprovisioningLess resources, less timeEasy to change

Today

5. Consume from pre-allocated bin

4. Select appropriate bin

3. Expose pre-allocatedbins

2. Pre-allocate static bins

1. Pre-define storageconfigurations

1. Define storage policy

2. Apply policy at VMcreation

VSAN

VSANShared

Datastore

Resource and data service areautomatically provisioned and

maintained

Overprovisioning (better safethan sorry!)

Wasted resources, wasted time Frequent Data Migrations

Virtual SAN Puts The App In Charge

Simpler and automated storage management through application centricapproach

13


14/115

VMware Virtual SANUse Cases


15/115

15

Management Clusters

Use Cases

Backup and DR

Target

DMZ / Isolated

Tier 2 / Tier 3

Test / Dev / StagingPrivate c lou d

Virtual Desktop

ROBO

VDI

Site A Site B

vSphereVSAN


16/115

VMware Virtual SANHardware Requirements


17/115

Hardware Requirements

17

Any Server on the VMwareCompatibility Guide

SSD, HDD, and Storage Controllers must be listed on the VMware Compatibility Guide for VSANhttp://www.vmware.com/resources/com patibility/search.php?deviceCategory=vsan

Minimum 3 ESXi 5.5 Hosts, Maximum Hosts Ill tell you later

1Gb/10Gb NIC

SAS/SATA Controllers (RAID Controllersmust work in pass -through or RAID0mode

SAS/SATA/PCIeSSD

SAS/NL-SAS/SATAHDD

At least 1of each

4GB to 8GB USB, SD Cards, SATADOM
http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanhttp://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanhttp://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan


18/115

Flash Based Devices

In Virtual SAN ALL read and write operations always go directly to the

Flash tier.Flash based devices serve two purposes in Virtual SAN

1. Non-volatile Write Buffer (30%) Writes are acknowledged when they enter prepare stage on SSD. Reduces latency for writes

2. Read Cache (70%) Cache hits reduces read latency

Cache miss retrieve data from HDD

Choice of hardware is the #1 performancedifferentiator between Virtual SANconfigurations.

18


19/115

Flash Based Devices

VMware SSD Performance Classes Class A: 2,500-5,000 writes per second

Class B: 5,000-10,000 writes per second

Class C: 10,000-20,000 writes per second

Class D: 20,000-30,000 writes per second

Class E: 30,000+ writes per second

Examples Intel DC S3700 SSD ~36000 writes per

second -> Class E

Toshiba SAS SSD MK2001GRZB ~16000writes per second -> Class C

Workload Definition Queue Depth: 16 or less

Transfer Length: 4KB

Operations: write

Pattern: 100% random

Latency: less than 5 ms

Endurance 10 Drive Writes per Day (DWPD), and

Random write endurance up to 3.5 PB on8KB transfer size per NAND module, or 2.5PB on 4KB transfer size per NAND module

19


20/115

Magnetic Disks (HDD)

SAS/NL-SAS/SATA HDDs supported 7200 RPM for capacity 10000 RPM for performance

15000 RPM for additional performance

NL SAS will provide higher HDD controller queue depth at same driverotational speed and similar price point NL SAS recommended if choosing between SATA and NL SAS

Differentiate performance between clusters with SSD selection, andSSD:HDD ratio. Rule of thumb guideline is 10% of anticipated capacityusage

20


21/115

Storage Controllers

SAS/SATA Storage Controllers Pass- through or RAID0 mode supported

Performance using RAID0 mode is controller dependent Check with your vendor for SSD performance behind a RAID-controller

Storage Controller Queue Depth matters Higher storage controller queue depth will increase performance

Validate number of drives supported for each controller

21


22/115

Storage Controllers RAID0 Mode

Configure all disks in RAID0 mode

Flash based devices (SSD) Magnetic disks (HDD)

Disable the storage controller cache Allows better performance as cache is controlled by Virtual SAN

Disks Device cache support Flash based devices leverage write through caching

ESXi may not be able to differentiate flash based devices frommagnetic devices. Use ESXCLI to manually flag the devices as SSD

22


23/115

Network

1Gb / 10Gb supported

10Gb shared with NIOC for QoS will supportmost environments

If 1GB then recommend dedicated links for Virtual SAN

Jumbo Frames will provide nominal performance increase Enable for greenfield deployments

Virtual SAN supports both VSS & VDS NIOC requires VDS

Nexus 1000v Should work but hasn't been fully tested

Network bandwidth performance has more impact on hostevacuation, rebuild times than on workload performance

23


24/115

Firewalls

Virtual SAN Vendor Provider (VSANVP) Inbound and outbound - TCP 8080

Cluster Monitoring, Membership, and Monitoring Services (CMMDS) Inbound and outbound UDP 12345 - 23451

Reliable Datagram Transport (RDT) Inbound and outbound TCP 2233

24


25/115

VMware Compatibility Guide

25


26/115

Multi-level cell SSD (orbetter) or PCIe SSD

SAS/NL-SAS HDDSelect SATA HDDs

A n y Server onvSphere HardwareCompatibility List

* Note: For additional details, please refer to Virtual SAN VMwareCompatibility Guide Page

6Gb enterprise-gradeHBA/RAID Controller

1 2 Build your ownVSAN Ready Node

with 10 different optionsbetween multiple 3 rd partyvendors available at GA

Preconfigured server ready touse VSAN

using the VSAN CompatibilityGuide*

Choose individual components

Two Ways to Build a Virtual SAN Node

Radically Simple Hypervisor-Converged Storage
http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanhttp://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan


27/115

VMware Virtual SANTechnical Characteristics and Architecture


28/115

Technical CharacteristicsVirtual SAN is a cluster level feature similar to:

vSphere DRS

vSphere HA

Virtual SAN

Deployed, configured and manage from vCenter through thevSphere Web Client (ONLY!) .

Radically simple Configure VMkernel interface for Virtual SAN Enable Virtual SAN by clicking Turn On

28


29/115

Virtual SAN Implementation Requirements

Virtual SAN requires: Minimum of 3 hosts in a cluster

configuration

All 3 host MUST!!! contributestorage vSphere 5.5 U1 or later

Locally attached disks Magnetic disks (HDD) Flash-based devices (SSD)

Network connectivity 1GB Ethernet 10GB Ethernet (preferred)

29

esxi-01

local storage local storage local storage

vSphere 5.5 U1 Cluster

esxi-02 esxi-03

cluster

HDDHDD HDD


30/115

Storage Policy-based Management

SPBM is a storage policy framework built into vSphere that enablesvirtual machine policy driven provisioning.

Virtual SAN leverages this new framework in conjunction withVASA APIs to expose storage characteristics to vCenter:

Storage capabilities Underlying storage surfaces up to vCenter and what it is capable of offering.

Virtual machine storage requirements Requirements can only be used against available capabilities.

VM Storage Policies Construct that stores virtual machines storage provisioning requirements based on

storage capabilities.

30


31/115

Storage Policy Wizard

SPBM

VSAN object

VSAN objectmanager

virtual disk

VSAN objects may be(1) mirrored across hosts &

(2) striped across disks/hosts tomeet VM storage profile policies

Datastore Profile

Virtual SAN SPBM Object Provisioning Mechanism


32/115

Virtual SAN Constructs and Artifacts

New Virtual SAN constructs, artifacts andterminologies:

Disk Groups. VSAN Datastore. Objects. Components. Virtual SAN Network.

32


33/115

Virtual SAN Disk Groups

Virtual SAN uses the concept of disk groups to pool together flashdevices and magnetic disks as single management constructs.

Disk groups are composed of at least 1 flash device and 1 magneticdisk . Flash devices are use for performance (Read cache + Write buffer).

Magnetic disks are used for storage capacity.

Disk groups cannot be created without a flash device.

33

disk group disk group disk group disk group

Each host: 5 disk groups max. Each disk group: 1 SSD + 1 to 7 HDDsdisk group

HDD HDDHDDHDDHDD


34/115

Virtual SAN Datastore

Virtual SAN is an object store solution that is presented to vSphere asa file system.

The object store mounts the VMFS volumes from all hosts in a clusterand presents them as a single shared datastore. Only members of the cluster can access the Virtual SAN datastore

Not all hosts need to contribute storage, but its recommended .

34


Each host: 5 disk groups max. Each disk group: 1 SSD + 1 to 7 HDDs

disk group

VSAN network VSAN network VSAN network VSAN networkVSAN network

vsanDatastore

HDD HDDHDDHDDHDD


35/115

Virtual SAN Objects

Virtual SAN manages data in the form of flexible data containers calledobjects . virtual machine files are referred to as objects.

Virtual machines files are referred to as objects. There are four different types of virtual machine objects:

VM Home VM swap VMDK Snapshots

Virtual machine objects are splitinto multiple components basedon performance and availabilityrequirements defined inVM Storage profile.

35



disk group

VSAN network V SAN network VSAN network VSAN networ kVSAN network

vsanDatastore

HDD HDD HDD HDD HDD


36/115

Virtual SAN Components

Virtual SAN components are chunks of objects distributes acrossmultiple hosts in a cluster in order to tolerate simultaneous failures andmeet performance requirements.

Virtual SAN utilizes a Distributed RAID architecture to distribute dataacross the cluster.

Components are distributedwith the use of two main techniques:

Striping (RAID0)

Mirroring (RAID1)

Number of component replicasand copies created is based onthe object policy definition.

36

disk group disk group disk group disk group disk group

VSAN network VSAN networ k VSAN network VSAN networ kVSAN network

vsanDatastore

replica- 1 replica-2RAID1

HDD HDD HDD HDD HDD


37/115

Object and Components Layout

37


Virtual SAN Storage Objects

R1

R0 R0 R0

Availability definedas number of copies

Low level storageobjects wouldreside on differenthosts

VMFS VMFS VMFS

rolo2.vmdk

The VM Home directory object isformatted with VMFS to allow aVMs configuration files to bestored on it.

Performance mayinclude a stripe width

VMFS

rolo1.vmdk

rolo.vmx, .log, etc

/vmfs/volumes/vsanDatastore/rolo/rolo.vmdk

disk group

HDD

disk group

HDD

disk group

HDD

disk group

HDD

disk group

HDD


38/115

Virtual SAN Network

New Virtual SAN traffic VMkernel interface. Dedicated for Virtual SAN intra-cluster communication and data replication.

Supports both Standard and Distributes vSwitches Leverage NIOC for QoS in shared scenarios

NIC teaming used for availability and not for bandwidth aggregation.

Layer 2 Multicast must be enabled on physical switches. Much easier to manage and implement than Layer 3 Multicast

38

Management Virtual Machines vMotion Virtual SAN

Distributed Switch

20 shares 30 shares 50 shares 100 shares

uplink1 uplink2

vmk1 vmk2vmk0


39/115

Virtual SAN Network

NIC teamed and load balancing algorithms:

Route based on Port ID active / passive with explicit failover

Route based on IP Hash

active / active with LACP port channel

Route based on Physical NIC load active / active with LACP port channel

Management Virtual Machines vMotion Virtual SAN

Distributed Switch

100 shares 150 shares 250 shares 500 shares

uplink1 uplink2

vmk1 vmk2vmk0

Multi chassis link aggregation capable switches


40/115

Virtual SAN Scalable Architecture

40

Scale up and Scale out architecture granular and linearly storage,performance and compute scaling capabilities Per magnetic disks for capacity

Per flash based device for performance

Per disk group for performance and capacity

Per node for compute capacity

disk group disk group disk group

VSAN network VSAN networkVSAN network

vsanDatastore

HDD

disk group

HDD HDD HDD

disk group

VSAN network

HDD s c a

l e u

p

scale out


41/115

VMware Virtual SANConfiguration Walkthrough


42/115

Configuring VMware Virtual SAN

Radically Simple configuration procedure

42

Setup VirtualSAN Network

Enable VirtualSAN on the

Cluster

SelectManual orAutomatic

If Manual ,create diskgroups


43/115

Configure Network

43

Configure the new dedicated Virtual SAN network

vSphere Web Client network template configuration feature.


44/115

Enable Virtual SAN

One click away!!!

Virtual SAN configured in Automatic mode , all empty local disks areclaimed by Virtual SAN for the creation of the distributed vsanDatastore.

Virtual SAN configured in Manual mode , the administrator must manuallyselect disks to add the the distributed vsanDatastore by creating DiskGroups.

44


45/115

Disk Management

Each host in the cluster creates a single or multiple disk groups whichcontain a combination of HDDs, and SSDs.

45


46/115


A single Virtual SAN Datastore is created and mounted, using storagefrom all multiple hosts and disk groups in the cluster.

Virtual SAN Datastore is automatically presented to all hosts in thecluster.

Virtual SAN Datastore enforces thin-provisioning storage allocation bydefault.

46


47/115

VM Storage Policies

VM Storage Policies are accessible from vSphere Web Client Homescreen.

47


48/115

Virtual SAN Capabilities

Virtual SAN currently surfaces five unique storage capabilities to vCenter.

48


49/115

Number of Failures to Tolerate

Number of failures to tolerate Defines the number of hosts, disk or network failures a storage object can

tolerate. For n failures tolerated, n+1 copies of the object are createdand 2n+1 host contributing storage are required.

49

vsan network

vmdkvmdk witness

esxi-01 esxi-02 esxi-03 esxi-04

~50% of I/O ~50% of I/O

Virtual SAN Policy: Number of failures to tolerate = 1

raid-1


50/115

Number of Disk Stripes Per Object

Number of disk stripes per object The number of HDDs across which each replica of a storage object is

distributed. Higher values may result in better performance.

50

vsan network

stripe-2b witness


stripe-1bstripe-1a stripe-2a

raid-0raid-0

VSAN Policy: Number of failures to tolerate = 1 + Stripe Width =2

raid-1


51/115

Virtual SAN Storage Capabilities

Force provisioning if yes, the object will be provisioned even is the policy specified in the

storage policy is not satisfiable with the resources currently available.

Flash read cache reservation (%)

Flash capacity reserved as read cache for the storage object. Specified as apercentage of logical size of the object.

Object space reservation (%) Percentage of the logical size of the storage object that will be reserved

(thick provisioned) upon VM provisioning. The rest of the storage object isthin provisioned.

51

l fl k l d


52/115

Virtual SAN I/O flow Write Acknowledgement

vsan network

vmdkvmdk


VSAN mirrors write IOs to all active mirrors,these are acknowledged when they hit the flash buffer!

witness

Destaging to HDD is doneindependently between hosts.

raid-1

i l SA /O fl 1 i i i


53/115

Virtual SAN I/O flow 1MB increment striping

vsan network

witness


stripe-1b

stripe-1a 1MB (1)

raid-0raid-0

VSAN is thin provisioned by default, stripes grow in increments of 1MB

raid-1

1MB (3) 1MB (5) 1MB (2) 1MB (4)

(x) indicates stripe segment.

C d Obj Vi li i


54/115

Components and Objects Visualization

Visualization of mapping and layout of all objects and components

vSphere Web Client RVC

54

S C bili i R d d P i


55/115

Storage Capabilities Recommended Practices

55

Storage Capability Use Case Value

Number of failures to tolerate(RAID 1 Mirror) Redundancy

Default 1Max 3

Number of disk stripes per object

(RAID 0 Stripe)Performance Default 1

Max 12

Object space reservation Thick Provisioning Default 0Max 100%

Flash read cache reservation Performance Default 0Max 100%

Force provisioning Override policy Disabled

VM S P li i R d i


56/115

VM Storage Policies Recommendations

Number of Disk Stripes per object Should be left at 1, unless the IOPS requirements of the VM is not being

met by the flash layer.

Flash Read Cache Reservation Should be left at 0, unless there is a specific performance requirement to be

met by a VM.

Proportional Capacity Should be left at 0, unless thick provisioning of virtual machines is required.

Force Provisioning Should be left disabled, unless the VM needs to be provisioned, even if not

in compliance.

56


57/115

VMware Virtual SANVirtual Machine Provisioning Operations

Vi t l M hi P i i i O ti


58/115

Virtual Machine Provisioning Operations

All VM provisioning operation include access to VM Storage Policies

58

Vi t l M hi P i i i g O ti


59/115


If the VSAN Datastore understands the capabilities in the VM Storage

Policy, it will be displayed as a matching resource.

59

Vi t l M hi P i i i g O ti


60/115


If the VSAN Datastore can satisfy theVM Storage Policy, the VM Summarytab will display the VM as compliant.

If not, due to failures, or the forceprovisioning capability, the VM will beshown as non-compliant.

60

Virtual Machine Policy Management


61/115

Virtual Machine Policy Management

Modify VM performance, capacity, and availability requirements withoutdowntime.

61


62/115

VMware Virtual SANResiliency & Failure Scenarios

Understanding Failure Events


63/115


Virtual SAN recognized two different types of hardware device eventsin order to define the type of failed scenario: Absent

Degraded

Absent events are responsible to trigger the 60 minutes recoveryoperations. Virtual SAN will wait 60 minutes before starting the object and component

recovery operations

60 minutes is the default setting for all absent events Configurable value via hosts advanced settings

63



64/115


Degraded events are responsible to trigger the immediate recoveryoperations. Triggers the immediate recovery operation of objects and components

Not configurable

Any of the following detected I/O errors are always deemed degraded : Magnetic disk failures

Flash based devices failures

Storage controller failures

Any of the following detected I/O errors are always deemed absent : Network failures Network Interface Cards (NICs)

Host failures

64

F il h dli hil h


65/115

Failure handling philosophy

Traditional SANs Physical drive needs to be replaced to get back to full redundancy Hot-spare disks are set aside to take role of failed disks immediately

In both cases: 1:1 replacement of disk

Virtual SAN Entire cluster is a hot -spare , we always want to get back to full redundancy

When a disk fails, many small components (stripes or mirrors of objects) fail New copies of these components can be spread around the cluster for balancing

Replacement of the physical disk just adds back resources

Managing Failure Scenarios


66/115

Managing Failure Scenarios

Through policies , VMs on Virtual SAN can tolerate multiple failures

Disk Failure degraded event SSD Failure degraded event

Controller Failure degraded event

Network Failure absent event

Server Failure absent event

VMs continue to run

Parallel rebuilds minimize performance pain SSD Fail immediately

HDD Fail immediately

Controller Fail immediately Network Fail 60 minutes

Host Fail 60 minutes

66


67/115

Virtual SAN Access Rules

Components Access Rules

At least 1 mirror copy intact All stripes must be intact Greater than 50% of components must be available

Including witnesses

1 Mirror Copy All stripesavailable

> 50%components

andwitnesses

Power onOperation

Logic is implemented per object

Magnetic Disk Failure Instant mirror copy


68/115

Magnetic Disk Failure Instant mirror copy

Degraded - All impacted components on the failed disk will beinstantaneously created onto other disk, disk groups, or hosts.

vsan network

vmdkvmdk witness


vmdk

new mirror copyInstant!

Disk failure, instant mirror copy of impacted component

raid-1

Flash Based Device Failure Instant mirror copy


69/115

Flash Based Device Failure Instant mirror copy

Degraded - All impacted components on the failed disk will beinstantaneously created onto other disk, disk groups, or hosts.

Greater impact on the cluster overall storage capacity

vsan network

vmdkvmdk witness


vmdk

new mirror copyInstant!

Disk failure, instant mirror copy of impacted component

raid-1

Host Failure 60 Minute Delay


70/115

Host Failure 60 Minute Delay

Absent will wait the default time setting of 60 minutes before startingthe copy of objects and components onto other disk, disk groups, orhosts.

Greater impact on the cluster overall compute and storage capacity.

vsan network

vmdkvmdk witness


vmdk

new mirror copy

60 minute wait

Host failure, 60 minutes wait copy of impacted component

raid-1

Network Failure 60 Minute Delay


71/115

Network Failure 60 Minute Delay

Absent will wait the default time setting of 60 minutes before startingthe copy of objects and components onto other disk, disk groups, orhosts.

NIC failures, physical network failures can lead to network partitions. Multiple hosts could be impacted in the cluster.

vsan network

vmdkvmdk witness


vmdk

new mirror copy60 minute wait

Network failure, 60 minutes wait copy of impacted component

raid-1

Virtual SAN 1 host isolated HA restart


72/115

Virtual SAN 1 host isolated HA restart

vsan network

vmdkvmdk witness


isolated!

HA restartraid-1

vSphere HA restarts VM

Virtual SAN 2 hosts isolated HA restart


73/115

Virtual SAN 2 hosts isolated HA restart

vsan network

vmdkvmdk witness

esxi-01 esxi-02 esxi-03 esxi-04isolated! isolated!

HA restartraid-1

vSphere HA restarts VM on ESXi-02 / ESXi-03, they own > 50% of components!

Virtual SAN partition With HA restart


74/115

Virtual SAN partition With HA restart

vsan network

vmdkvmdk witness

esxi-01 esxi-02 esxi-03 esxi-04Partition 1 Partition 2

HA restart

vSphere HA restarts VM in Partition 2, it owns > 50% of components!

raid-1

Maintenance Mode planned downtime


75/115

Maintenance Mode planned downtime

3 Maintenance mode

options:Ensure accessibility

Full data migration

No data migration


76/115

VMware Virtual SANInteroperability Technologies and Products

Technology Interoperability


77/115

Technology Interoperability

Virtual SAN is fully integrated with many of VMwares storage andvSphere availability enterprise features.

77

Supported Not applicable Future

Virtual MachineSnapshots Storage IO Control (SIOC) 62 TB VMDKs

vSphere HA Storage DRS vCOPS

vSphere DRS Distributed Power Management(DPM)

vMotion

Horizon View


78/115

Virtual SAN and Horizon View:

Handle peak performance such as boot, login, read/write storms

Seamless granular scaling without huge upfront investments

Support high VDI density

Support high end virtual desktop GPU requirements

Virtual SAN is compatible with the followingHorizon View versions:

Horizon View 5.3 (SPBM manually implemented)

Policies maintained across operations such as refresh/refresh noneed to re-associate


Harddisks

Hard disksSSD SSD Harddisks

SSD

Full Clone Policies

FTT = 1 for persistent

FTT = 0 for non-persistent

Provisioning 100%reserved

Linked Clone Policies

OS Disk: FTT = 1 for dedicatedpools,

OS Disk: FTT = 0 for floating pool

Replica Disk: FTT = 1

Replica Disk: Read CacheReservation 10%

Provisioning: Thin

vSphere Replication and Site Recovery Manager


79/115

p p y g

Virtual SAN is compatible with: vSphere Replication 5.5 (vSphere Web Client)

SPBM configured as part of replication

vCenter Site Recovery Manager 5.5 (vSphere C#)

SRM configuration based on VR replication

vSphere Replication & vCenter SiteRecovery Manager Asynchronous replication 15 minute RPO

VM-Centric based protection

Provide automated DR operation & orchestration

Automated failover execution of user defined plans

Automated failback reverser original recovery plan

Planned migration ensure zero data loss

Point-in-Time Recovery multiple recovery points

Non-disruptive test automate test on isolated network

vCenterServer

VR/SRM

vSphere

VMFS

vCenterServer

VR/SRM

production site recovery site

replication

Harddisks

SSD


Harddisks

SSD Harddisks

SSD

vSphere Data Protection


80/115

p Virtual SAN and vSphere Data Protection

Radically simple to deploy and manage

Integrated User Interface vSphere Web Client

Highly available storage solution

Increase operation efficiency

vSphere Data Protection Advanced 5.5

Source and target De-duplication capabilities

Bidirectional replication capabilities

Secure, easy, reliable, network-efficient replication

Application-consistent backup and recoverycapabilities

Higher RTO and RPO 24 hours RTO, minutes hours RPO

Incorporated technologies

vStorage API for Data protection

Change Block Tracking (CBT)

Avamar variable-length segment algorithm

vCenterServer

Harddisks

SSD


Harddisks

SSD Harddisks

SSD

vSphere

VMFS

vCenterServer

vCloud Automation Center


81/115

vCloud Automation Center providesVirtual SAN: Centralized provisioning, governance, infrastructure

management capabilities

Simple and self-service consumption capabilities

Entitlement compliance monitoring, and enforcement

Leverage existing business processes and tools

Delegation control of resources

Custom use of VM Storage Policies: Virtual SAN default policy

Blueprints VM templates

Via vCenter Orchestrator with custom workflow Via vCloud Automation Center designer modifying

provisioning workflow


Harddisks

Hard disksSSD SSD Hard

disks

SSD

OpenStack


82/115

p

Virtual SAN and OpenStackFramework Cloud Ready App to Hypervisor

Converged solution

Leverage the use of Flash Optimizedstorage in OpenStack

Resiliency for legacy and Cloud Ready

applications vSphere Web Plug-in for OpenStack UI

Virtual SAN interoperates withOpenStack Framework.

vSphere Driver vSphere Datastore

Swift

object store

Glance

image store

HorizonDashboard

OpenStack FrameworkKeyStone

identity service

NSX

driver

Neutron

networking

Nova

compute node

vspheredatastore

driver

Cinder

volume service

vsphere

driver



SSD


83/115

VMware Virtual SANDesign & Sizing Guidelines Exercise



84/115

Distributed datastore capacity determined by aggregating the diskgroups found across multiple hosts that are members of a vSpherecluster and the size of the magnetic disks.

Only the usable capacity of the magnetic disks count towards the totalcapacity of the Virtual SAN datastore.

The capacity of the flash based devices is specifically dedicated toVirtual SAN's caching layer.



disk group


vsanDatastore

HDD HDDHDDHDDHDD

Objects


85/115

j

Individual storage block device that is compatible with SCSI semantics.

Each object that resides on the Virtual SAN datastore is comprised ofmultiple components.

Objects are assigned storage performance and availability servicesrequirements through VM Storage Profiles.

Object Types DefinitionsVM Home Location where all virtual machines configuration files reside (.vmx, log files, etc.)

Swap Unique storage object only created when virtual machines are powered on.

VMDK Virtual machine disk file

Snapshots Unique storage object created for virtual machines

Components


86/115

p

Objects are comprised of components that are distributed across hostsin vSphere cluster.

Virtual SAN 5.5 currently supports a maximum of 3000 components perhost.

Objects greater than of 255 gigabytes in capacity are automaticallydivided into multiple components.

Each component consumes 2 megabytes of disk capacity for metadata.

Witness


87/115

Witness components are part of every storage object.

Only contain object metadata.Serve as tiebreakers when availability decisions are made in the VirtualSAN cluster in order to avoid split-brain behavior.

Each Virtual SAN witness component also consumes 2 megabytes ofcapacity.

Virtual SAN Datastore Sizing Considerations


88/115

It is important to understand the impact of availability and performancestorage capabilities on the consumption of storage capacity. Number of Failures to Tolerate Number of Disk Stripes per Object Flash Read Cache Reservation Object Space Reservation

Disk Groups


89/115

Disk Groups

A single flash based device (SAS/SATA/PCIe SSD) and one or moremagnetic disks (SAS/SATA HDD).

Disk Groups make up the distributed flash tier and storage capacity ofthe Virtual SAN Datastore.

Formatted with a modified on-disk file system (VMFS-L) and are thenmounted onto the Object Store File System datastore as a singledatastore

VMFS-L on-disk file system formatting consumes a total of 750megabytes of capacity per disk.

Artifacts Minimums MaximumsDisk Groups 1 Per Host 5 per host

Flash Devices (SAS/SAS/PCIe SSD) 1 Per Disk Group 1 Per Disk Group

Magnetic Disk Devices 1 HDD Per Disk Group 7 HDD Per Disk Group

Disk Formatting Overhead 750 MB Per HDD 750 MB Per HDD

Number of Failures to Tolerate


90/115

Largest impact on the consumption of storage capacity in Virtual SAN.

Based on the availability requirements of a virtual machine, the settingdefined in a VM Storage Policy can lead to the consumption of up to fourtimes the virtual machine or individual disks capacity

2 full copies of data + 1 witness

Number of Disk Stripes Per Object


91/115

If the Number of Disk Stripes per Object is increased beyond the defaultvalue of 1 , then each stripe will count as a separate component.

This has an impact on the of total number of components supported perhost.

Disk Group Design


92/115

One Flash Device Per Disk Group

Multiple flash based devices, multiple disk groups will be created toleverage the additional flash

Higher the ratio of flash based device capacity to magnetic diskscapacity, the greater the size of the cache layer.

Define and reduce the storage failure domains.

Failuredomain



disk group


HDD HDDHDDHDDHDD

Flash Capacity Sizing


93/115

The general recommendation for sizing Virtual SAN's flash capacity is to have10% of the anticipated consumed storage capacity before the Number ofFailures To Tolerate is considered.

Total flash capacity percentage should be based on use case , capacity andperformance requirements.

10% is a general recommendation, could be too much or it may not beenough.

Measurement Requirements ValuesProjected VM space usage 20GB

Projected number of VMs 1000

Total projected space consumption per VM 20GB x 1000 = 20,000 GB = 20 TB

Target flash capacity percentage 10%

Total flash capacity required 20TB x .10 = 2 TB

Sizing Exercise Formulas


94/115

94

Constraints VSAN components and VMFS metadata overhead (VSANmetaDataOverhead) :

1GB per disk

Variables Number of Hosts Per cluster (Hst) = 8 Number of Disk Groups (DskGrp) = 5 Number of Disks Per Disk Group (DskPerDskGrp) = 7

Size of Disks (SzHDD) = 4000 GB Number of Failures To Tolerate (ftt) = 1 Number of Virtual Machines (VMs) = 800 Number of Disks per Virtual Machine (NumOfVMDK) = 1 Memory Per Virtual Machine (vmSwp) = 10 GB

Cluster RAW Capacity Formula: Hst x NumDskGrpPerHst x NumDskPerDskGrp x SzHDD = y Example: 8 x 5 x 7 x 4000 GB =1,120,000 GB =1,120 TB



95/115

95

VMFS Meta Data Formula: VMFSMetadata x NumDskGrpPerHst x NumDskPerDskGrp = y Example: 750 MB x 5 x 7 = 26,250 MB = 26.2 GB VMFS Metadata

Objects Formula: VMs x [ VMnamespace + vmSwap + NumOfVMDK] = y Example: 800 x [1 + 1 + 1] = 2400 Objects

Note: Snaps, Clones and >1 Disk Stripes would add more objects

Components Formula: Object x [ ftt x 2 + 1] = y Example: 2400 x (1 x 2 + 1) = 7200 Components = 900 average components

per host (max is 3000 per host)

Components Metadata Formula: NumComponents x compMetadata = y Example: 7200 Components x 2 MB = 14.4 GB Component Metadata



96/115

96

VSAN Meta Data Formula: compMetadata + VMFSMetadata = y Example: 14.4 GB + 26.2 GB = 40.6 GB VSAN Metadata

Swap Utilization Formula: (VMs x vmSwp x 2) Example:

Swap Space = (100 x 10GB x 2) = 2000 GB

Available Capacity = Raw Capacity Swap Capacity = 1120000 GB (100 x 10GB x 2) = 1120000 2000 = 1118000 = 1,118 TB Disk Capacity

Usable Capacity Formula : (DiskCapacity VSAN Meta Data) / (ftt + 1)

Example: (1118000 GB - 41 GB) / 2 = 1117959 GB / 2 = 558,980 GB UsableCapacity

Best practice is to utilize no more than 80% of usable capactiy

Memory and CPU


97/115

Memory requirements for Virtual SAN are defined based on the numberof disks groups and disk that are managed by hypervisor.

As long as vSphere hosts have greater memory configurations than 32gigabytes of RAM, they will be able to support the maximum disk groupand disks configuration supported in Virtual SAN.

Virtual SAN is designed to introduce no more than 10% of CPUoverhead per hosts. Consider this fact in Virtual SAN implementationswith high consolidation ratios and CPU intensive applicationsrequirements.

Network


98/115

Virtual SAN network activities can potentially saturate and overwhelm anentire 1GbE network, particularly during rebuild and synchronizationoperations.

Separate the different traffic types (Management, vMotion, VirtualMachine, Virtual SAN) onto different VLANs and use shares as a Qualityof Service mechanism to sustain the level of performance expectedduring possible contentions scenarios.

Virtual SAN requires for IP multicast to be enabled on the layer 2physical network segment utilized for Virtual SAN communication


99/115

VMware Virtual SANMonitoring & Troubleshooting

Network Status reports


100/115

Misconfiguration detected: Verify physical network Enable multicast

Disabling IGMP snooping Configure IGMP snooping for

selective traffic

Validate the virtual switchconfiguration VLAN

VSAN Traffic service enabled

NIC team failover policy

Failover Policy


101/115

NIC Teaming failover load balancing: policy with route based on port ID Active / Standby

101

Command Line Tools


102/115

VMKPING vmkping

Example 10.4.90.27

To validate network accessibility

ESXCLI esxcli vsan network list

102

Disk Claiming Operation


103/115

Automatic disk claiming operation fails to claim disks Is local: true disks are automatically claimed Is local: false disks are shared thus not automatically claimed but can be

manually marked local

103

Ruby vSphere Console


104/115

RVC VSAN

vsan.disks_info Size, disk type, manufacturers, model, local/non-local

104

Disk Groups Creation Fails


105/115

Disk Groups Creation Fails VSAN license needs to be added to the cluster

Home > licenses > Cluster tab > Select cluster object > Assign License Key

vSphere Web Client refresh time out Log out and back in

Unable to delete Disk Group VSAN disk claiming operation set to automatic, change to manual

vsan.host_wipe_vsan_disks -- force wipe disks used by VSAN

CONFIDENTIAL 105

Observing performance


106/115

Monitor performance:Ruby vSphere Console & VSAN Observer

In-depth monitoring of VSANs physical disk layer performance, cache hit rates,latencies, etc.

VSAN Observer


107/115

Starting the VSAN Observer Performance stats

1

2

VSAN Observer Monitoring Flash Devices


108/115

Monitor read cache hit rate

Flash based devices evictions to magnetic disks

108

VSAN Observer


109/115

Monitor disk groups aggregate and disk layers

Virtual SAN Logs


110/115

Virtual SAN related logs. Individually maintained per hosts

110



111/115

Disk Capacity used and reserved capacity

111

Monitoring VSAN Component Limits



112/115

Virtual SAN what if failure analysis

Simulate host failure impact to cluster



113/115

VSAN Observer recommendations

Deploy a VCVA appliance to use for the Observer Run the observer session on the newly deployed or remote VCVA appliance

Increase the data gathering time beyond the default (2 hours) if necessary.

Oh yeah! Scalability..


114/115

114

vsanDatastore

2.2 Petabytes

915K IOPS

THANK YOU


115/115

Rawlinson Rivera

virtual san technical walk-through.pdf

Documents