networker 8.1 enterprise backup protection for …

NETWORKER 8.1 ENTERPRISE BACKUP PROTECTION FOR VIRTUALIZED DATA CENTERGururaj KulkarniQA Manager, EMC [email protected]

Anupam SharmaSoftware QA Engineer, EMC [email protected]

Naveen RaoSenior Software QA Engineer, EMC [email protected]

mailto:[email protected]



2014 EMC Proven Professional Knowledge Sharing 2

Table of Contents

Introduction ................................................................................................................................ 3

Legacy Workflow ........................................................................................................................ 4

Practical Approach ..................................................................................................................... 5

Basic Configuration Steps .......................................................................................................... 6

Proxy selection and association ................................................................................................. 7

Test Analysis ............................................................................................................................. 8

Backup Scalability .................................................................................................................. 9

Incremental Backups .............................................................................................................. 9

Full vs. Incremental backups .................................................................................................10

Image level recovery .............................................................................................................11

Full Backup vs. Recovery ......................................................................................................11

Resource Utilization ...............................................................................................................12

CPU ...................................................................................................................................12

Memory ..............................................................................................................................12

Sizing Factors ...........................................................................................................................13

Recommendations ....................................................................................................................13

Conclusion ................................................................................................................................14

Appendix ...................................................................................................................................15

Disclaimer: The views, processes, or methodologies published in this article are those of the

authors. They do not necessarily reflect EMC Corporation’s views, processes, or

methodologies.


Introduction

In today’s world every technology is trying to passage towards virtualization. It has been a real

game changer in terms of how the storage is being considered now. Having its footprints

everywhere, backup domain is also not left. With the day by day new innovations, EMC has

empowered its backup solution with the virtualization handle. Backup applications are integrated

with virtualization solution, which enables it to backup large data center environments with the

effective performance.

The latest NetWorker® integration with VMware using Avamar® Virtual Backup Appliance (VBA)

is a great leap in protecting virtualized data centers with greater flexibility and ease of use. EMC

NetWorker 8.1 launched a new way to protect virtual data centers using this integrated solution.

It also adds a scalable model and provides a greater competitive edge in the market. The policy-

driven approach of protecting VMs with a centralized user interface provides end users with a

value-add solution. The solution is implemented using Industry-leading Avamar technology to

protect the VMs using user configurable policy. The policy helps backup administrator simply

define the policy with less effort and assign an action either to protect or clone the virtual

machine backup. This integration also leverages the EMC Data Domain® as target device.

Changed block tracking (CBT) helps to protect the VM by copying only changed blocks. The

solution has a centralized proxy which facilitates the data transfer between VMware and a

NetWorker target device such as Data Domain. It can co-exist with legacy work flow and

addresses a few of the challenges from earlier integration by reducing the complexity in

configuration and leveraging true CBT. The VBA leverages the advanced transport modes such

as hotadd for SAN-based protection and also provides nbd capability for protecting VMs using

the network.

The solution scales easily to protect up to 10,000 VMs in a data center being managed by single

or multiple virtual centers. Each VBA can easily scale to protect up to 500 VMs with one or two

external proxies. Additionally, each VBA has 8 internal embedded proxies and users can also

configure an external proxy being managed by VBA which in turn will have 8 proxies built.


Legacy Workflow

The following legacy NetWorker VADP integration limitations are addressed by Networker 8.1

enterprise backup protection:

• vCenter UI Integration – vCenter provides a plug-in architecture which allows 3rd party

applications such as NetWorker to provide a user interface from directly within vCenter.

This allows VM administrators to manage their own backup and recovery of VMs.

• Incremental Forever – There is no support for performing incremental forever backups

of VMs in NetWorker. This is comparable to synthetic full backups except greatly

speeds up backups on Data Domain devices when performing block-based backups of

virtual disks.

• Currency with VMware – VMware is aggressively rolling out new technologies and

products which 3rd party products such as NetWorker are expected to embrace and

leverage. NetWorker’s current VADP integration is falling behind in many areas including

vCloud and vApp support.

• Proxy Pools – NetWorker’s native VADP solution requires administrators to manually

associate VMs with the VADP proxy which will back it up. In Nemo, this relationship is

managed automatically by the VBA node.


Practical Approach

Networker Enterprise Backup Solution for Virtualized data center Environment

1000 VMS

UCS1ESX5.1

UCS2ESX5.1

UCS12ESX5.1

10G

10G

VBA VM

External Proxy1VM

External Proxy2VM

External ProxyNVM

VM1

VM2

VM3

VM1000

NW Backup server

vCenter 5.1 VM

VBA: Vmware Backup ApplianceNW: Networker server

Data Domain 890 OS: 5.3

Figure 1

The overall Backup and Recovery infrastructure is shown below:

1. Infrastructure includes 12 Cisco UCS servers.

10G network between UCS chassis and Data Domain.

2. All VMs are residing on ESX 5.1 connected to VNX storage

3. All datastores are shared across the ESXs.

4. Fresh Installation of NetWorker Server, with NetWorker 8.1 builds was carried

out.

5. Throughout the test cycle the NetWorker Server platform used was Windows

2008 R2.

6. VMware vCenter 5.1 has been used as the hypervisor.

7. Different VBA versions have been deployed and used as proxy host for backing

up the VMs.


8. Dedicated datastores have been provided for each VBA. So there is no other VM

residing on that particular datastore.

9. The target devices used were

Data Domain type device for DD-BOOST. OS: 5.3.0.1-348040 Model: DD890

and Avamar GSAN

10. Different backup policies have been created and assigned with specific number

of VMs.

11. Test has been carried out by scaling the number of VMs from 1 to 500 and

increasing the number of vCenters, VBAs, External proxys, and DD devices.

12. Test has been carried out keeping the backup concurrency in mind while

performing backup and recovery.

Basic Configuration Steps

Steps to take for setting up EMC NetWorker 8.1 for VM protection are shown below. Detail

configuration can be found in EMC NetWorker documentation (EMC NetWorker and VMware

Integration Guide)

1. Deploy NetWorker Server, VMware VCenter, VBA, and External proxies

2. Connect VBA to VCenter and NetWorker Server

3. Create Data Domain device on NetWorker Server for Data Domain-related tests

4. Create Backup policies on NetWorker Server

5. Add Data Domain or GSAN device to the policy

6. Log in to vSphere web client and access the VBA UI to get the policies

7. Attach VMs to the respective policies in the VBA UI

8. Start backup policy from NetWorker to take the backup.

9. Start recovery from VBA UI to recover the VMs

10. Calculate the backup/recover throughput based on overall backup data and the

total time taken by policy run


Proxy selection and association

Suppose a data center has clusters made up of one or more ESX servers with each ESX server

managing one or more VMs. The proxy selection and transport mode used during backup

depends on availability of proxy and transport mode. The VBA supports nbd and hotadd

transport mode. The nbd used to back up VM over LAN is generally slower due to TCP and

bandwidth limitation. The other transport mode is “hotadd” which provides more reliable and

faster SAN-based backup.

Cluster-1

ESX

VMS

ESX

VMS

ESX

VMS

ESX

VMS

VBA

Cluster-1 Cluster-3

ESX

VMS

ESX

VMS

ESX

VMS

ESX

VMS

Proxy-1

ESX

VMS

ESX

VMS

ESX

VMS

ESX

VMS

Proxy-2

LAN

VC

(Max stream: 100)

NW server

(Max stream: 1024)

SAN

DS DS DS DS DS

DD990

(Max stream:540)

Max Stream:26 Max streams per

component : 8 DSLocal to proxy-1 Local to VBA Local to proxy-2


By default, each proxy checks hotadd mode and if hotadd is not available, it will not make VM

backup wait but it will fall back to “NBD” mode. In a situation where VBA installed in cluster-1

and proxies in cluster-2 and 3, the proxy selection happens randomly by VBA. So, if all slots on

external proxies 2 and 3 have filled and VBA has free slots, cluster-2 and 3 VMs backup will fall

back to NBD mode to VBA. The same flow applies where VBA has filled with all slots and

external proxy slots are available; cluster-1 VMs backups may happen with NBD mode to either

of cluster-2 or 3 external proxies. The simple illustration above shows this selection

process. When a user registers an external proxy to VBA, they can see all proxies are listed as

avamarclients on VBA with all available data stores selected for each of these avamarclients.

This is to ensure NetWorker always protects the VM even if hotadd fails.

Test Analysis

Successful backups and good performance for a single VBA can be seen up to the

26 VMs while performing the concurrent backups.

Successful backup and good performance can be observed while performing

48 VMs backup with 2 VBAs having 1 vCenter and 2 proxies on each VBA

writing to 4 Data Domain devices and 4 policies keeping 3-4 minute delays

between each policy start.

96 VMs backup with Data Domain results in a performance gain as the VM in

ratio of 24/VC/VBA is increased for concurrent backups

Figure 2: Concurrent backup throughput


Backup Scalability

A single VBA can protect up to 2500 VMs with an ideal VM size of 30GB and change

rate of 1-2% with 2 external proxies and 24 concurrent streams.

Scaling per VBA can slightly affect the performance based on VBA queue size. As a

single VBA can handle 26 concurrent backups, the graph below shows the policy

with 500VMs; throughput slightly dips after 96 VMs starting at a time.

Figure 3: Scalability backup throughput

Incremental Backups

0% Incremental

Backups with no change or 0% incremental shows expected results and

excellent performance.

5% Incremental

Backups with 5% incremental shows linearly improved performance

with scaling VM backups from 1 to 24 VMs.

Back up on Data Domain in comparison with GSAN shows ~2 times

better backup performance. If the number of VMs increases with small

change rate, the backup performance linearly increases. The change

block tracking (CBT) sends only changed blocks during the next

consecutive backups.


Figure 4: 5% incremental backup throughput

Full vs. Incremental backups

While using Data Domain as backup target device, incremental backup (5%

incremental) is 2-3x times faster than FULL backup. If the number of VMs

increases with small change rate, the backup performance linearly increases.

The change block tracking (CBT) sends only changed blocks during the next

consecutive backups.

Figure 5: Full vs. Incremental backups


Image level recovery

Recover performance on Data Domain increases consistently with the

increased number of VMs.

Recover performance on GSAN shows a throughput dip after 8 VMs. This is

because GSAN recovery is bounded by memory and CPU and there is high

memory utilization on VBA.

Figure 6: Image level recovery

Full Backup vs. Recovery

The recovery needs to rehydrate the data from backup target. The

concurrent recovery with more VM shows 20% slower performance

compared to backup operation.

Single VM shows Recover performance better than Backup; behavior is

same for DD as well as GSAN.

Major difference can be seen with 24 VMs where for DD shows a ~20% dip

in the Recover performance compare to Backup.


Figure 7: Full Backup versus recover throughput

Resource Utilization

CPU

The backup to GSAN is CPU intensive. GSAN backup CPU usage is

almost double in comparison to Data Domain.

Incremental backup GSAN CPU usage is 2 times that of Data Domain.

Maximum CPU usage can be seen with 500 VM backup with 1 VBA.

Memory

Memory usage for recovery on GSAN is comparatively higher than Data

Domain.

Full Backup memory usage on Data Domain show high utilization

compared to GSAN.

Memory usage increases significantly from 200 VM backup to 300 VM

backup, nearly double.

High memory consumption can be seen with 400 VM and 500 VM backups

with a single VBA.


Sizing Factors

Scalability factors

Factors Recommended

count Notes/Comments

Number of VMs per VBA (DDR backup)

1000-2500

With an average size of 25-30 GB per VM, the 0.5 TB OVA can accommodate up to 2500 VMs with retention period of 30 days. An average of 1000 VMs is an ideal choice per VBA if size of VMs is exceeding average size.

Number of VMs per policy

Up to 800 Single NetWorker backup policy is able to manage up to 800 VMs

Concurrency/Parallelism

Component Concurrency

count Notes

VBA 26 concurrent VMs

The maximum concurrent sessions per VBA is 26, Irrespective of the target device.

External proxy

24 concurrent hotadd of VMDKs

External proxy has only 2 SCSI controllers which limits the concurrent hotadd sessions to 24. If you back up more than 24 VMDKs, the backup uses NBD mode.

VC 25 concurrent session

EMC recommends a maximum of 25 concurrent VM backups per vCenter. If vCenter runs on a stand-alone server with the vCenter database running on SQL, a single vCenter can process more than 25 VMs at a time.

Recommendations

Performance and scalability of the NetWorker VMware Protection solution depends on several

factors, including which VBA is deployed, the number of vCenters and number of proxies, and

whether a large number of concurrent VM backups are performed. Use the following

recommendations and best practices as a guideline for achieving the best performance in your

environment:

The performance with backup target as DDR is better compared to GSAN internal

storage. This is due to processing overhead by GSAN while storing to its internal

storage. This processing overhead is offloaded some while backing up to DDR.

The 0.5TB VBA is recommended for DDR backups. If there is no DDR and user has

only VBA, 4TB VBA should be used to store the backup to internal GSAN.


An EMC Backup and Recovery appliance can back up up to 8 VMs in parallel. If a user

wants to run up to 24 VM backups in parallel, add up to 2 external proxies. Each

external proxy can back up up to 8 VMs.

For better performance, it is recommended to use a dedicated datastore for VBA,

especially for backups and recoveries performed from internal storage of the VBA.

I/O contention may occur during snapshot creation and backup read operations when

all VMs reside on a single datastore.

It is recommended to set an appropriate NetWorker server parallelism value to reduce

queuing on the NetWorker server.

o Each VM backup to DDR consumes one session on the Data Domain device. If

you exceed the maximum limit of 60 sessions, EMC recommends configuring

additional devices.

o When using 1 VBA and 2 External proxy VBAs for Data Domain backup, the

best performance occurs for backups of up to 48 VMs. Backup throughput

does not improve significantly beyond 48 VMs.

o To achieve the best concurrent backup performance in a setup that requires

additional vCenters, VBAs, or proxies, it is recommended to use 1 VBA and 2

External proxies per vCenter.

Conclusion

NetWorker 8.1 enterprise backup protection successfully meets the performance requirements

for virtual data centers that have a large number of VMs to be protected. There are certain

guidelines and recommendations to be followed to achieve desired performance. Backup

protection is effective as well as scalable which makes it a better solution compared with legacy

systems. The solution works effectively with changed block tracking (CBT) which makes it more

efficient. Another key is how the solutions easily scale and can support huge environment. This

article is meant to spread knowledge on effective use of this solution to achieve the best results

from the available system.


Appendix

Resource Utilization Graphs

Backup Concurrency

Figure 8a: Concurrent backups CPU

Figure 8b: Concurrent backup memory

Backup Scalability

Figure 9a: Scalability backups CPU

Figure 9b: Scalability backups memory


Incremental Backups

Figure10a: Incremental backups CPU

Figure10b: Incremental backups memory

Full vs. Incremental Backup

Figure11a: Full vs. incremental backup CPU

Figure11b: Full vs. incremental backup memory


Image level recovery results

Figure12a: Image level recovery CPU

Figure12b: Image level recovery memory

Backup vs. Recovery

Figure13a: Backup vs. Recovery CPU

Figure13b: Backup vs. Recovery memory


EMC believes the information in this publication is accurate as of its publication date. The

information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION

MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO

THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED

WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an

applicable software license.

networker 8.1 enterprise backup protection for …

Documents