Nimble Array Self-Install Document For 2.x Code Installations

Jason Johnson – Sr. Systems Engineer jason.johnson@nimblestorage.com Aaron Thomas – Account Executive amthomas@nimblestorage.com

2

Table of Contents Audience: ....................................................................................................................................................................3 Scope: .........................................................................................................................................................................3 First Steps ...................................................................................................................................................................4 Understanding IP’s .....................................................................................................................................................4 iSCSI Switch Selection Guidelines ...............................................................................................................................5 Known good Switches.................................................................................................................................................5 Network Best Practices ...............................................................................................................................................6 Pre-Install Checklist ....................................................................................................................................................7 Terminology ................................................................................................................................................................8 Recommended Multi-Switch Connectivity .................................................................................................................8 Network Best Practices ...............................................................................................................................................9 Windows Timeouts .....................................................................................................................................................9 VMware Settings ..................................................................................................................................................... 10 Nimble Array configuration ..................................................................................................................................... 10 Volume Creation ...................................................................................................................................................... 18 Volume Connection Verification ............................................................................................................................. 19 VMware Vcenter Plug-In registration: ..................................................................................................................... 19 Port use and Firewall considerations ...................................................................................................................... 20 Support Service Levels ............................................................................................................................................. 21 Popular CLI commands (user name for CLI login is “admin”) ................................................................................ 21 Shutdown Automation ............................................................................................................................................ 22 10GbE Cable Detail .................................................................................................................................................. 23 Short name is SFP+. Long version is below (from the hardware user guide). ......................................................... 23

3

Audience: This document is intended for new customers’ storage system administrators or Nimble reseller system engineers wishing to perform a self-installation or to provide pre-information regarding the installation process.

Scope: This document provides the basic information to self-install a Nimble Storage array. This document is not intended to replace other Nimble documentation but merely to augment and facilitate a quick and easy self-startup. This document is not all inclusive and is not detailed as to every aspect of the operation. Use this document as a quick guide to get the system installed and basic verification then refer to other related documentation like the Nimble User Guide for more in-depth information on use and management.

4

First Steps – Before you receive your array: • Call Nimble Support and request a customer login: (877) 364-6253 X2 • Once you have this log on to the support site and download the following:

http://support.nimblestorage.com/download/download.html o Latest Release Notes o Latest User Guides o Latest CLI Reference Guide o Windows Integration Toolkit o VMware Integration Toolkit (if applicable) o Related Best Practice Guides

• Fill in the Pre-Install checklist on page 5, one for each system you are installing, and send the completed form to your Nimble SE and/or your reseller SE.

Understanding IP’s The array management IP addresses: The array management IP address is a "floating" address, and is assigned to the physical port by the system. It is the IP address used to access the array's GUI and CLI. The address is used by the controller taking the active role. Best Practice: This IP address can be used for data, but this is not desirable: specific target IP addresses of the interface pairs should be used instead. The target discovery IP address: The discovery IP address is also a "floating" address, and is assigned to the physical port by the system. It is the IP address used to allow the iSCSI initiator to discover the array. This can be the same as the management IP address or a different IP address. Best Practice: This IP address can be used for data, but this is not desirable: specific target IP addresses of the interface pairs should be used instead. The data IP addresses: Each interface pair in the array (up to a total of six (CS220/CS240/CS260/CS420/ CS440/CS460) or four (CS210) or two (CS220G/CS240G/CS260G/CS420G/CS440G/CS460G) can be assigned an IP address and subnet. Both members of the interface pair use this address, but because of the active-standby nature of the controllers, the IP address is never used simultaneously by both. These IP addresses cannot be the same as the Management/Target Discovery IP addresses. These are the IP addresses to which iSCSI initiators should be assigned access. By specifying a physical target portal using your Advanced iSCSI initiator feature, you can ensure that different applications and volumes use different ports, avoiding a situation in which all data and management traffic goes through one physical port while leaving the others unused. Instruct your iSCSI initiator to use a specific interface pair for data traffic to and from volumes. The two controller diagnostic IP addresses: One static IP address is assigned to each controller to ensure that controllers can be accessed regardless of any network problem that might occur. This allows direct access to the controller at any time for diagnostic purposes: regardless of the state of the array or the controller, access to the

5

controller is always available. The system determines the physical port to which these IP addresses are associated.

iSCSI Switch Selection Guidelines These are some basic guidelines for selecting a good switch for the iSCSI LAN between your Nimble and the servers. It’s best to have a separate physical redundant switch pair for your iSCSI traffic, but if you do chose to run this traffic on your campus LAN you should VLAN that traffic to keep it separate. Switch selection basics:

• It should be a good quality layer 2 or layer 3 managed switch, stacked is preferred, ISL’s at least. • It needs to have dedicated port buffer space of at least 512K for each port. • In needs to have hardware flow control (this was the most important) • It is good if the switch can support jumbo frames (at the same time as flow control…many HP Procurves

can do one or the other) • The backplane need to be able to handle the full bandwidth of the data flowing through the switch, i.e.,

if the Nimble can handle has 6 data ports and can handle 720MB/sec the switch backplane should be able to do better.

Known good Switches These are some known good switch options in no particular order. This is not an exclusive list. 1GbE Switches:

• Juniper EX3300 • Juniper EX4200 • Brocade ICX-6610 • Brocade VDX-6710 • Cisco 3750X • HP 2810-24G - you cannot have Jumbo Frames and Flow Control on simultaneously. • HP 2910

10GbE Switchs:

• Juniper EX4500 • Brocade VDX-6720 • Juniper EX2500 • Juniper EX4200 • Brocade VDX-6720-

6

Network Best Practices Best Practice Details

Do not use Spanning Tree Protocol (STP) Do not use STP on switch ports that connect to iSCSI initiators or the Nimble storage array network interfaces.

Configure flow control on each switch port Configure Flow Control on each switch port that handles iSCSI connections. If your application server is using a software iSCSI initiator and NIC combination to handle iSCSI traffic, you must also enable Flow Control on the NICs to obtain the performance benefit.

Disable unicast storm control Disable unicast storm control on each switch that handles iSCSI traffic. However, the use of broadcast and multicast storm control is encouraged.

Use jumbo frames when applicable Configure jumbo frames on each switch that handles iSCSI traffic. If your server is using a software iSCSI initiator and NIC combination to handle iSCSI traffic, you must also enable Jumbo Frames on the NICs to obtain the performance benefit (or reduce CPU overhead) and ensure consistent behavior. Do not enable Jumbo Frames on switches unless Jumbo Frames is also configured on the NICs.

Multicast filtering off.

IP precedence off

7

Pre-Install Checklist Fill in this pre-install checklist and return this to your installing Nimble SE or reseller SE. The “Nimble Array Setup Manager” wizard will ask you to input the “Phas2 1 information, then you will be redirected to a browser session to log into where you will complete phase 2 through 5. Having this already determined and readily at hand will speed up and simplify the process. It is much easier to get the install right the first time than it is to change some configuration items later. The whole process should take less than15 minutes. The system is case sensitive and will not accept spaces or underscores, dashes are OK.

Phase 1 Required Info Example Notes Array Name Nimble-01 Management IP 192.168.10.50 Subnet 255.255.255.0 Default Gateway(s) 192.168.10.1 For management subnet Password P@ssw0rd 8 characters minimum Phase 2 Discovery IP 192.168.1.100 Type (Data,Manag., Both) Data IP MTU (Stand, Jumbo) Example Eth 1 192.168.10.51 Eth 2 ** 192.168.10.52 Eth 3 or TG1 192.168.1.101 Eth 4 or TG2 192.168.1.102 Eth 5 (not CS210 or g’s) 192.168.1.103 Eth 6 (not CS210 or g’s) 192.168.1.104 Example Notes Controller “A” IP 192.168.10.53 For support use Controller “B” IP 192.168.10.54 For support use Phase 3 Domain Name bs.com Fully Qualified DNS Server 4.2.2.2 On management subnet Phase 4 Time Zone Europe/London NTP Server 192.168.10.5 Phase 5 SNMP Server snmp.bs.com Make sure relay is set Email From Nimble-01@bs.com Email To IT-Admin@bs.com Public/Campus Network iSCSI network

**The install wizard assumes two management ports, so if you are only configuring one management port skip the eth2 port and change it back to a data port in the GUI after the wizard installation

8

Terminology • Sibling interfaces

o Interfaces on same controller o E.g., ControllerA.eth1 and ControllerA.eth2

• Mirrored interfaces o Pair of corresponding interfaces across controllers o E.g., ControllerA.eth1 and ControllerB.eth1

• Cross link o Link between switches

• Connection from host to array o Direct: does not go through the cross link o Cross: goes through the cross link

• Recommended connections o Are direct under recommended network connectivity o Conn1: Host.eth1 to Array.eth1 o Conn2: Host.eth2 to Array.eth2

• Non-recommended connections o Are cross under recommended network connectivity o Conn3: Host.eth1 to Array.eth2 o Conn4: Host.eth2 to Array.eth1

Recommended Multi-Switch Connectivity The easiest way to get this right is to connect the even ports to the even switch number and the odd ports to the odd switch number. i.e., eth1 to switch 1, eth2 to switch 2, eth3 to switch 1, eth4 to switch 2, for each controller.

Controller A

eth1

S1

Host eth1 eth2

eth2 Controller B

eth1

S2

eth2

cross

Active link Standby link

9

Network Best Practices

Best Practice Details Do not use Spanning Tree Protocol (STP) Do not use STP on switch ports that connect to iSCSI

initiators or the Nimble storage array network interfaces.

Configure flow control on each switch port Configure Flow Control on each switch port that handles iSCSI connections. If your application server is using a software iSCSI initiator and NIC combination to handle iSCSI traffic, you must also enable Flow Control on the NICs to obtain the performance benefit.

Disable unicast storm control Disable unicast storm control on each switch that handles iSCSI traffic. However, the use of broadcast and multicast storm control is encouraged.

Use jumbo frames when applicable Configure jumbo frames on each switch that handles iSCSI traffic. If your server is using a software iSCSI initiator and NIC combination to handle iSCSI traffic, you must also enable Jumbo Frames on the NICs to obtain the performance benefit (or reduce CPU overhead) and ensure consistent behavior. Do not enable Jumbo Frames on switches unless Jumbo Frames is also configured on the NICs.

Windows Timeouts (Set to 45 seconds) 1) Changing iSCSI Timeouts on Windows

o Update the registry setting on your Windows server to 45 seconds. For more information about this, see the Microsoft iSCSI guide at http://technet.microsoft.com/en-us/library/dd904411(WS.10).aspx

o HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\instance-number\Parameters

o MaxRequestHoldTime set to 60 seconds (0X3C) o LinkDownTime set to 45 seconds (0X2D) o HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Disk o TimeOutValue set to 60 seconds (0X3C) o Setup MPIO on your server and make sure it’s active. o This will require a reboot of the server to make both the registry edit and the MPIO active so do this

ahead of time so as to not delay installation. o Setup your email gateway/relay prior to the installation o See page 24 of the Nimble Windows Installation Tool Kit

10

VMware Settings 2) Changing iSCSI Timeouts on Vmware

o None Needed 3) Changing iSCSI Timeouts on Linux (iscsid.conf)

o For Linux guests attaching iSCSI volumes o node.conn[0].timeo.noop_out_interval = 5 o node.conn[0].timeo.noop_out_timeout = 12

4) Configure Round Robin ESX 4.1 only (4.0 will be different): To set the default to Round Robin for all new Nimble volumes type the following, all on one line:

o esxcli nmp satp addrule --psp VMW_PSP_RR --satp VMW_SATP_ALUA --vendor Nimble 5) Configure Round Robin ESXi5 only: To set the default to Round Robin for all new Nimble volumes type the

following, all on one line: o esxcli storage nmp satp rule add --psp=VMW_PSP_RR --satp=VMW_SATP_ALUA --vendor=Nimble

6) Configure the Vcenter Plug-In: o Pre-1.3.x firmware use the CLI command:

� vmwplugin --register --username arg --password arg --server arg • *See Nimble VMware Integration Guide • *Support mode to get CMD line (Turn on: Configuration-Security profile-properties-

remote tech support ssh) o For 1.3.x code or newer configure via the GUI under the “Administration” tab

Nimble Array configuration via Nimble Install Manager • Prerequisites

o Static IP. Set your IP address to the same subnet as your array management will be on. o Have your array controllers A & B correctly cabled to your switch fabric per the previous

drawings. o Complete all your switch configurations for Flow Control, Jumbo Frames, Spanning tree, Unicast,

etc. o Install the Windows Integration Tools (WIT) on the Laptop or Server you’re installing with.

11

1) Start the “Nimble Array Setup Manager” 2) Select the Array to install and click “Next” 3) Enter the array name (No spaces or underscores, case sensitive) 4) Set your management IP address,

Subnet mask & Default Gateway 5) Enter and confirm your array

password. (Examples shown) 6) Click “Finish”

12

Enter Management IP

If you get this screen click this selection to continue

7) You should get this screen after a few seconds. 8) Click “Close” and follow the instruction by opening a browser and pointing it to your management IP address 9) If a browser connection does not open up

automatically, open a browser and enter the management IP address as the URL.

13

Select the type of network you are configuring: Most likely this middle selection

10) Log into GUI interface using the password you just created. Note for CLI login via an SSH connection the username is “admin”, the password is the same as the GUI interface.

11) Select the type of network topology you intend to use. The typical and most desirable is a separate

Management and Data subnet, but you can also set a flat network where data and management are on the same subnet.

Log in with the password you just set

14

Set your diagnostic IP addresses

12) Set you IP addresses per the pre-installation checklist document you filled out. 13) Set you controller support IP addresses. These should be on the management subnet and are used by

Nimble support.

Set your physical IP addresses

15

Set your Domain and DNS Servers

14) Enter your domain and DNS servers

15) Select your time zone and enter your NTP server.

Set your Time Zone and NTP server

16

Enter you email server

Enter you email alert addresses

16) Setup your email alerts. Aliases are recommended. Make sure you have the following addresses in your Exchange server for unauthenticated relay: Management IP: X.X.X.X Ctrlr-A IP: X.X.X.X Ctrlr-B IP: X.X.X.X

17) Click finish and make not of the information regarding firewall ports on the next screen. Follow these instructions

17

Open the AutoSupport page

18) Your system is now functional. Let’s setup the AutoSupport “Call home” capabilities. Go to the “Administration” Tab and select the “AutoSupport/HTTP Proxy” line.

19) Check the Send AutoSupport box 20) Check the Enable Secure tunnel box (this is optional but a good idea) 21) Select the test “AutoSupport Setting”. The test will run, you should get a passing indicator for each line. If

you do not check your firewall settings (see tables below) to make sure the proper ports are open for the

Check this box

Check this box or not as desired.

Test the

Send

18

management IP and the Controller support IP. Re-test until these all pass. If you have trouble, contact our support so they can help you. (877) 364-6253 X2.

22) Once the tests have passed select “Send AutoSupport” and verify you received an autosupport test email. If you did not then check your email alert settings and relays.

Volume Creation Your system is now ready to create volumes and map to servers. Please refer to other documents listed below for details on these steps: • Nimble Users Guide – This is embedded in the GUI firmware and can be located under the ‘Help” button • VMware Integration Guide

If you are going to be connecting to a VMware environment refer now and complete the steps in the VMware integration guide.

1) Select the “Management” tab, then the “Initiator Group” line: a. Create a new initiator group. Give the imitator group a name that make sense to you based on

what servers it includes b. Add the IQN numbers (initiators) to the group for each of the servers that will mount the volume

you create. Note: This should be one server to one volume unless it’s a clustered volume like a VMware or HyperV datastore. With clustered volumes you will have multiple hosts per given volume.

2) Select the “Management” tab, then the “Volume” line: a. Select the “New Volume” button and create a new volume by following the GUI wizard steps. b. In the process select or create a performance profile that matches your volume type, i.e.,

VMware ESX, or matches the block size of the application that will be using the volume. c. Allow access to the volume via an Initiator group and select the initiator group you created in

step #1. d. Do not set a protection scheme at this time, choose “none”

3) Now refer to the appropriate documentation on how to have your Operating system attach, mount, and format this volume.

19

Volume Connection Verification After connecting a volume to a server verify that you are making the proper number of connections. This will help insure that you have your networking and multi-pathing setup properly. In the array navigate to: Monitor >> Connections

1 row for each volume initiator combination Verification Formula:

Note: 2 switches with same VLAN/subnet trunked together is 1 switch 2 switches with same VLAN/subnet NOT trunked is 2 switches VMware Vcenter Plug-In registration: In 1.3.x firmware you can register the Vcenter Plug-In via the GUI. Navigate to >>Administration>>Plugins and fill in the Vcenter host information and credentials.

(Physical hosts * Physical Ports per Host * Array Data Ports)

(Count of subnets * Switches per Subnet )

20

Port use and Firewall considerations (See User Guide for more details)

**Also allow ICMP echo request to nsdiag.nimblestorage.com

21

Support Service Levels Service Levels P1 telephone response less than 30 minutes (current average is <5min); Immediate escalation to Engineering. P2 response less than 4 business hours P3 response less than 8 business hours Severity Definitions P1 – Not serving data; severe performance degradation; single controller not operational P2 – Performance degradation, intermittent SW faults, network degradation P3 – Problem or defect causing minimal business impact; request for information

Popular CLI commands (user name for CLI login is “admin”) • halt –array (shuts down the array in a safe and orderly manner and powers it off) • stats --perf sys --latency –diff (plots out latency) • array –info (lists array configuration information) • vmwplugin --unregister --username adminname --password adminpassword --server vcenterhost

o Un registers the VMware plug-in • Vol –list (lists volumes) • Vol --info volname (Info on volume name given)

22

Shutdown Automation To automate a Nimble Array shutdown you can use this. Created a simple script using private ssh keys, putty and plink.exe for their Windows host. The script just calls this: plink.exe “ssh session name” halt --array

Where the session name is a putty session configured for the Nimble array hostname and private SSH key information.

23

10GbE Cable Detail

Short name is SFP+. Long version is below (from the hardware user guide). SFP specifications The following applies to the "G" version of the CS220 and CS240. These models include 10 GigE ports. About the SR optical transceiver 10GBASE-SR ("short range") uses the IEEE 802.3 Clause 49 64B/66B Physical Coding Sublayer (PCS) and 850 nm lasers. It delivers serialized data over multi-mode fiber at a line rate of 10.3125 Gbit/s. Over deployed multi-mode fiber cabling, it has a maximum range of between 26 metres (85 ft) and 82 metres (269 ft) depending on cable type. Over new 50 μm 2000 MHz•km OM3 multi-mode fiber (MMF), it has a maximum range of 300 metres (980 ft). The transmitter can be implemented with a vertical-cavity surface-emitting laser (VCSEL) which is low cost and low power. MMF has the advantage of having lower cost connectors than SMF because of its wider core. OM3 is now the preferred choice for structured optical cabling within buildings. 10GBASE-SR delivers the lowest cost, lowest power and smallest form factor optical modules. Specifications

As shipped, the CS220G/CS240G include two hot pluggable Finisar SFPs per controller with the following specifications: • Part FTLX8571D3BCL • Family Fiber Optics - Transceivers Series • Data Rate 10Gbps (9.95 to 10.5) • Wavelength 850nmVCSEL laser • Applications 10GBASE-SR/SW 10G • Voltage, per power supply 3.3V • Connector type LC Duplex • Mounting type SFP+ • Power dissipation < 1W • RoHS RoHS-6 compliant • Operating temperature 0°C to 70°C • Maximum link length 300m on 2000 MHZ-km MMF

24

Alternatively customers could unplug the Finisar SFP+ module from the 10gigE port and use copper 10Gigabit Twinax.

Also known as 10GSFP+Cu, 10GBase-CR or 10GBase-CX1, SFP+ Direct Attach uses a passive twin-ax cable assembly and connects directly into an SFP+ housing. SFP+ Direct Attach has a fixed-length cable, typically 3 or 5m in length, and like 10GBASE-CX4, is low power, low cost and low latency with the added advantages of using less bulky cables and of having the small form factor of SFP+. SFP+ Direct Attach today is tremendously popular, with more ports installed than 10GBASE-SR. Below is a picture of a TWINAX cable. It has SFP modules welded on both ends of the cable. One end plugs into the switch; the other end plugs into the Nimble array. The distance you can cover with TWINAX is only 5m (passive TWINAX cable) or 10m with an active TWINAX cable.