m01res01

Copyright © 2013 EMC Corporation. All rights reserved 1 Module 1: Isilon Cluster Architecture

This module focuses on cluster and node architecture, gaining access to the cluster, data protection levels, configuration name resolution, and the SmartConnect service.


This lesson covers OneFS architecture, the differences between scale-up and scale-out NAS, node types, node features, and reviews how to add a node to the cluster.

Copyright © 2013 EMC Corporation. All rights reserved

In traditional NAS scale-up solution, the file system, volume manager, and the implementation of RAID are all separate entities. Each entity is abstracted from the other. The functions of each are clearly defined and separate. In a scale-up solution you have controllers which provide the computational throughput, connected to trays of disks. The disks are then carved up into RAID GROUPS and into LUNs. If you need additional processing you can add an additional controller which can run Active/Active or active/passive. If you need additional disk you can add another disk array. To administer this type of cluster there is an overarching management console that allows for single seat administration. Each of these components are added individually and may have an upper limit of 16TB although some solutions may be higher. This type of solution is great for specific types of workflows especially those applications that require block level access.

In a Scale-out solution the computational throughput, the disk and disk protection, and the over arching management are combined and exist within a single node or server. OneFS creates a single file system for the cluster that performs the duties of the volume manager and applies protection to the cluster as a whole. There is no partitioning, and no need for volume creation. Because all information is shared among nodes, the entire file system is accessible by clients connecting to any node in the cluster. Because all nodes in the cluster are peers, the Isilon clustered storage system also does not have any master or slave nodes. All data is striped across all nodes in the cluster. As nodes are added, the file system grows dynamically and content is redistributed. Each Isilon storage node contains globally coherent RAM, meaning that, as a cluster becomes larger, it also becomes faster. Each time a node is added, the cluster's concurrent performance scales linearly.

3 Module 1: Isilon Cluster Architecture


Redundant Array of Independent Nodes (RAIN) uses individual servers connected via a high speed fabric, and configured with an overlaying management software. RAIN allows for redundancy and data protection in NAS (network attached storage) clustering technology. In a RAIN topology the data is stored, or striped, at the file system level, across the nodes in the cluster. This allows for fault tolerance and redundancy against node and drive failures.

Redundant Array of Independent Disks (RAID) is a technology that stripes data across the physical disks in an array. RAID technology is data protection configured at the hardware level where all of the data stored on a RAID array is protected at the same protection level.



In an enterprise network environment, clients connected to the enterprise network can connect to the resources stored on an Isilon cluster using standard file access protocols. Each node in an Isilon cluster is also connected to a back-end InfiniBand network that enables each to coordinate and continually adjust to the changing needs of the cluster a whole.



The key to Isilon’s storage cluster solutions is the architecture of OneFS, which is a distributed cluster file system. This means that a single file system spans across every node in a storage cluster and, as nodes are added, that file system automatically redistributes content across the entire node. Data redundancy is accomplished by striping data across the nodes instead of the disks so that redundancy and performance are increased. For the purposes of data striping, you can consider each node as an individual device.

There is no single master device that controls the cluster. Each node is a peer that shares the workload and acts as a controller for incoming data requests independently, preventing bottlenecks caused by multiple simultaneous requests. This also prevents outages caused by hardware failures since there is no single controlling interface to the cluster.

For more information, see the OneFS Administration Guide and the OneFS Command Reference Guide.



Isilon file system (/ifs) directory permissions are initially set to allow full access for all users—a setting that is convenient but not secure. Any user can delete any file, regardless of the permissions on the individual file. Depending on your particular environment, this might or might not be acceptable. You may need to change the default configuration by establishing some basic permission restrictions. By default, the SMB and NFS protocols are enabled and a single access point is configured for each. An \ifs share is created by default for Windows clients and an /ifs export is created by default for UNIX users on all Isilon clusters. The default setting is Full Control permissions for newly created folders in the root of /ifs. The default /ifs export is configured to allow clients to mount any subdirectory, which gives end users access without requiring much administration. Two subdirectories are created by default: /ifs/home and /ifs/data. The /ifs/home is created for user home directories and the /ifs/data directory is for other data. Storing all organizational data in /ifs is not recommended. The /ifs export is configured with the Enable mount access to subdirectory option enabled by default. This allows any directory underneath /ifs to be mounted without expressly creating an export for that subdirectory. This assumes that all other required permissions and host rules are properly configured.

If this option is enabled, you can use the following command:

mount –o tcp 10.9.44.11:/ifs/data/eng /mnt/export

The Isilon file system (/ifs) directory permissions are initially set to allow full access for all users. Depending on your particular environment, you may need to change the default configuration by establishing some basic permission restrictions.



Isilon nodes provide the hardware base on which OneFS executes. The hardware is composed of commodity, enterprise-quality, components produced by manufacturers, such as Intel, Hitachi, and SuperMicro. On this commodity hardware base, the OneFS operating system enables data protection, automated data balancing and migration as well as the ability to seamlessly add storage and performance capabilities without system downtime. The hardware includes a high-speed battery-backed NVRAM journal. Each node in an Isilon clustered storage system is a peer, so any node can handle a data request. Also, as each node is added to a cluster, it increases the aggregate disk, cache, CPU, and network capacity of the cluster as a whole. As a result of this aggregate increase, a 100-node cluster can access as much 9.6 TB of globally coherent, shared cache.



OneFS runs on the Isilon hardware nodes. The hardware is comprised of commodity, enterprise-quality, components. On this commodity hardware base, the OneFS operating system enables data protection, automated data balancing and storage tiering, as well as the ability to seamlessly add storage and performance capabilities without system downtime.

The hardware includes a high-speed battery-backed NVRAM. Each node in an Isilon clustered storage system is a peer, so any node can handle a data request. Also, as each node is added to a cluster, it increases the disk, cache, CPU, and network capacity of the cluster as a whole. As a result, a 144-node cluster can access as much 13.8 TB of globally coherent cache.



The Isilon product family consists of three storage node series; the S-Series, the X-Series, and the NL-Series. The S-Series is for ultra-performance primary storage and is designed for high-transactional and IO-intensive tier 1 workflows. The X-Series strikes a balance between large capacity and high-performance storage. X-Series nodes are best for high throughput and high concurrency tier 2 workflows and also for larger files with fewer users. The NL-Series is designed to provide a cost-effective high density solution for tier 3 workflows such as nearline storage and data archiving. The NL-Series provides the lowest TCO for files that are accessed infrequently, but still must be stored and protected. It is ideal for nearline archiving and for disk-based backups.

In addition to the storage nodes, Isilon offers accelerators. There are two types of accelerator nodes: Backup Accelerators and performance accelerator nodes. Performance accelerator nodes add client connection ports, CPU and memory resources to a cluster without additional storage space. Performance accelerators provide 10GbE connections for high-single-stream client connections. The recommended ratio is one performance accelerator node per three storage nodes.

Backup Accelerators enable high-speed backup of file-system data to locally attached tape or media-changer devices. Each Backup Accelerator node can support up to four paths to each locally connected tape or media-changer device via four 4 Gbps Fibre Channel ports. The Backup Accelerator supports a wide range of data management applications (DMAs), tape libraries, and switches.



The current S-Series model are the configure to order (CTO) S200 nodes. Previous S-Series nodes had fixed amount of memory and drive sizes. The S200 enables the customer to customize the amount of memory and drive sizes as and offers SSD drive options and 1 GbE or 10 GbE network interfaces. The S200 has 24, 2.5” SAS or SSD drives.

The X-Series offer CTO models in both the X200, 2U offering and the X400, 4U offering. The X200 model has 12, 3.5” drives to enable the customer to customize the amount of memory, size of drives and SSD drives and 1 GbE or 10 GbE network interfaces. The X400 offers a higher density throughput platform with 36, 3.5” drives.

The NL-Series is a cost effective way to ensure quick access to nearline or archive data. The NL-Series is purpose-built for large capacity storage, enabling nearline storage performance. The Isilon NL-Series is ideal for complementing and dramatically improving existing tape backup, archiving, and data protection strategies and for economical storage and rapid disk-based access to reference data.

Isilon performance-accelerator nodes may be added to an existing cluster to increase overall throughput. For access intensive applications, adding Accelerator node can improve overall performance throughput. The recommended ratio is one Accelerator node per three storage nodes.

The Accelerator-X connects to X-Series storage nodes and adds CPU and memory resources to support a greater client load. It also provides additional 10 GbE connections for high single-stream client connections.

For streaming scenarios Accelerator nodes can provide single-stream uncached reads of 350 MB/sec and concurrent streaming IO up to 700 MB/sec.



This lesson covered OneFS architecture, the differences between scale-up and scale-out NAS, node types, node features, and reviews how to add a node to the cluster.


This lab covers installing the first node in a cluster.


This lab covers joining nodes to the cluster.


This lesson covers internal network connections, name resolution configuration, and an explanation of the SmartConnect service.


Internal networks enable communication between Isilon nodes in a cluster. To enable an internal failover network, the int-a ports of each node in the cluster must be physically connected to one switch, and the int-b ports on each node must be physically connected to another switch. When the following conditions are met, the failover function is automatically enabled:

• Both the int-a and int-b interfaces are configured.

• Separate IP ranges for the int-a, int-b, and failover internal networks are configured.

Each InfiniBand switch may only serve one cluster, and no other devices. Adding other devices or additional clusters to a switch is an unsupported configuration. Chaining switches is also an unsupported configuration.



If you want to name your cluster for ease of management you can use the SmartConnect feature. SmartConnect is a client connection balancing management module that enables client connections to be balanced across all nodes in an Isilon IQ cluster or across selected nodes. It does this by providing a virtual host name for clients to connect to, which greatly simplifies connection mapping. To configure SmartConnect you need to define a SmartConnect Service IP (SSIP) address and a SmartConnect zone name. The SmartConnect service IP address (SSIP) is the IP address used by a Domain Name Service (DNS) server to forward SmartConnect zone lookups to the cluster. A SmartConnect zone defines the name that clients use to connect to the cluster. The SmartConnect zone name is used like a server name when connecting across the network.



The Domain Name System (DNS) is a hierarchical distributed database. The names in a DNS hierarhical form a tree and this tree is called the DNS namespace. There are a set of protocols specific to DNS to allow for name resolution, more specifically, a Fully Qualified Domain Name (FQDN) to IP Address resolution. The top level of the DNS architecture is called the ROOT domain and it represented by a single “.” dot. Below the ROOT domain is the Top Level Domains. These domains are used to represent companies, educational facilities, non-profits, and country codes: .com, .edu, .org, .us, .uk, .ca, etc and are managed by a Name Registration Authority. The Secondary Domain would represent the unique name of the company or entity. EMC, Isilon, Harvard, MIT, etc. The last record in the three is the HOST record, which indicates an individual computer or server.



A Fully Qualified Domain Name (FQDN) is the DNS name of an object in the DNS hierarchy. A DNS resolver query must resolve a FQDN to its IP address so that a connection can be made across the network or the internet. If a computer cannot resolve a name or FQDN to an IP address the computer will not be able to make a connection, establish a session or exchange information.

An example of an FQDN looks like this: student.isilon.training.com

Read from left to right, A FQDN starts with the most specific information, in this case the local computer/desktop name (student), then the delegated domain or sub-domin (isilon), then the secondary or parent domain (training) and lastly, the Top Level Domain (.com).

In DNS a FQDN will have an associated HOST or A record mapped to it so that the server can return the corresponding IP addresss.

student.isilon.training.com A 192.168.0.31



DNS Name Resolution and Resolvers

When a client needs to resolve a Fully Qualified Domain Name (FQDN) it follows the following steps:

1 - The client will look in its’ local cache to see if it has already done a lookup for that host or FQDN. If it has, it will cache the hosts resource record also known as an A or AAAA (quad A) record, and will use the name-to-IP mapping that sits in its’ local cache.

2 - If there is not an entry in the local cache the computer will make a call to the DNS server configured within the OS. This request is called a resolver or resolver query. The request asks the DNS, “Do you know the IP address of sales.isilon.training.com.

3 - The DNS server that receives the request will check it’s local zones to see if they contain a zone for isilon.training.com. If it has a copy of the zone (all of the DNS entries for a particular secondary domain) it will query the zone for the hosts A or AAAA record and return the host-to-IP mapping to the client. An A record looks like this: cluster.isilon.training.com A 192.168.0.31.

4 - The DNS server returns the IP to the client, who caches the information, and then attempts to make a connection directly to the IP address provided by the DNS server.



Delegation in a DNS hierarchy provides the administrator with the option to divide the namespace into one or more sub-zones or sub-domains. This is done if there is a need to delegate the management and administration of a portion of the namespace to another department or physical location. For example if you are a Seattle based company and you open a new location in Boston and want the administrators in Boston to be responsible for their own DNS zone. You could delegate control of BOSTON.ABCCOMPANY.COM



By integrating DNS and Smartconnect, clients can access different groups of nodes in your cluster for either performance isolation or workflow requirements.



SmartConnect enables you to configure the cluster to respond to name resolution requests. After you configure SmartConnect, a requesting client system views the cluster as a single network element. Both cluster and client performance are enhanced when connections are more evenly distributed. Even in its basic implementation, it can remove nodes that have gone offline from the request queue and prevent new clients from mounting a node that is down.



Using SmartConnect zones, cluster nodes/specific interfaces within a node can be added to a zone, allowing a more granular control of where a connection is directed. SmartConnect Basic supports only one SmartConnect zone per subnet, while SmartConnect Advanced supports multiple SmartConnect zones per subnet.

Because each SmartConnect zone is managed as an independent SmartConnect environment, they can have different attributes, such as the Client Connection policy. For environments with very different workloads, this provides flexibility in how resources in the cluster are allocated. In the example here, two SmartConnect zones have been configured, a performance zone and a general use zone. Clients use one DNS name to connect to the performance zone nodes and another to connect to the general use nodes. The performance zone could use CPU usage as the basis for distributing client connections, while the general use zone could use Round Robin or connection count, which will optimize the allocation of cluster resources based on the client requirements and workloads.

Additional tools for configuration options include the ability to zone by node interface and the ability to Include/Exclude any particular node from participating. Connection balancing policies can be customized for each zone by defining which nodes will participate and what balancing policy to apply.



This lesson covered internal network connections, name resolution configuration, and an explanation of the SmartConnect service.


This lesson covers data breakdown and an explanation of the N+M protection approach.


OneFS stripes data across nodes and disks. During a write, the system breaks data into smaller logical sections called stripes and then logically places the data in a stripe unit. As the system lays data across the cluster, it fills the stripe units until the maximum width of the cluster is reached. Each OneFS block is 8 KB, and a stripe unit or stripe unit consists of 16 blocks, for a total of 128 KB per stripe unit.

OneFS uses advanced data layout algorithms to determine data layout for maximum efficiency and performance. Data is evenly distributed across nodes as it is written. The system can continuously reallocate data and make storage space more usable and efficient. Depending on the file size and the stripe width (determined by the number of nodes), as the cluster size increases, the system stores large files more efficiently.

Within the cluster, every disk within each node is assigned both a unique GUID and logical drive number and is subdivided into 32MB cylinder groups comprised of 8KB blocks. Each cylinder group is responsible for tracking, via a bitmap, whether its blocks are used for data, inodes or other metadata constructs. The combination of node number, logical drive number and block offset comprise a block or inode address and fall under the control of the aptly named Block Allocation Manager (BAM).

Module 1: Isilon Cluster Architecture 27


In OneFS, protection is calculated on individual files. To calculate protection individual files are logically broken down into 128 KB stripe units. Stripe width is the number of stripe units that you can create before you need to create a protection stripe unit. Each file is broken down into smaller 128 KB stripes units, then protection is calculated for the file and protection stripe units are created. The data stripe units and the protection stripe units together form a stripe. Stripe units are sent to individual nodes in the cluster. As a result, when a file is needed, multiple nodes in the cluster are able to deliver the data back to the requesting user or application. This dramatically improves overall performance, especially when hundreds, and even thousands, of these requests are made simultaneously from an application. Due to the way in which OneFS applies protection, files that are 128 KB in size or smaller are actually mirrored.

OneFS does not use RAID to protect cluster data. OneFS uses the Reed-Solomon algorithm for N+M protection. Reed-Solomon is used because it is an industry standard that enables data to have very high protection levels. In the N+M data protection model, N represents the number of data stripe units and M represents the number of simultaneous failures of nodes or drives—or a combination of nodes and drives—that the cluster can withstand without incurring data loss. M also equals the number of protection stripe units that are created within a stripe. N must be larger than M. For many N+M protection levels, there are no RAID equivalents. On an Isilon cluster you can enable N+1, N+2, N+3, or N+4 protection, which allows the cluster to sustain two, three, or four simultaneous failures without resulting in data loss.


Copyright © 2013 EMC Corporation. All rights reserved 29

Data on an Isilon cluster is separated in to 128 KB data stripe units. FEC calculations create protection stripe units that is used to reconstruct user data in the event of a drive or node loss. FEC protection level is represented as N+M, where N is the number of data stripe units, and M is the number of protection or FEC stripe units. These stripe units together form a data stripe. In N+M, the number of data stripe units must be always larger than the number of FEC stripe units. So the highest N+M protection level available on a five node cluster is N+2. The data stripe on the five node cluster with a 384 KB file and N+2 protection, there are three data strip units and two protection or FEC stripe units created. Isilon striped data across nodes rather than across the disks in single node.

Isilon supports FEC protection levels from N+1 to N+4.

Module 1: Isilon Cluster Architecture

Copyright © 2013 EMC Corporation. All rights reserved 30

In addition to supporting the N+M protection, Isilon supports an N+M:B option. With N+M, M represents the number of drives or nodes that can fail with no data loss. Because the likelihood of a node failure is significantly lower than that of a drive failure, the :B option separates the drive failure protection level from the node failure protection level. Using N+M:B, M is the number of drive failures that can be tolerated with no data loss and B is the number of node failures that can be tolerated with no data loss. So, in a N+2:1 configuration, the system can tolerate two drive failures or 1 node failure. N+2:1 is the default protection level for an Isilon cluster. This option provides a significant amount of capacity savings by reducing the amount of protection information that is needed.

Isilon supports protection levels of N+2:1 and N+3:1 in addition to the N+M options.

Module 1: Isilon Cluster Architecture


Isilon supports different methods of data protection. The first method is mirroring. Mirroring creates a duplicate copy of the data being protected, similar to RAID-1 on traditional storage systems. However the OneFS operating system supports multiple mirror copies of the data. In fact, it is possible to create up to 7 mirrors of the data on a single cluster. Mirroring has the highest protection overhead in disk space consumption. However, for some types of workloads, such as NFS datastores, mirroring is the preferred protection option.

The primary protection option on an Isilon cluster is known as Forward Error Correction (FEC). The Isilon system uses the Reed-Solomon algorithm, which is an industry standard. FEC works much like RAID-5, in that it generates protection data blocks and stores it them separately from the data blocks. OneFS can support protection levels of up to N+4. The data can be protected with a N+4 scheme, where up to 4 drives, nodes or a combination of both can fail without data loss.

One of the key differences between how Isilon protects data and traditional RAID systems lies in the flexibility of protection. In RAID systems, the protection is applied at the physical disk level and all data is protected identically. Isilon allows you to define protection level at the node pool (group of similar nodes), directory or even individual file level, and have multiple protection levels configured throughout the cluster. This feature makes it possible to match the protection level to the value of the data.

In addition to protecting file data, the Isilon system protects the metadata associated with that data. The metadata is protected at one level higher than the data using metadata mirroring. So, if the data is protected at N+3, then the metadata is protected at 4X.



When you protect data from hardware failure, you lose an amount of disk space to protection. The protection overhead depends on the protection setting, the file size and number of nodes in the cluster. The percentage of protection overhead for an Isilon cluster declines as the cluster nodes are added.

The reason that the overhead declines as the number of nodes grows is because the stripe width increases as the number of nodes increases. Stripe width is the number of stripe units that are in a stripe. The maximum stripe width in an Isilon cluster is 20 stripe units, up to 16 of which can be data stripe units and up to 4 of which can be protection stripe units. Since a large file is broken down into data and protection stripe units, files that are larger than 2 MB (16 X 128 KB) need more than one stripe.

For example:

N+1 at 3 nodes = 2+1 (max stripe width of 3)






The protection overhead for each protection level depends on the file size and the number of nodes in the cluster. The percentage of protection overhead declines as the cluster gets larger. In general, +1 protection has a protection overhead equal to one node's capacity, +2 protection has a protection overhead equal to two nodes' capacity, +3 is equal to three nodes' capacity, and so on.

OneFS also supports optional data mirroring from 2x-8x, allowing from two to eight mirrors of the specified content. Data mirroring requires significant storage overhead and may not always be the best data-protection method. For example, if you enable 3x mirroring, the specified content is explicitly duplicated three times on the cluster; depending on the amount of content being mirrored, this can require a significant amount of capacity.



This lesson covered data breakdown and an explanation of the N+M protection approach.


This lesson covers cluster access via web the administration interface, application of the OneFS CLI, OneFS command structure, and common isi commands.


The new web administration interface provides enterprise-class functionality, with a clean look and intuitive navigation. The web administration interface is role-aware, and adapts to the privileges of the logged in user. The features for which the user does not have permission are not available (greyed out).



Once a user has been assigned to a role, all administrative interfaces, including the web administration interface, recognize the privileges of the logged in user. Notice in the screenshot on the slide that this user only has privileges to NFS and SMB. The navigation for all other protocols are greyed out and unavailable to this user.



Isilon has a GUI, the Isilon web administration interface. Accessing the administration application requires that at least one IP address is configured on one of the external Ethernet ports on one of the nodes. The Ethernet port IP address must be configured either manually or by using the Configuration Wizard. To access the web administration interface from another computer, an internet browser application must be installed and used on that computer.

To use the web administration interface, you must be either root, admin or be a member of a role which has the ISI_PRIV_LOGIN_PAPI privileges assigned to it, which is discussed later.



A specific URL is needed to access the web administration interface. The URL is https://clustername:8080, where clustername is the SmartConnect/DNS name of the cluster. If SmartConnect is not yet configured, specify an external IP address of a node instead. The web administration interface can be accessed from any node.

The web administration interface enables an administrator to monitor the status of the cluster, configure networking, setup and configure Isilon’s suite of software modules, configure file sharing access and other cluster parameters. You can navigate within the administration interface by clicking the menus and tabs, then selecting an option to view or modify on a page.

The Dashboard has the two sub-menus, Cluster Overview and Events. Under Cluster Overview there are three tabs: Cluster Status, Client Connections and Throughput Distribution Charts.

On the Cluster Status page, there are several sections that provide an overview of the status of the cluster. On the Client Connections page, there are two sections: Current Connections and Connections per node. On the Throughput Distribution page, the file system throughput displays, which is the total data input and output per node.

The Events section displays cluster events and has there are four tabs: Summary, Event History, Event Notification Rules, Notification Settings. The Summary tab displays New Events and Quieted Events.

The Events History section displays a history of cluster events. Event notification rules are configured on the Event Notification Rules page. On the Notification Setting page, you can Send Test Events, configure SupportIQ parameters, enter Email Settings and Contact Information for the cluster.



To initially configure an Isilon cluster, the CLI must be accessed by establishing a serial connection to the node designated as node 1. The serial port is usually a male DB9 connector. This port is called the management port. Connect a serial null modem cable between a serial port of a local computer, such as a laptop, and the management port on the node designated as node 1. As most laptops today no longer have serial connections, you might need to use a USB-to-serial converter. On the local computer, launch a serial terminal emulator, such as PuTTY. Configure the terminal emulator utility to use the following settings:

• Transfer rate = 115,200 bps

• Data bits = 8

• Parity = none

• Stop bits = 1

• Flow control = hardware

Either a command prompt or a Configuration Wizard prompt will appear. The command prompt displays the cluster name, a dash (-), a node number, and either an hash (# ) symbol or a percent (%) sign. If you log in as the root user, it will be a # symbol. If you log in as another user, it will be a % symbol. For example, Cluster-1# or Cluster-1%. This prompt is the typical prompt found on most UNIX and Linux systems.



There are two primary methods for adding a new node to an existing cluster. The first method is, after you have connected the InfiniBand and power cables to the new node, log on to the node by establishing a serial connection. Connect a null modem serial cable to the serial port of a local computer, such as a laptop. Then, connect the other end of the serial cable to the serial port on the back panel of the node. Next, launch a serial communication utility such as Minicom for UNIX clients and Tera Term or HyperTerminal for Windows clients and configure the connection utility to use the following settings:

• Transfer rate = 115,200 bps

• Data bits = 8

• Parity = none

• Stop bits = 1

• Flow control = hardware

After you connect to the node, press ENTER to display the prompt shown. To join the node to the cluster, type two, and then press ENTER. A list of available clusters displays. From the list, type the number of the cluster that you want to join, and then press ENTER.



You can also join a node to a cluster using the LCD control panel that is located on the front panel of the node. When an unconfigured node is powered on, the LCD control panel prompts you to select a cluster to join. Use the scroll up and scroll down buttons to select the existing cluster you want the node to join. When the cluster you want to join displays, press the center button to accept the selection. The node join should take less than sixty seconds. There should only be one cluster to join. If there is more than one, contact Isilon support.



Isilon clusters use an active-active InfiniBand internal network configuration. When a node joins the cluster, either during initial configuration or after a reboot, the node pings both InfiniBand switches. The first switch to respond is set as the node’s primary switch. The other switch then becomes the secondary and failover switch.

When a node initiates a communications session with other nodes in the cluster, the primary switch is used to communicate with the other nodes. In the illustration Node 1 has Internal Switch A set as its primary switch. When a client attached to Node 1 writes a file or requests a read of an existing file, the communications for that transaction occurs using Internal Switch A. A node can respond to any communications request over either InfiniBand switch. A communications transaction remains on the same switch path.

To test the node to switch interface assignment run the isi_eth_mixer_d showprimary command. The output will list the node numbers and the primary InfiniBand interface assigned to each node.

Module 1: Isilon Cluster Architecture 43


When you add new nodes to a cluster, the cluster gains more CPU, memory, and possibly disk space. Accelerator nodes don’t add storage, but do add other capabilities. As nodes are added to the cluster, the nodes are assigned a node number and an IP Address. They are numbered in the order they are added to the cluster. If a node attempts to join the cluster with a newer or older OneFS version the cluster will automatically reimage the node to match the cluster’s OneFS version. After this reimage completes, the node finishes the join. A reimage should not take longer than 5 minutes, which brings the total amount of time taken to approximately 10 minutes. For clusters that use a OneFS version prior to 5.5.x, do not join the node to the cluster. First, reimage the node to the same OneFS version as the cluster before joining the node.



The OneFS CLI extends the standard UNIX command set to include additional commands that enable users to manage an Isilon cluster outside of the web administration interface. The CLI can be accessed by opening a secure shell (SSH) connection to any node in the cluster. This can be done by root or any user with the ISI_PRIV_LOGIN_SSH privileges which is discussed later.

Command references are available and provide an alphabetical list of isi commands that can be run to configure, monitor, and manage an Isilon clustered storage system and the individual nodes in a cluster. The most common isi commands are shown. Please refer to the command reference for their usage and description, use the man isi page, or use isi -- help, for more information.



OneFS can be accessed through both a graphical user interface (GUI) called the Isilon web administration interface and the CLI. To access the CLI out-of-band, a serial cable is used to connect to the serial port on the back of each node. CLI can also be accessed in-band once an external IP address has been configured for the cluster. Both ways are done using a terminal emulation application such as PuTTY.

The CLI can be used to do many things, among them running the Configuration Console, which includes the settings that were first configured during the Configuration Wizard. The CLI can also be used to view and change configuration settings for individual nodes and the cluster.



The man isi or isi --help command is probably the most important command for a new administrator. It provides an explanation of the many isi commands available. You can also view a basic description of any command and its available options by typing its name followed by the -h option at the command line: Command -h.

To view more detailed information at the command line, refer to the isi man page: man isi Command or the Isilon OneFS Version 7.0 Command Reference.



The isi config command opens the Configuration Console. The Configuration Console opens a new prompt in which many of the settings first configured in the Configuration Wizard are available. The command-line prompt changes to indicate that you are in the isi config subsystem. While you are in the isi config subsystem, other OneFS commands are unavailable and only isi config commands are valid. This configuration console allows configuration parameters to be used inside. When you are in the isi config subsystem, the syntax for each option is simplified by only requiring the individual command to be typed.

Once you have made all the necessary configuration updates, a commit command must be executed save the changes. If a reboot is required to activate the configuration, a prompt will display requiring confirmation before the reboot will occur.



The isi config command opens the Configuration Console where node and cluster settings can be configured. When in the isi config console, other Isilon configuration commands are unavailable and only isi config commands are valid. The Configuration Console contains settings that are configured during the Configuration Wizard that ran when the cluster was first created. After you make all the necessary configuration updates to the cluster they are saved and you are prompted if you need to reboot the cluster as needed. The changes command displays a list of changes to the cluster configuration that are entered into the Configuration Console, but have not been applied to the system yet. ========================================

changes deliprange help ipset migrate netmask remove timezone

commit encoding interface joinmode mtu quit shutdown version

date exit iprange lnnset name reboot status wizard

For example joinmode [<mode>]

Displays the current cluster add node setting, when executed without any argument.

Sets the cluster add node setting, when appended with one of the following arguments:

- manual: Configures the cluster to add new nodes in a separate, manually executed process.

- secure: Configures the cluster to disallow any new node from joining the cluster externally. It also makes some other aspects of the operation more secure.



The isi get command displays information about a set of files, including their protection policy, current protection level, and whether write-coalescing (SmartCache) is enabled or disabled. It can provide file information down to the individual disk on which a file is stored in a node.

For example:

isi get -a /ifs/README.txt

POLICY LEVEL PERFORMANCE COAL FILE

default 8+2/2 concurrency off README.txt

isi get -D /ifs/README.txt

POLICY W LEVEL PERFORMANCE COAL ENCODING FILE IADDRS

default 8+2/2 concurrency off UTF-8 README.txt <2,1,1075423668617216:512>, <5,24,35427339747328:512>, <6,10,128069884985344:512> ct: 1242756543 rt: 0



The isi networks command displays the internal and external network interfaces and pool membership. First time cluster configuration requires the storage administrator to create internal and external network connections. The command line configuration wizard navigates you through this task. Once the cluster has rebooted with the finalized network information, the storage administrator can use the isi networks command structure to create, modify, or view external network interfaces: isi networks: Provides a means to manipulate the external network settings as needed.

isi networks create subnet –n=”name” –g=”gateway IP Address”: Creates the new external subnet connections to the cluster.

isi networks create pool –n=Subnet0:Pool0: creates a pool in a configured cluster subnet. Which will create a pool called “Pool0” in Subnet named “Subnet0”

isi networks modify pool –n=Subnet0:Pool0: Modifies the configuration of a pool in a configured cluster subnet. Which will create a pool called “Pool0” in Subnet named “Subnet0”.

Note: typically the cluster at first configuration will configure Ext-1 (1GE) interfaces under subnet0:pool0, the names can be changed manually or by using the Web GUI.

isi networks --sc-rebalance-all: Manually rebalances client connections across the cluster after a previously unavailable node has come back online.

isi networks --dns-servers=: Sets the list of DNS servers used by the cluster.

It is recommended to capture the output of the commands before making changes as a backup information , and dump them into a text file :

isi networks list subnet –v : to list the subnet configuration(s)

isi networks list pool -v : to list the pool configuration(s)

isi networks list ifaces –v: Displays a list of configured external interfaces and basic internal network information.



The isi smb and isi nfs commands are used to view cluster and node activities around client connections. The syntax for some of these commands has changed slightly from previous versions of OneFS.

isi smb settings global: Configures global smb parameters.

isi smb settings shares list: Displays a list of share configuration parameters.

isi smb openfiles: Displays information on open files, including their file IDs.

isi smb shares permission list: Displays a list of existing permissions on a folder or file.

isi smb sessions list: Displays a list of existing open sessions.

isi smb shares create: Creates a new SMB share.

isi smb shares list: Displays a list of currently created shares.

isi nfs nlm sessions list: Displays a list of all clients holding NLM locks.

isi nfs exports list -v -r: Displays detailed information on existing export rules, their default settings and see whether clients are resolved to them.

isi nfs locks list: Displays a list of current locks.

isi nfs netgroup: Displays a list of configured netgroups. This is a new command available in OneFS 7.0.



The isi services command provides a mechanism to view and modify the state of various cluster system services. The syntax for this command is isi services [-la] [service [enable|disable]]. The services command provides a mechanism to view and modify the state of various system services.

The -l parameter includes all services that the user may change the state of along with a short description of each and their current state. This list may be augmented to include all services with the -a flag. Alternatively, the status of a particular service is displayed when the name of that service is specified with the command. If enable or disable is specified along with the name of a service, the specified service will be brought to the new state, if possible.



Three main commands that enable you to view the cluster from the command line are isi status, isi devices, and isi statistics. The isi status command displays information on the current status of the cluster, alerts, and jobs. To view information on the cluster, critical events, cluster job status, and the basic identification, statistics, and usage run isi status at the CLI prompt.

The isi devices command displays information about devices in the cluster and changes their status. There are multiple actions available including adding drives and nodes to your cluster.

The isi statistics command has approximately 1500 combinations of data you can display as statistical output of cluster operations.



Nodes are identified by two different numbers: node ID and logical node number (LNN). When a node joins a cluster, it is assigned a unique node ID number, for example, ID1. Node ID numbers are never repeated or duplicated in a cluster, and they never change. If a node is replaced with a new node, the new node is assigned a new node ID by the cluster. Since each node ID is always unique, when seen in cluster logs individual nodes are easily identified.

A node’s LNN is based on the IP address assigned to the node’s internal network interface. The LNN of a node can change if the IP address of the node changes, or if a new node replaces a node that was removed from the cluster. The LNN of a node displays in the output of the isi status command. In logs, the LNN displays with the name of the node, for example: clustername-1. The name and LNN also display in the output of the isi nodes command.

You can change an LNN in the configuration console for a cluster. To open this console, at the command-line interface type isi config, and then press ENTER. At the configuration console prompt, type renum <OldNode#> <NewNode#>.

The LNN represents the relative position of the node's back end IP address within the range. For example, given a range of 192.168.1.2 through 192.168.1.9, a node with an IP address of 192.168.1.2 has a LNN of 1, a node with an IP address of 192.168.1.9 has a LNN of 8, and so forth, regardless of what other IP addresses are in use.



Disk drives and drive bays in a node have different numbering schemes. In a node, drive bays are numbered from left to right and top to bottom. This means that the drive bay that is in the upper left hand corner of the back of the node is Bay 1. Since the drive bays are fixed physical bays in a node, bay numbers do not change. In contrast, a disk logical numbers (Lnum) are tied to the physical bay in the node by a different scheme. The last column is a logical number that is assigned by OneFS to a drive as it becomes available to the cluster. To display the drive bay numbers and the Lnum of each disk in a node run the isi devices –d command.



The isi statistics command provides a set of cluster and node statistics. The statistics collected are stored in a database that is in the /ifs folder on the cluster. Additionally, other Isilon services such as InsightIQ, the web administration interface, and SNMP gather needed information using the isi statistics command.

The isi_stat_d command enables you to view cluster throughput based on connection type, protocol type, and open files per node. You can also use this information to troubleshoot your cluster as needed.



Some isi statistics parameters include the following list:

• isi statistics system --top --nodes --interval=2

• isi statistics protocol --classes=read,write,namespace_read,namespace_write

• isi statistics client --remote_names =“<IP Address>”

• isi statistics protocol heat --classes=read,write

• isi statistics drive --nodes=all --orderby=timeinq

You can use the isi statistics command within a cron job to gather raw statistics over a specified time period. A cron job can run on UNIX-based systems to schedule periodic jobs.



The isi statistics command with no additional parameters.



To get more information on isi statistics, run man isi statistics from any node.

• To display usage help

isi statistics system --help

isi statistics protocol --help

isi statistics client --help

Isi statistics drive --help

• isi statistics list stats

Can list over 1500 statistics

Dumps all collected statistics

Useful when you want to run the query subcommand on a specific statistic

Can be used to build a custom isi statistics query that is not included in the provided subcommands (such as drive, protocol, etc.)



First time directory services configuration requires you to connect directory services to the cluster. You can perform this through the web administration interface, or use the isi auth command to create, modify, or view the commands below:

isi auth ads join --domain=<DNSDomain> --user=<string> --password=<Password>: Joins cluster to an Active Directory domain. If the command is run while joined to a domain, the cluster is rejoined. This command is not atomic. Failure could result in lost domain connectivity.

isi auth ads accounts list: Displays Active Directory machine accounts available to the cluster.

isi auth list --ldap-: Configures the LDAP service parameters.

isi auth ads spn create --spn=<SPN>: Adds an SPN for a machine account. An SPN is the name by which a client uniquely identifies an instance of a service. The Kerberos authentication service can use an SPN to authenticate a service. When a client connects to a service, it locates an instance of the service, composes an SPN for that instance, connects, and presents the SPN to authenticate.

isi auth ads users list: Displays a list of users on Active Directory domains.

isi auth ldap modify: Configures LDAP attributes for non-standard directories.

isi auth ldap list: Lists current LDAP attribute configuration settings.

isi auth status: Displays the current status of authentication providers.

isi auth local groups create: Creates a local group on the cluster.

isi auth local users create: Creates a new local user account on the cluster.

isi auth local users list: Displays a list of local users on the system. If no options are specified, the system displays all local users.



This lesson covered cluster access via web the administration interface, application of the OneFS CLI, OneFS command structure, and common isi commands.


This lab covers the Isilon cluster CLI


This module focused on cluster and node architecture, gaining access to the cluster, data protection levels, configuration name resolution, and the SmartConnect service.

m01res01

Documents

type of cluster

node architecture

single node

isilon storage node

node types

node features

single file system

disk protection