network attached storage architecture

7/28/2019 Network Attached Storage Architecture

1/9

COMMUNICATIONSOF THE ACM November 2000/Vol. 43, No. 11 37

SAN with Fibre Channel network hardware that hasa greater effect on a users purchasing decisions. Thisarticle is about how emerging technology may blurthe network-centric distinction between NAS andSAN. For example, the decreasing specialization ofSAN protocols promises SAN-like devices on Ether-

net network hardware. Alternatively, the increasingspecialization of NAS systems may embed much ofthe file system into storage devices. For users, it isincreasingly worthwhile to investigate networkedstorage core and emerging technologies.

Today, bits storedonline on magnetic

disks are so inexpen-sive that users are find-ing new, previouslyunaffordable, uses forstorage. At DataquestsStorage2000 confer-ence last June inOrlando, Fla., IBMreported that online disk storage is now signifi-cantly cheaper than paper or film, the dominanttraditional information storage media. Not surpris-ingly, users are adding storage capacity at about100% per year. Moreover, the rapid growth ofe-commerce, with its huge global customer base

and easy-to-use, online transactions, has introducednew market requirements, including bursty, unpre-dictable spurts in capacity, that demand vendorsminimize the time from a users order to installationof new storage.

In our increasingly Internet-dependent business and computing

environment, network storage is the computer.

NETWORK ATTACHEDSTORAGE ARCHITECTURE

THE GROWING MARKET FOR NETWORKED STORAGE IS A RESULT OF THE EXPLODING DEMAND

FOR STORAGE CAPACITY IN OUR INCREASINGLYINTERNET-DEPENDENT WORLD AND ITS TIGHT

LABOR MARKET. STORAGE AREA NETWORKS (SAN) AND NETWORK ATTACHED STORAGE

(NAS)ARE TWO PROVEN APPROACHES TO NETWORKING STORAGE. TECHNICALLY, INCLUDING

A FILE SYSTEM IN A STORAGE SUBSYSTEM DIF-

FERENTIATES NAS, WHICH HAS ONE, FROMSAN, WHICH DOESNT. IN PRACTICE, HOW-

EVER, IT IS OFTEN NASS CLOSE ASSOCIATION

WITH ETHERNET NETWORK HARDWARE AND

Garth A. Gibson and Rodney Van MeterATOMIC-FORCEMICROSCOPESC

ANO

FARECORDEDB

ITONA

DISK'SPHASECHANGE

MATERIALCOATING,

SIROSTECH

NOLOGIES,

SANJ

OSE,

CA


2/9

38 November 2000/Vol. 43, No. 11 COMMUNICATIONS OF THE ACM

The rapid increase in theneed for online capacityfuels other business tech-nology trends. First, capitalinvestment in storage is

now more than 50% of allcapital investments in cor-porate data centers. Indus-try analysts predict thispercentage will reach 75%by 2003. Second, personnelcosts for storage manage-ment (for, say, tuning per-formance and maintainingbackups) now dominatecapital costs over the equip-ments useful lifetime. Ven-dors estimate this recurringcost to be as much as $300

per gigabyte per year; thatis, each years recurring costis comparable to, and oftenexceeds, its one-time capi-tal-cost counterpart. Coupled with the shortage ofinformation technology professionals and the bursty,unpredictable capacity needs of e-commerce, it istherefore not surprising that a new market hasemerged for data-center outsourcing for both com-plete applications (application service providers) andstorage systems only (storage service providers).Third, the increasing cost of storage management,coupled with the continuing decline in the cost of

storage capacity, has prompted analysts to predict thatby 2005 the primary medium for storage backup willbe online hard disks.

Various system properties are improved by separat-ing storage from application servers and clientmachines and locating it on the other side of a scal-able networking infrastructure (see Figure 1). Net-worked storage reduces wasted capacity, the time todeploy new storage, and backup inconveniences; italso simplifies storage management, increases dataavailability, and enables the sharing of data amongclients.

It reduces wasted capacity by pooling devices andconsolidating unused capacity formerly spread over

many directly attached storage servers. It reduces thetime needed to deploy new storage, because clientsoftware is designed to tolerate dynamic changes innetwork resources but not the changing of local stor-age configurations while the client is operating. Databackup traditionally occupies application data serversand hosts for much of the night, a significant incon-venience for global and always-open organizations.

With networked storage, backup can be made lessinconvenient, because data can be transferred tooffline tape storage when the devices are not busyday or nightwithout application-server involve-ment. Networked storage also simplifies storagemanagement by centralizing storage under a consoli-dated manager interface that is increasingly Web-based, storage-specific, and easy to use.

Inherent availability, at least in systems in which all

components are provided by the same or cooperatingvendors, is improved, because all hardware and soft-ware in a networked storage system is specificallydeveloped and tested to run together. Traditionalservers are more likely to contain unrelated hardwareand software added and configured by users. Suchuser additions can be destabilizing, because all com-ponents are unlikely to have been sufficiently testedtogether, resulting in more frequent crashes. Finally,the sharing of data among clients is improved,because all network clients can access the same net-

worked storage.The principal disadvantages of networked storage

are due to the increased complexity of systems spread

across a network. More machines have to functioncorrectly to get work done; network protocol process-ing is more expensive than local hardware deviceaccess; and data integrity and privacy are more vulner-able to malicious users of the network. While networkstorage technology must address these disadvantages,all distributed and Internet applications share them;hopefully, network storage technology will share their

Figure 1. Networked storage versus direct-attached storage.

Network

Direct-attached storage

Networked storage

Networked storage

clients


3/9

solutions, as well as their problems.Most networked storage systems fall into one of

two technology families: NAS systems (such as theF700 file-server systems from Network Appliance ofSunnyvale, Calif.) typically accessed via Ethernet net-

works; and SAN systems (such as the Symmetrix diskarray from EMC of Hopkinton, Mass.) typicallyaccessed via Fibre Channel networks [1]. Both NASand SAN storage architectures provide consolidation,rapid deployment, central management, more conve-nient backup, high availability, and, to varyingdegrees, data sharing. It is therefore no surprise that anIT executive might view NAS and SAN as solutions

to the same problems and the selection of networkinginfrastructure as the principle differentiator amongthem. The technology trends we discuss here are likelyto blur this simplistic, network-centric differentiationbetween NAS and SAN, so we begin with the princi-ple technological difference.

NAS and SAN InterfacesTechnologically, NAS and SAN systems offer differ-ent abstractions to the software using them [1, 2]. Atits simplest, the difference between a NAS storageabstraction and a SAN storage abstraction is thatNAS systems offer file system functionality to theirclients and SAN systems do not.

SAN systems provide a simple, untyped, fixed-size(block), memory-like interface (such as getblock, set block) for manipulating nonvolatilemagnetic media. From the perspective of a datapathabstraction, there is little functional differencebetween the interfaces of a SAN system and a tradi-tional attached disk. Even though a SAN networkmakes it possible for multiple clients to access the

same storage device directly, thistype of sharing requires coordina-tion not presented by the SANinterface to ensure that concurrentaccess is synchronized.

Here, we group together thestorage and the storage-area net-work hardware to which the stor-age is attached, referring to themcollectively as the SAN system. Inthis sense, a Fibre Channel disk isa SAN device. A more narrowinterpretation of SAN includesonly Fibre Channel hubs,switches, and routers. In the fol-lowing sections, we use the termSAN systems to mean SAN-attached storage systems.

NAS systems provide a richer,

typed, variable-size (file), hierarchical interface(includingcreate or delete file, open orclose file, read or write file sub-set, get or set file attribute, cre-ate or delete directory, insert orremove link between directory and

file or directory, and lookup filename). NAS systems internally interface with non-volatile magnetic media, usually through a SAN-likeinterface to ensure data reliability. From the perspec-tive of a datapath abstraction, there is little functionaldifference between the interfaces of a NAS system andthose of a traditional local file system.

Figure 2 outlines the relationship between the NASand SAN interface abstractions, using as an example asingle user file containing a presentation slide image.On the left, a users document is preserved betweenediting sessions in a file whose local name ismyfile. Stored in a file system, this file has beenlocated in a collection of files (a directory), calledbob in the example. Uniquely naming the users filerequires listing its name and its ancestor directoriesnames, /bob/myfile in the example. As a side-effect of its creation, myfile is labeled with suchattributes as the date of its creation and the amount ofstorage used by its contents. In the example, theseattributes are November 2, 2000 and 30 KB,

respectively.The diagram in the middle of Figure 2 outlines

how myfile might be represented in the datastructure of a Unix-like file system [8]. There are fourdifferent types of uses for fixed-size blocks in this datastructure: user data, directory entries, file descriptors,and indirect pointers. All data structures other thanthe user data are known as metadata and must be con-


m vel eum irure dolor in

prehenderit in voluptate ve

ielt esse molestaie son cons

equat vel illium dolore

t nulla pariatur.Lorem ipse

m sit amet,consecte

pscing elit,sed diam nonu


t nulla pariatur.Lorem ip

m sit amet,consectetur ad

ipscing elit,sed diam nieltesse molestaieson cons

ipscingelit,sed diam n

my eisand tempor inciden

derit in voluptate veielt oriejjgkg

quat vel illium dolore fugikfmrjl

at nulla pariatur.Lorekdkfjff

sem sit amet,consetetur

adipscing elit,sed di xlssjjasjs

This 30KB slide isstored in myfile,written on Nov. 22000, and locatedin directory/bob.

Abstract file object

NAS SANNAS

File system data structures(files and directories)

Nonvolatile storage volume(blocks)

"myfile"

"bob"

NASD


ipscing elit,sed diam nmy eisand tempor inciden

NASD


ipscing elit,sed diam nmy eisand tempor inciden

NASDm vel eum iruredolor inprehenderitin voluptateveieltessemolestaieson consequatvel illium doloretnullapariatur.Lorem ipsem sitamet,consecte

pscingelit,sed diam nonu

ieltessemolestaieson cons

tnullapariatur.Lorem ipm sitamet,consectetur adipscingelit,sed diam n

deritin voluptateveielt

quatvel illium dolorefugiatnullapariatur.Lore

sem sitamet,conseteturadipscingelit,sed di

deritin voluptateveielt

quatvel illium dolorefugiatnullapariatur.Lore

sem sitamet,conseteturadipscingelit,sed di

ieltessemolestaieson cons

tnullapariatur.Lorem ipm sitamet,consectetur adipscingelit,sed diam n

m vel eum iruredolor inprehenderitin voluptateveieltessemolestaieson consequatvel illium doloretnullapariatur.Lorem ipsem sitamet,consecte

pscingelit,sed diam nonu

Figure 2. Internal storage interfaces and data structuresand their relationships with one another.

11/02/00 30720

"myfile""bob"


4/9

sulted as a file system executes user requests on datafiles.

In this example, four of the boxes at the bottom arethe four data blocks needed to store the contents ofmyfile. The two boxes at the top of the diagram

are blocks containing directories. A directory associ-ates human-readable file names with pointers to thecorresponding files descriptors. The two blocks in themiddle of the diagram contain the file systems tableof file descriptors. A file descriptor is a fixed-sizerecord containing in its hundreds of bytes a fewattributes of the file, a few (direct) pointers to the firstfew data blocks of the file, and a few pointers to indi-rect blocks.

On the right-hand side of Figure 2 is an illustrationof a SAN systems internal components. Such systems

contain an undifferentiated set of fixed-size blocksnamed according to their position in a list of all such

blocks. This set of fixed-size blocks is stored on non-volatile storage (typically, magnetic-disk media), suchthat a sequential walk through the list of blocksexploits a devices maximum data rate. Dotted linesrepresent the relationship between the file systemsdata and metadata structures and their persistent stor-age on the blocks of the SAN system.

While NAS and SAN interfaces are functionallysimilar to traditional file systems and storage deviceinterfaces, NAS and SAN offer much more manage-able storage to a data centers staff. SAN devices,whether internally or with SAN-specific software,typically represent multiple disks as if there were onlyone disk (a virtual disk volume), simplifying space

management, and transparently maintain redundantdata (or redundant array of inexpensive disks, RAID,encodings), thus increasing availability and reliability.NAS systems inherit these advantages because theyare usually built on SAN storage. Moreover, new NASsystems can be installed and configured without inter-rupting the execution of client machines. And NASusers on different machines can see and share the

same collection of files without special effort. Thesemanageability advantages of NAS and SAN systems

are compelling to those responsible for an organiza-tions information technology, given the generalscarcity of storage administration professionals.

Requirements for Emerging SystemsFuture network storage systems must provide fea-tures to meet all existing requirements, includingresource consolidation, rapid deployment, centralmanagement, convenient backup, high availability,and data sharing, as well as the following emergingrequirements.

Geographic separation of system components.Because online commerce is increasingly global andcompetitive, remote data must be continuously avail-

able, and all data must have remote copies updatedfrequently to protect against regional disasters. More-over, with the Internet infrastructures bandwidthgrowing at upward of 300% per year, employing theInternet in an organizations own internal network isincreasingly affordable and effective.

Increasing risk of unauthorized access to storage.Security is an increasingly critical storage property as

40 November 2000/Vol. 43, No. 11 COMMUNICATIONSOF THE ACM

Table 1. Scaling size stressesimplementation mechanisms.

What is being scaled What mechanisms are stressed?

number of client

and server nodes

distance

aggregate and

individual bandwidth

number and

size of files

directory size and

tree depth

resource discovery; network

bandwidth and congestion control;addressing

network congestion and flow control;

latency and round-trip dependencies;

resource discovery; security;

network routing

interconnect technology; protocol

processing in client and server OS

application addressing;

file metadata management

file metadata management; round-

trip time for repeated name lookup

Networked storage reduceswasted capacity, the time todeploy new storage, and backupinconveniences; it also simplifiesmanagement, increases dataavailability, and enables thesharing of data among clients.


5/9

online commerce becomes important and as elec-tronic crime (hacking) becomes increasingly sophisti-cated and common. Moreover, storage-systeminterconnects, including most SANs today, were orig-

inally designed as extensions of the internal buses oftheir hosts; their security provisions are limited ornonexistent. Extended buses may be easily securedfrom hostile traffic through physical means, but theyare also limited in geographic distribution and thenumber of attached devices they can support. Linkingextended buses to other networks, now possible withstorage interconnects like Fibre Channel, greatly

weakens these physical controls. Mechanisms forrestricting access to network-storage servers are evenmore important on the Internet than in standaloneFibre Channel networks.

Need for performance to scale with capacity. Per-

formance, measured either as accesses per second ormegabytes per second, needs to scale with storagecapacity to accommodate the increasing power andnumber of client machines, as well as the increasingsize of datasets (for applications manipulating suchdata as sensor traces, transaction records, still images,and video).

In general, the increasing scale and scope of the useof storage systems drives these emerging requirements.See Table 1 for the system parameters and implemen-tation mechanisms most likely to be stressed beyondtheir design goals by the increasing scale and scope ofhow these systems are used.

The storage systems designed to meet these

requirements are likely to be structured around theiranswers to a few critical architectural questions:

What is the storage abstraction for the networkinterface? How many network crossings per unit ofapplication work are required? What are the bottle-neck functions? Do control and data travel the samepath? How is clustering used? What parts of the sys-tem are trusted?

NAS places a network between client and file sys-tem, and SAN places a network between the file sys-tem and storage media. Other options for theabstraction of a networked storage interface might beto place a network between client and application, as

is done by such application servers as databaseengines, or between the file-system directory func-tions and the files themselves, as in the object-basedstorage devices, such as the Network-Attached SecureDisks (NASD) system, discussed later. Moreover, theattributes associated with stored blocks, objects, orfiles could be much richer; in addition to recordingdates, sizes, and permissions, attributes could be usedto set performance goals, trigger backups, or coordi-nate shared access with locks and leases.

The number of network crossings needed to com-plete a unit of application work is critical when com-ponents of the system may be separated by largegeographic distances. Block-level abstractions offered

by SAN systems send more storage requests across thenetwork to do the same work, because client-basedfile systems may have to sequentially fetch multiplemetadata blocks before they are able to fetch datablocks. In contrast, NAS systems contain the file sys-tem that interprets metadata, so they do not send asmuch metadata across the network.

Traditional NAS and SAN systems have manydisks per storage-controller processor to amortize thecost of the storage controller. This architecture ren-ders the controller a bottleneck, because todays disksmove data efficiently (compared to general-purposeprocessors), and file-system command processing uses

relatively large numbers of server processor cycles.One strategy for avoiding controller bottlenecks is toseparate control and datapaths. This classic direct-memory-access approach allows the datapath to bespecialized for speed. Another strategy for avoidingbottlenecks is to parallelize the bottleneck controllercomponent into a coordinated, load-balanced clusterof controllers.

Deciding what components and which networkmessages to trust is the core of any security architec-ture. Traditionally, SAN storage trusts everythingattached to and received over its network. NAS sys-tems have traditionally trusted less, mainly the oper-ating systems of their clients but not the users of these

clients. Todays security techniques, such as virtualprivate networks like IPSec, firewalls, and Fibre Chan-nel zoning, allow SAN systems to reduce their trustdomains to mainly their clients operating systems bylimiting traffic on the network to only participatingsystems. Cryptographically authenticated connectionsbetween end users and servers would further improvethe ability to discriminate among authorized and


Figure 3. Convergence of SCSI networking(SAN) and network file systems (NAS).

Workstations

1985 1990 1995 2000

Small

Computer

System

Interconnect

(SCSI)

Network

AttachedStorage

(NAS)

HighlyGeneral-

PurposeDesign

HighlyStorage-SpecializedDesign

Cus to m V LS I F as t R IS Cs $ 1, 00 0 P Cs , Gb s wi tc hi ng

Network

File Server

(Sun)Streamlined

Filer

(NetApp)

NAS

Clusters

Specialized

File Server

(Auspex)

SCSI 1SCSI 2

FCP

iSCSI


6/9

unauthorized requests on sharednetworks.

Converging of NASand SAN

Although the placement of filesystem functions is an importantdifference in the interfaceabstraction of NAS and SAN sys-tems, in other ways these tech-nologies are becoming moresimilar [10]. Discussed earlier washow they provide solutions to thesame set of customer problems,making it reasonable for an exec-utive responsible for informationtechnology to view them as inter-changeable alternatives. However,they are also converging in

another way: The degree towhich their implementations arespecialized for storage is increasing in NAS systemsand decreasing in SAN systems. Specialization inhardware, software, or networking often yields moreefficient use of resources (including memory band-width, die space, and manufacturing costs). How-ever, specialization is costly and restricts the marketsize from which its development costs can be recov-ered. Generalized components benefit from amortiz-ing development costs over much larger markets, sothe rate of improvement is often much faster. More-over, the use of generalized components creates

opportunities for leaps in technology capability,because it is sometimes possible to reuse completeand complex solutions developed for other problems.

Figure 3 reflects the specialization trends in NASand SAN systems, using the Small Computer SystemsInterface (SCSI)the dominant SAN commandprotocolto illustrate SAN specialization.

More specialized NAS systems. Network attachedstorage, originally known as network file service, wasdeveloped as a side effect of Ethernet LANs and engi-neering workstations. These file servers employedgeneral-purpose operating systems on general-pur-pose hardware, usually the same machines sold ascomputing servers, or even workstations. This solu-

tion is still popular today, as file servers built withMicrosoft and Linux operating systems employ stan-dard server hardware and software.

Not long after general-purpose network file serversbecame popular in workgroups in the 1980s, Auspexof Santa Clara, Calif., developed an aggressively spe-cialized NAS system using custom hardware, anunusual interconnect, asymmetric multiprocessing,

and a specialized operating and file system. Auspexfile servers provided higher performance and an earlyexample of the benefits of consolidating the storageresources of multiple workgroups.

Novell of Provo, Ut., and Network Appliance ofSunnyvale, Calif., chose a less-aggressive form of spe-cialization, using general-purpose high-performance

workstations and specialized (streamlined) operatingand file systems [5]. Because of the performanceimprovement in the killer micros of the early 1990s,this approach was cost-effective while still providing

high availability and simplified storage managementthrough specialized software.Today, there is a resurgence of storage hardware

and software specialization in a variety of advanceddevelopment laboratories. One theory for this resur-gence, beyond increasing market size, follows an anal-ogy with network infrastructure. Some people reasonthat just as general-purpose routers gave way to spe-cialized hardware and software switches now beingscaled to terabit speeds, storage infrastructure shouldalso be specialized around very high-bandwidth inter-nal-interconnection networks. The major alternativeto this super fileserver approach is the use of PCclusters with a low-latency cluster interconnect based

on network interface cards offloading protocol pro-cessing from each machines main processor. Suchnetwork-specialized cluster approaches also requireextensively specialized software; Network Applianceand Intel of Santa Clara, Calif., have proposed such anew file system architecture and protocol called theDirect Access File System [9].

Less-specialized SAN systems. For block servers

42 November 2000/Vol. 43, No. 11 COMMUNICATIONS OF THE ACM

Figure 4. Function and network links in the case studies.

1. Local file system.

LAN

application

2. Network file server

and Storage appliance3. Network file server

with RAID controller

4. Network file server

with DMA bypass.

5. NASD (OSD):Object-based storage device)

6. NASD cluster

7. Petal cluster

8. iSCSI devices

directories object store media

read, write

bulk data transfer

read, write

SAN

LAN SANSAN

LAN SAN

SAN

LAN

LAN LAN

LAN

LAN

LAN

SAN


7/9

with many disks, SCSI has been the leading storage-command protocol for almost 20 years. Over thistime, SCSI command-level capabilities have evolvedslowly, but the physical implementation has changeddramatically. As with most communication protocols,SCSI can be implemented as a layering of physicalsignal management, signaling and congestion control,and command interpretation. As technology advancesdrove changes in SCSIs lower levels, these levels havebeen replaced by faster, more flexible technologies.Fibre Channel is the dominant high-performance

SAN technology today, using the same physical wiresand transmission mechanisms as Gigabit Ethernet,although its signaling and congestion control is spe-cialized to SCSI.

This trend toward generalization in SAN systemswas also seen more than 15 years ago in Digital Equip-ment Corp.s VAXclusters, which first used a propri-etary physical interconnect (VAX-CI), then moved to

the more standard and more gen-eral-purpose Ethernet [6].

Four ArchitecturesIt may be too early to know how

the changing levels of specializa-tion in storage systems will affecthow computers are built and sold.

We can, however, identify therange of possible outcomes, par-ticularly four architectures underactive or recent research anddevelopment: storage appliances,iSCSI, NASD, and Petal. Storageappliances, commercially avail-able today, are nonclustered spe-cialized NAS systems. iSCSI isthe least-specialized SAN inter-face. Petal and NASD were exper-

imental research projects forclustering storage without serverbottlenecks.

Figure 4 and Table 2 comparethese architectures. The figurecontrasts the organization of func-tion into boxes and numbers ofnetwork transits when accessingdata. (In it, boxes are computers;horizontal lines are communica-tion paths; vertical lines are inter-nal and external interfaces; LAN isan Internet network, like Ethernet;

and SAN is a SCSI network likeFibre Channel.) The table showseach of the four case studies

answers to the critical architectural questions dis-cussed earlier.

Storage appliances. Storage appliances are NAS sys-tems intended to be especially simple to manage,ranging from Network Appliances terabyte servers toa disk drive with an Ethernet plug [5]. The Snap!server from Quantum of Milpitas, Calif., is an exam-ple of the lowest-price storage appliance availabletoday. Externally, it is a standard NAS server. Inter-nally, it includes a low-cost small-form-factor, single-board PC attached to one or two PC disk drives and

packaged in a box smaller than the typical PC, withonly a power cable and an Ethernet connector asexternal attachments.

Snap! servers consist of two printed circuit boards:one for the computer, one for the disk-drive con-troller. A more specialized storage appliance mightcombine these boards, integrate their processors, andcombine their code and datapaths. Cables, connec-


Figure 5. Network-attached secure disks (NASD).

File Manager

Protocol Stack

Object Capability

Client Network

NASDAccess Control,NamespaceConsistencyNetwork Net Controller

NASD

Net Controller

1 2

3,4

3 4

34

Table 2. Case studies.

StorageAppliance

Interface Abstraction

Geographic

distribution

Trusted components

Load-balance

commands

Consistent

concurrent access

Redundancy for

availability

Bottlenecks

Speed of

deployment limits

Control and data

Resources

consolidated

Management

executed by

files + directories

client OS

no

serialized

inside the box

controller

save/restore

files

same path

inside the box

controller

NASD

file objects

metadata server

yes (reads/writes)

distributed

locking

across the cluster

metadata changes

online

reallocation

separated paths

by metadata

server

metadata server

Petal

blocks

additional

round trips

client OS, network

yes

distributed

locking

across the cluster

configuration changes

online

reallocation

same path

server cluster

any server

blocks

additional

round trips

client OS, network

no

serialized

inside the box

controller

stop, copy,

reboot

same path

inside the box

controller

iSCSI


8/9

tors, boards, chips, and memory may be eliminatedand made less costly. However, because storagedevices are required to be one of the most reliableparts of any computer system, integrating NAS soft-ware into the disk-drive controller may initially

decrease reliability and inhibit product acceptance.Moreover, the hardware design of todays disk con-trollers assumes the microprocessor on the disk doesnot need to read much of the data as it goes by, butNAS software typically moves every byte through theprocessor multiple times. Accordingly, hardware inte-gration may be less than straightforward.

When the integration of NAS server and disk con-troller is achieved, disk-drive manufacturers will havea significant cost advantage over NAS system vendorsoffering two-board solutions. Thus, this highly inte-

grated specialization will be driven by user require-ments for file servers in the lowest-price packages.

Unfortunately, users want large-capacity configura-tions more often than they want small-capacity con-figurations. Hence, storage-appliance vendorsconcentrate on making each appliance as easy to con-figure as possible, then rack-mount many appliancestogether. Today, this combination of low cost andmanual management of aggregation is popular forproviding information content over the Web.

NASD. NASD was a research project at CarnegieMellon University (beginning in 1995) pursuing themotion that aggregation of storage devices is bestmanaged through a central policy server (possibly acluster of servers), while most commands and datatransfers move directly between device and client,

bypassing the server; Figure 5 outlines this asymmet-ric control and datapath [3]. Upon approval by thecentral server, clients can access (directly and in paral-lel) all devices containing data of interest.

There are many ways to ensure that the server insuch asymmetric systems controls client access to stor-age. The most common trusts client operating sys-tems to request and cache file system metadata [4].

Each client temporarily functions as a server for thefiles whose metadata it has in cache. Unfortunately,this role as temporary server increases the risk of secu-rity breaches, because devices continue trusting anyclient, and clients are notoriously easy to penetrate,

especially by partially authorized employees.A more secure asymmetric control-and-data archi-

tecture places the metadata at the device rather thanat the client. NASD stored file descriptors and indi-rect blocks in the device permanently and let clientsread and parse directories. NASD policy servers con-struct authorization tokens (capabilities) that clientsdigitally sign on every request and that NASD devicestest on every command. By storing file metadata inthe device, NASD offered a file interface abstraction.However, it did not offer directory operations,

because it assumed data is spread over multipleNASDs and that the file systems directories are glob-

ally defined. Instead, a NASD policy server storesdirectory information in NASD files clients can parsewithout the device recognizing the difference betweena regular file and a directory.

Petal. Petal was a research project at Compaqs Sys-tems Research Center [7] based on arrays of storage-appliance-like disk bricks but offering ablock-oriented interface rather than a file interface.Petal scaled by splitting the controller function over acluster of controllers, any one of which had access toconsistent global state. As a Petal systems capacitygrew, so did the number of Petal servers in the clusteralong with the performance they sustained. Logically,Petal could be viewed as a RAID system implemented

on a symmetric multiprocessor, though it used dis-tributed consensus algorithms instead of sharedmemory for global state.

For administrators, management in a Petal systemwas especially easy; any Petal server had access toglobal state, because servers could be added or lost atany time, and because device configurations could bechanged at any time.

44 November 2000/Vol. 43, No. 11 COMMUNICATIONSOF THE ACM

New NAS systems can be installed andconfigured without interrupting theexecution of client machines. AndNAS users on different machines cansee and share the same collection offiles without special effort.


9/9

Petal was built using Internet (LAN) protocols butlogically designed SAN interface. The availability ofiSCSI, discussed next, provided an ideal alternative toPetals custom LAN-based SAN protocol.

iSCSI. Layering a block-level SAN protocol over

Internet protocols, such as was demonstrated by theNetstation project at the University of Southern Cali-fornias Information Sciences Institute starting in1991, is a natural way for storage to exploit the influ-ence of the Internet [12]. Beginning earlier this year,an Internet Engineering Task Force (IETF) workinggroup has sought to standardize a block-level com-mand-and-data-movement system over the principleInternet protocol, conveniently named the InternetProtocol (IP). Known informally as iSCSI, for Inter-net SCSI, the IPS (IP Storage) working group is alsochartered to work on security, naming, discovery, con-figuration, and quality of service for IP storage.

A number of startup companies have declared their

interest in implementing iSCSI in storage routersthat may turn out to be similar to RAID systems. LikeFibre Channels command protocol FCP, iSCSI is aSAN interface. Although it does not provide file sys-tem functionality, it should interoperate with systemsdesigned for other SANs, once the storage industryfigures out how to configure and tune different SANinterconnects interoperably.

iSCSI is a step toward generalization in storage-device networking. Taking advantage of the existingbody of work on Internet communication protocolsand media, it intends to give storage networking thescaling properties of IP networks. Not yet clear to IPS

working group observers, however, is how much ofthe Internet protocol suite will be used without mod-ification. For example, most Internet applicationsemploy a transport mechanism called TCP to reliablydeliver data over IP. The IETF working group usesTCP as it is defined today; other groups are proposingsmall changes to TCP to make high-speed data trans-fer more efficient; still others are proposing an entirelydifferent congestion-control algorithm they think isbetter for storage traffic.

ConclusionStorage systems are becoming the dominant invest-ment in corporate data centers and a crucial asset in

e-commerce, making the rate of growth of storage astrategic business problem and a major businessopportunity for storage vendors. In order to satisfyuser needs, storage systems should consolidateresources, deploy quickly, be centrally managed, behighly available, and allow data sharing. It shouldalso be possible to distribute them over global dis-tances, make them secure against external and inter-

nal abuse, and scale their performance with capacity.Putting storage in specialized systems and accessingit from clients across a network provides significantadvantages for users. Moreover, the most apparentdifference between the NAS and SAN versions of

network storageuse of Ethernet in NAS and FibreChannel in SANis not a core difference and maysoon not even be a recognizable difference. Instead,

we may have NAS servers that look like disks, disksthat connect to and operate on Ethernet, arrays ofdisk bricks that, as far as the user is concerned, func-tion as one big disk, and arrays of smart disks thatverify every command against the rights of individ-ual users.

References1. Benner, A. Fibre Channel: Gigabit Communications and I/O for Com-

puter Networks. McGraw Hill, New York, 1996.2. Callaghan, B. NFS Illustrated.Addison Wesley Publishing Co., Read-

ing, Mass., 2000.

3. Gibson, G., et al. A cost-effective, high-bandwidth storage architecture.In Proceedings of the ACM 8th International Conference on ArchitecturalSupport for Programming Languages and Operating Systems (ASPLOS)(San Jose, Calif., Oct). ACM Press, New York, 1998, 92103; see alsowww.pdl.cs.cmu.edu.

4. Hartman, J. and Ousterhout, J. The Zebra striped network file system.In Proceedings of ACM Symposium on Operating Systems Principles(SOSP) (Ashville, N.C., Dec.). ACM Press, New York, 1993, 2943.

5. Hitz, D., Lau, J., and Malcolm, M. File systems design for an NFS fileserver appliance. In USENIX Winter 1994 Technical Conference Pro-ceedings(San Francisco, Jan. 1994).

6. Kronenberg, N., et al. VAXclusters: A closely coupled distributed sys-tem.ACM Transact. Comput. Syst. (TOCS) 4, 2 (May 1986), 130146.

7. Lee, E. and Thekkath, C. Petal: Distributed virtual disks. In Proceed-ings of the ACM 7th International Conference on Architectural Support forProgramming Languages and Operating Systems (ASPLOS) (Cambridge,Mass., Oct). ACM Press, New York, 1996, 8492.

8. McKusick, M., et al. A fast file system for Unix. ACM Transact. Com-put. Syst. (TOCS) 2, 3 (Aug. 1984).

9. Network Appliance, Inc. DAFS: Direct Access File System Protocol, Ver-sion 0.53 (July 31, 2000); see www.dafscollaborative.org.

10. Sachs, M., Leff, A., and Sevigny, D. LAN and I/O convergence: A sur-vey of the issues. IEEE Comput. 27, 12 (Dec. 1994), 2432

11. Satran, J., et al. iSCSI (Internet SCSI), IETF draft-satran-isci-01.txt(Jul.10, 2000); see www.ece.cmu.edu/~ips.

12. Van Meter, R., Finn, G., and Hotz, S. VISA: Netstations virtual Inter-net SCSI adapter. In Proceedings of the ACM 8th International Confer-ence on Architectural Support for Programming Languages and OperatingSystems (ASPLOS) (San Jose, Calif., Oct.). ACM Press, New York,1998, 7180; see also www.isi.edu/netstation/.

Garth A. Gibson ([email protected], [email protected]) isthe chief technology officer of Panasas, Inc., Pittsburgh, PA, and anassociate professor (on leave) in the School of Computer Science atCarnegie Mellon University, Pittsburgh, PA.Rodney Van Meter ([email protected]) is a seniorsoftware engineer in Nokia Internet Communications in Santa Cruz,CA.

Permission to make digital or hard copies of all or part of this work for personal or class-

room use is granted without fee provided that copies are not made or distributed for

profit or commercial advantage and that copies bear this notice and the full citation onthe first page. To copy otherwise, to republish, to post on servers or to redistribute to

lists, requires prior specific permission and/or a fee.

2000 ACM 0002-0782/00/1100 $5.00

c


network attached storage architecture

Documents