implementing san in linux...
TRANSCRIPT
HousekeepingHousekeeping
• EvaluationsEvaluations– OSS2– Be HonestBe Honest– Comments
• Questions? Raise your hand wait for the • Questions? Raise your hand, wait for the Microphone.Ch k h l f h• Check the on-line version of this presentation for updates.
STORAGE!STORAGE!
• 1956 - IBM Model 350 the first-ever hard disk 1956 - IBM Model 350, the first-ever hard disk drive.
• Ab t 4 4 M B t !• About 4.4 MegaBytes!
The Good News?The Good News?
• Storage is getting cheaper!Storage is getting cheaper!
Year Cost per MegaByte1956 $10,000.001980 $192.311984 $87.861989 $12.001994 $0 951994 $0.951999 $0.022004 $0 001152004 $0.00115
Why are you here?Why are you here?• Learn to build your own SAN or NAS (DIY)• SAVE $$$• Know your systems better – hardware, softwareKnow your systems better hardware, software• Supplement your PRIVATE CLOUD
S 2 B kBl b il Sept 2009 BackBlaze built a 67 TB storage unit for $7,867http://bit ly/s0Emghttp://bit.ly/s0Emg
SAN & NAS in LinuxSAN & NAS in Linux
• SAN - AoE: Target & Initiator configAN f• SAN - iSCSI: Target & Initiator config
• NAS – Samba/CIFS• Openfiler• DRBD – Block level replicationDRBD Block level replication
What is SAN?What is SAN?
• Stands for Storage Area NetworkStands for Storage Area Network• Wikipedia: an architecture to attach remote
computer storage devices to servers in such a computer storage devices to servers in such a way that the devices appear as locally attached to the operating system
• Block-level operations• Protocols: Fibre Channel, iSCSI, ATA over Protocols: Fibre Channel, iSCSI, ATA over
Ethernet (AoE)• SAN is not NASSAN is not NAS
What is NAS?What is NAS?
• Stands for Network Attached StorageStands for Network Attached Storage.• Wikipedia: A NAS unit is a computer connected
to a network that onl pro ides file based to a network that only provides file-based data storage services to other devices on the networknetwork.
• Protocols used: NFS, SMB/CIFS, AFP.
What is Block Level?What is Block Level?
• a block is a sequence of bytes or bits having a block is a sequence of bytes or bits, having a nominal length (a block size).
• Wikipedia• Wikipedia:Blocking is used to facilitate the handling of the data stream by the computer program data-stream by the computer program receiving the data. Blocked data are normally read a whole block at a timeread a whole block at a time.
SAN: AoE on LinuxSAN: AoE on Linux
• network protocol developed by the Brantley network protocol developed by the Brantley Coile Company
• light weight, does not rely on network layers light weight, does not rely on network layers (TCP or IP)
• designed for simple, high-performance access designed for simple, high performance access of SATA storage devices over Ethernet networks.
• like FCoE not like iSCSI• AoE exists on Linux as a kernel moduleAoE exists on Linux as a kernel module
AoE: Why use it?AoE: Why use it?
• Ethernet: is getting fasterEthernet: is getting faster• and cheaper!• and Ethernet can be multiported (switches).• Enterprise ATA (or SATA) disks are also getting
cheaper. • ATA over Ethernet (AoE) — a SAN at a ( )
fraction of the cost of Fibre Channel.
AoE Target setup on Linux SANAoE Target setup on Linux SAN
• Hardware RAID – make sure you Hardware RAID make sure you have Linux drivers
• Or you can use Software RAIDOr you can use Software RAID• Check AoE is enabled in kernel –
modprobe aoemodprobe aoe• AoE Target setup:
a. Install aoetools (sourceforge)a. Install aoetools (sourceforge)b. export block devices
vblade 0 1 eth0 /dev/sdb
AoE Target setup on Linux SANAoE Target setup on Linux SAN
vblade 0 1 eth0 /dev/sdbvblade 0 1 eth0 /dev/sdbWhat does this mean?
vblade = commandvblade = command1st number 0 = shelf2nd b 1 l t2nd number1 = sloteth0 = ethernet device at address 0/dev/sdb = name of exported block device
AoE Target setup on Linux SANAoE Target setup on Linux SAN
IMPORTANT!IMPORTANT!
fIf you are planning to have many AoE Target, make sure there are
f #uniqueness to your shelf and slot #s!
vblade 0 1 eth0 /dev/sdb
AoE Initiator (Linux client)AoE Initiator (Linux client)
On Linux server: On Linux server:
1. Make sure AoE exists in kernel d l # d bas a module # modprobe aoe
2. # aoe-stat3. # ls –l /dev/etherd4. # mkfs.ext3 /dev/etherd/e0.0/ / /5. # mount /dev/etherd/e0.0 /mnt
AoE Initiator (Windows client)AoE Initiator (Windows client)
On Windows server:On Windows server:
1. WinAoE: http://bit.ly/8UmigK2. RocketDivision StarPort iSCSI
and AoE Initiator: // /9http://bit.ly/9xUDEt
SAN: iSCSI on LinuxSAN: iSCSI on Linux
• iSCSI = Internet Small Computer Systems iSCSI Internet Small Computer Systems Interface
• IP based storage networking standard for • IP-based storage networking standard for linking data storage facilities
C / 860 3260• Uses TCP/IP Ports 860 and 3260
iSCSI Target setup on Linux SANiSCSI Target setup on Linux SAN
• Make sure hardware RAID is supported ppby Linux or use software RAID
• Install and configure iSCSI Enterprise T h //bi l /54GETarget http://bit.ly/54GEne
• Or in Debian# apt-get install iscsitarget# apt get install iscsitarget
• Edit config files:# nano /etc/default/iscsitarget
/ /# nano /etc/ietd.conf# nano /etc/initiators.allow
• Start the service• Start the service…
iSCSI Initiator (Linux client)iSCSI Initiator (Linux client)
On Linux:On Linux:
1. Install open-iscsihttp //www open iscsi org/http://www.open-iscsi.org/or apt-get install open-iscsi in DebianDebian
2. Edit config file# / / / d f# nano /etc/iscsi/iscsid.conf
iSCSI Initiator (Windows client)iSCSI Initiator (Windows client)
On Windows:On Windows:
1. Windows 7 & Windows 2008 is b ilt inbuilt-in
2. Windows XP & Windows 2003:MS SCS fDownload MS iSCSI software
initiator v2.08h //b l /2 AChttp://bit.ly/2F7AC1
3. Or use Starport’s Initiator
SAN Best PracticesSAN Best Practices
• Design and Plan your networkDesign and Plan your network• Network isolation (security)
Logical network isolation VLAN- Logical network isolation – VLAN- Physical network isolation
( )• Replication (Block level replication)
QUESTIONSQUESTIONS
• When setting up your Linux SANWhen setting up your Linux SAN
NAS on LinuxNAS on Linux
• Centralized “file-based” storageCentralized file based storage• Wikipedia- “ NAS unit is a computer connected to
a network that only provides file-based data a network that only provides file-based data storage services to other devices on the network.”
• “NAS uses file based protocols such as NFS • NAS uses file-based protocols such as NFS (popular on UNIX systems), SMB/CIFS (used with MS Windows systems), or AFP (used with Apple MS Windows systems), or AFP (used with Apple Macintosh computers). NAS units rarely limit clients to a single protocol.”g p
Simply said:Simply said:
• NAS is file sharingNAS is file sharing• NAS can be a file share on a server
NA• NAS can also be a dedicated hardware device• NAS can also be an embedded device
Linux Distro: Openfiler NSALinux Distro: Openfiler NSA
• Base on rpath - 32/64 bit http://bit ly/ySPPZBase on rpath - 32/64 bit. http://bit.ly/ySPPZ• Web-based GUI
M N N• Multiple NICs (NIC bonding)• Protocol rich:
1. Block level: iSCSI, Fibre Channel2. File level: NFS, SMB/CIFS, Http/DAV,
FTP, rsync• Does not support AoE out of the box BUT…pp
Openfiler NSA (SAN)Openfiler NSA (SAN)
1 iSCSI is built-in-GUI-web-configure-goodness1. iSCSI is built-in-GUI-web-configure-goodness.2. AOE is not built-in.
aoetools needs compiling and it worksaoetools needs compiling and it works.a. before compiling, install gcc
# conary update gcc# conary update gccb. download aoetools from Sourceforge
http //bit ly/cj2lawhttp://bit.ly/cj2law# make && make install
Openfiler NSA (NAS)Openfiler NSA (NAS)
• Industry-standard protocol suite – SMB NFS Industry-standard protocol suite SMB, NFS, FTP, WebDAV
• Share management• Share management• Quota / resource allocation – group, user,
guest• Accounts management – PAM, NIS, LDAP, Active
Directory
Other DistrosOther Distros
• FreeNAS (FreeBSD) – SAN and NASFreeNAS (FreeBSD) SAN and NASSMB/CIFS,FTP, NFS, TFTP, AFP, RSYNC, Unison, iSCSI (initiator and target)iSCSI (initiator and target)
• Other Linux distros –a Debian like distros Ub nt etc SAN and a. Debian-like distros, Ubuntu, etc. – SAN and
NAS = command lineb Other lin distrosb. Other linux distros
Why Replicate Data?Why Replicate Data?
• Fail safe (backup)Fail safe (backup)• HA (high availability)
W li i f• Wikipedia: “Replication is the process of sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to
l b l f l limprove reliability, fault-tolerance, or accessibility.”
Software ReplicationSoftware Replication
• SAN Block Level Replication – DRBD (Distributed SAN Block Level Replication DRBD (Distributed Replicated Block Device)
62 AttorneysColumbus
62 Attorneys
4/8 TB4/8 TB
Offices
ClevelandMarion2 Attorneys4/8 TB
Marion1 Attorney
Replicated Block DeviceReplicated Block Device
• RAID 1RAID 1• Between servers, instead of hard drives.
I Use UbuntuI Use Ubuntu
• Create identically sized unmounted partitionsCreate identically, sized unmounted partitions.– One partition for DRBD metadata.
One partition for user data– One partition for user data.
• Give servers static IPs.• Make sure time on servers is in sync.
Implementing DRBDImplementing DRBD
• All of this is command line…All of this is command line…• In.stall DRBD from repository.• Configure drbd conf fileConfigure drbd.conf file.
– ComplementaryPrimary or Secondary– Primary or Secondary
• Load DRBD: modprobe DRBD• I t ll d fi H tb t f it• Install and configure Heartbeat from repository.• Set virtual IP address.
S DRBD• Start DRBD.
What does it look like?What does it look like?
Filesystem Size Used Avail Use% Mounted ony/dev/sda5 4.6G 430M 4.0G 10% /tmpfs 126M 0 126M 0% /dev/shm/dev/sda1 89M 11M 74M 13% /boot/dev/sda1 89M 11M 74M 13% /boot/dev/drbd0 24G 33M 23G 1% /data