27th november 2000hepsysman 20001 a report on the 2nd annual linux storage management workshop pete...
Post on 19-Dec-2015
214 views
TRANSCRIPT
27th November 2000 HEPSYSMAN 2000 1
A report on the2nd Annual Linux Storage
Management Workshop
Pete Grönbech
Systems Manager
27th November 2000 HEPSYSMAN 2000 2
Workshop Topics
Local File Systems Volume and Device Management Kernel Development Distributed & Cluster File Systems Clusters and High Availability Backup NFS and Device Management Storage Management Tutorials
27th November 2000 HEPSYSMAN 2000 3
Local File Systems
Stephen Tweedie, Red Hat: ext2fs & ext3fs Hans Reiser, Namesys: ReiserFS Steve Best, IBM: IBM’s journaled File System Steve Lord, SGI: SGI’s XFS Steve Pate, Veritas: VxFS & VxVM 2.4 Kernel Ted Ts’o Comments
Volume and Device Management
27th November 2000 HEPSYSMAN 2000 4
Stephen Tweedie
2.4 Kernel better SMP scalability 2.2 was better than 2.0 but not scalable above 4 CPU’s 2.4 has large memory support; increased from 4GB to 64GB
using PAE36 Need Distributed Lock Manager Journaling filesystems
ext3 and reiserfs being used in production Raid support
Software raid 1/5 are integrated Mylex, DPT, 3ware hardware raid controllers
Clustering Already have failover and load balancing
27th November 2000 HEPSYSMAN 2000 5
ReierFS
Perhaps more stable than ext3 at present Shipped on latest SuSE CD?
27th November 2000 HEPSYSMAN 2000 6
IBM JFS
Scaleable 64-bit filel system File size max 512TB w/ 512 block size File size max 4 PB w/4k block size (Limited by Linux I/O structures not being 64-bit)
Journaling of meta-data only Restarts after crash in seconds B+tree use is extensive throughout JFS Store names using Unicode
JFS shipped 2/2/2000 Alpha software, Beta end of the year. http://oss.software.ibm.com/developerworks/opensource/jfs/
index.html Lots of bureaucracy in IBM before a single line of code can be
released as open source. Lawyers have to check it against all their patents.
27th November 2000 HEPSYSMAN 2000 7
SGI’s XFS
SGI has ported its XFS filesystem and associated utilities to Linux. XFS is a full 64-bit file system that can scale to handle extremely large files (2^63-1 byte) and file systems.
Journaling Delayed write allocation, for better disk layout Direct I/O DMAPI support Beta code on 2.4.0-test5 2TB block device limit
27th November 2000 HEPSYSMAN 2000 8
Veritas
Industrial Strength Commercial product VxFS and VxVM
$1Billion Revenue in 2000 (Not open source!!) Ported to many unix variants
Solaris, HP-UX, NT, W2K, AIX and Linux
Port to Linux 2.4 kernel at Beta stage. Has all the features:
Journaling, online grow-shrink,defrag, snapshots, clustered file-system.
27th November 2000 HEPSYSMAN 2000 9
Linux Present 2.4 almost ready to be released (really!) 2.4 is much more scaleable
64 Gig memory on IA32 64 bit file access on 32-bit platforms (LFS API) 32-bit uid, gid Much better SMP scaleability
– Fine-grained locking (networking, VFS, etc.)
Better BUS support (PCMCIA, USB, firewire) NFS v3, NFS improvements RAW I/O
27th November 2000 HEPSYSMAN 2000 10
Volume and Device Management
Heinz Mauelshagen, Sistina Software: LVM
Richard Gooch, University of Calgary: Linux devfs A virtual File system similar to /proc with devfs /dev reflects the hardware you have register devices by name rather than device numbers Can support hot plugin of USB, PCMCIA and Firewire.
Ben Rafanello, IBM: IBM’s LVM IBM has volume group based LVMs and a partition based LVM.
Linux has LVM…. Enterprise Volume Management System to emulate multiple LVMs
within a single LVM, (Uses plug in modules)
27th November 2000 HEPSYSMAN 2000 11
Kernel Development
Rik van Riel, Connectiva: VM system Eric Youngdale, MKS, Inc: Linux SCSI mid-layer Justin Gibbs, Adaptec: Low-level SCSI drivers
27th November 2000 HEPSYSMAN 2000 12
Distributed & Cluster File Systems
Peter Braam, Carnegie Mellon University: Intermezzo A replicating high availability file system and file synchronization tool
Ken Preslan, Sistina Software: GFS Chris Feist, StorageTek: Secure File System Jeremy Allison, VA Linux: Samba Rob Ross, Argonne: PVFS
27th November 2000 HEPSYSMAN 2000 13
GFS
New networking technologies allow multiple machines to share storage devices. File systems that allow these machines to simultaneously mount and access files on these shared devices are called shared-disk file systems. This is in contrast to traditional distributed file systems where the server controls the devices.
GFS is a shared device, cluster file system for linux. GFS supports journaling and rapid recovery from client failures. Nodes within a GFS cluster share the same storage by means of Fibre Channel or shared SCSI devices.
The file system appears to be local on each node and GFS synchronises file access across the cluster.
27th November 2000 HEPSYSMAN 2000 14
Storage Cluster
Storage Area Network
RAID RAID
Node Node NodeGFS GFS
Switch TCP/IP
27th November 2000 HEPSYSMAN 2000 15
Parallel Virtual File System
Use of multiple distributed I/O resources by a parallel application Goal is to increase aggregate I/O performance Accomplished by reducing bottlenecks in I/O path
no single I/O device no single I/O bus no single network path
Target is medium to large clusters (64 or more nodes) Applications using MPI (ROMIO provides the interface, MPI-IO) Linux 2.2 kernel TCP data transfer only Use UNIX interface to store data on local file system (eg ext2fs,
reiserfs)
27th November 2000 HEPSYSMAN 2000 17
Clusters and High Availability
Alan Robertson, SuSE: Heartbeat Lars Marowsky-Bree, SuSE: Failsafe Brian Stevens, Mission Critical Linux: Kimberlite Philip Reisner, Qubit: drdb
27th November 2000 HEPSYSMAN 2000 18
Heartbeat
Memership Services Notice when machines join/leave the cluster Notice when links go down/come back
Communication Services Cluster Manager
Currently limited to 2 nodes Resource Monitoring
Not yet Storage Resource I/O Fencing
STONITH Shoot the other node in the head Reset or Power cycle other node
Load Balancing (Optional)
27th November 2000 HEPSYSMAN 2000 19
distributed replicated block device
disk mirroring via the network used for implementing high-availability servers under linux.
webservers, fileservers data mirroring, cheaper than with shared disks higher overheads at writes monitoring of nodes with “heartbeat” currently 2.2 kernel
27th November 2000 HEPSYSMAN 2000 20
Backup
John Jackson, Purdue: Amanda Gawain Lavers, Big Storage: Linux DMAPI
27th November 2000 HEPSYSMAN 2000 21
NFS and Device Management
Andy Adamson, University of Michigan: Linux NFS V4 Sept 1st 2.2.14 Kernel Network Appliance sponsoring NFS V3/4 performance project
Dave McAllister, 3ware: Storage protocols over IP
Holger Smolinski, IBM Germany: Dynamic registration of SCSI devices
27th November 2000 HEPSYSMAN 2000 22
Storage Protocols over IP
With the advent of gigabit ethernet, and the planned drive to 10 gigE, the capability of supporting storage, and SAN over IP becomes attractive.
Being defined by 3ware, working with IETF iSCSI 3ware….defining a protocol that allows multiple ATA(IDE) drives
to be presented as one or more SCSI drives Allows storage on same network as Lan traffic. Over comes ordering restrictions that have hampered SCSI and
FC Capabilities of FC SAN, but at much lower cost Their hardware had embedded linux, but they changed to
FreeBSD and got a 3 fold (22-70MB/s) performace improvement. This is because the 2.4 kernel does not have zero copy yet.
27th November 2000 HEPSYSMAN 2000 23
Storage Management
Ric Wheeler, EMC: Smart Storage and Linux: An EMC perspective
Daniel Pillips, innominate AG: Tux 2 Filesystem Like a journaling FS but no journal, uses a db type approach.
Sang Oh, SANux: SANux File System SAN based cluster file system. Has a DLM.
27th November 2000 HEPSYSMAN 2000 24
My Conclusions
(Too) Many different Journaling File systems to choose from ext3fs mainstream (from Red Hat) ReiserFS may be more stable now (from Suse)
Low cost data mirroring with drbd For SAN’s look at GFS For SMP 2.4 Kernel is key
For more details and copies of the slides see
http://www2.physics.ox.ac.uk/gronbech