HandyFS: A Simple Ad-hoc Distributed File System only
using FUSE and SSHFSNan DUN Kenjiro Taura Akinori Yonezawa
The University of Tokyo
Sep 12, 2007 1http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx
Background Related Work
◦SSHFS◦FUSE
HandyFS◦Designs and Implementation◦Evaluation and Problems
Future Work
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 2
Outline
User explicit sharing operation◦rcp, scp[OpenSSH]
◦FTP[J. Postel, et al. 1985.], GridFTP[Globus]
Conventional Distributed File System◦Inner-Cluster Sharing NFSv3, NFSv4
◦Inter-Cluster Sharing AFS[OpenAFS], Gfarm[Tatebe et al. ‘04]
Background: File Sharing on Grid
Sep 12, 2007 3http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx
Complexity of conventional DFSs◦ Client-Server Configurations◦ Administrative Privilege◦ Typical Firewall Setting of the Grid
Simplicity of SSHFS [http://fuse.sourceforge.net/sshfs.html]
◦ Only need available SSH connection◦ Client-side mount only◦ User-level Operations without privileges
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 4
SSHFS: A simple alternative
Create local mount point◦user@hongo000:~$ mkdir mnt_chiba
Mount remote directory to mount point◦user@hongo000:~$ sshfs chiba000.intrigger.nii.ac.jp:/sharedir mnt_chiba
Access remote files via local mount point◦user@hongo000:~$ [ls, rm, cp, mkdir] mnt_chiba
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 5
SSHFS: Usage
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 6
SSHFS: Framework
fuse
hostA_mountpoint hostB_mountpoint
SSH client
hostA
SSHFS
User Application
General File System Call
SSH Commands
Translationfuse
SSH client
hostB
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 7
SSHFS: Evaluation
chiba suzuk okubo imade0
5
10
15
20
25
30chiba; scp;
24.79
suzuk; scp; 27.33 okubo; scp;
24.92imade; scp;
20.63
chiba; sftp; 21.67
suzuk; sftp; 26.49
okubo; sftp; 17.71
imade; sftp; 12.85
chiba; sshfs (fuse2.6.5);
15.52
suzuk; sshfs (fuse2.6.5);
17.53 okubo; sshfs (fuse2.6.5);
12.86 imade; sshfs (fuse2.6.5);
9.45
scp sftpsshfs (fuse2.6.5) sshfs (fuse2.7.0)
Data Transfer Rate (Mbps)
SSHFS-FUSE overhead
Disadvantages of Merely Using SSHFS◦ Non-scalable
Users have to create many mount points for hosts◦ Non-unified namespace
Files under different mount points (hosts) should be manipulated separately
Motivation of HandyFS◦ Inherit simplicity from SSHFS◦ Create a scalable mount for many hosts◦ Ad-hoc create a unified namespace for files on
many hosts
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 8
HandyFS: Motivation
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 9
HandyFS: Usage Create ONE local mount point
◦user@hongo000:~$ mkdir mnt_all Mount MULTIPLE remote directories to
mount point◦user@hongo000:~$ handyfs chiba000.intrigger.nii.ac.jp:/sharedir1
chiba001.intrigger.nii.ac.jp:/sharedir2 ...... mnt_all
Access remote files via local mount point◦user@hongo000:~$ [ls, rm, cp] mnt_all
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 10
HandyFS: Framework
FUSE Kernel Module
User Application HandyFS SSHFS
Host1 Host2
User
Kernel
SSHFS
The trick is using FUSE twice1. SSHFS mount for remote host2. Local mount for directory merge
/handyfs_mnt /sshfs_mnt1
/sshfs_mnt1
Source directory Tree at HostB
Source directory Tree at HostA
Directories Merge◦ Source directories and Target (virtual) directory
Source directory: SSHFS mount points for each host Target directory: Directory in HandyFS namespace
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 11
HandyFS: File Operations
/mnt_hostA
/A /B
/B/C
/mnt_hostB
/B /C
/B/D
/
/A /B
/B/C
Virtual directory Tree
/C
/B/D
Directories Merge◦ Source directories and Target (virtual) directory◦ Directory Table
To remember virtual directory is merged from which source directory
◦ On-Demand Merge Operation Merge operation occurs only when request reach
non-merged path
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 12
HandyFS: File Operations
/cmndir /srcdirA/cmndir
Virtual Directory Source Directories
/srcdirB/cmndir
Implemented by using BerkeleyDB to achieve good performance
File Lookup
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 13
HandyFS: File Operations
Lookup a file with full pathname
Retrieve source directory entries from
directory table
Lookup file in each retrieved source directory entries
If target pathname has not been
merged, then merge and store into directory table Return result to user
On-demand Merge
Directory Split◦ Remove one or several source directories from
virtual directory, we have two alternatives Remove directory table entries
Heavy Operation: Search entire directory table Re-insert overhead if source become available again
Blind user view of source directories by filtering Use a in-memory filter to mark current available sources File lookup results are filtered before returning to user Efficient and re-usable
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 14
HandyFS: File Operations
Directory Split
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 15
HandyFS: File Operations
Lookup a file with full pathname
Retrieve source directory entries
from directory table
Lookup file in each retrieved source directory entries
If target pathname has not been
merged, then merge and store into directory table
Return result to user
Filtering out results that are from
removed source directories
File Creation◦ File is created at random source directory◦ For directory file, an entry is created in directory
table
Directory Read◦ Similar as file lookup
Lookup target directory Read all entries from source directories that
composed of target directory
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 16
HandyFS: File Operations
Some Practical Problems◦ Same file from multiple sources
Not compatible with UNIX file semantics Define priority of source directory
Only files from source directory with high priority are visible to user
◦ Stale of Directory Table Remote directory tree may be modified by its local
operations Periodically update
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 17
HandyFS: File Operations
Experimental Environments◦ InTrigger Platform:
https://www.logos.ic.i.u-tokyo.ac.jp/intrigger/
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 18
HandyFS: Evaluation
Hardware/Software Specification
CPU Core2 Duo 2.13Hz
Memory 4GB
Network Gigabit Ethernet
OS Linux 2.6.18
FUSE Kernel Interface 7.8
FUSE Library 2.7.0
SSHFS 1.8
Experimental Results◦ Data Transfer Rate
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 19
HandyFS: Evaluation
chiba suzuk okubo imade0
10
20
30
40
50
60
70
223 222
ftp sftp sshfs HandyFS
SSHFS-FUSE overhead
TWICE-FUSE-MOUNT overhead
Experimental Results◦ Source Compilation
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 20
HandyFS: Evaluation
OpenSSH4.6 Source Compilation Benchmark0
50
100
150
200
250
300
350
Local FUSE-Local NFSSSHFS HandyFS
Wide-Area LatencySSHFS
Overhead
TWICE-FUSE-MOUNT overhead
Performance◦ HandyFS itself
To remove MOUNT-TWICE overhead◦ SSHFS performance
Need be tuned at source level
Scalability◦ “One-to-All” mount “All-to-All” mount
Sep 12, 2007 http://www.yl.is.s.u-tokyo.ac.jp/~dunnan/talk/070912.JSSST07.pptx 21
Future Work