DIFFSYNCA file synchronization program

CLASS PROJECT – EEL6883

BY

ADAM RADWAN

GARGI CHIPALKATTI

FILE SYNCHRONIZATION

Process of making sure that two or more locations contain the same up-to-date files

One-way (Mirroring) or two-way file synchronization Useful for mobile workers, multiple computer users;

used for backup purposes Version control tools and File synchronization tools Various existing solutions, none of which were found

by OpenPublic Networks, LLC to be robust enough for production use in a distributed environment

GOALS

Develop a file synchronization program which is capable of working over a diverse range of geographical placement

Must be capable of running securely over common public network backbone, such as the Internet

Capable of two-way synchronization of files for changes to served web pages

Ability for a non-functional server to gracefully degrade while other servers detect this and recover appropriately to keep services running

RSYNC

Open Source software for Unix systems Mirroring takes place with only one transmission in

each direction Used to synchronize local directories or directories

via a remote shell such as RSH or SSH Two checksums: a weak “rolling“ 32-bit checksum

and a strong 128-bit MD4 checksum

UNISON

Runs on Unix-like operating systems (Linux, Mac OS X, and Solaris), and on Windows

Can synchronize files across platforms Detects 'conflicts' where a file has been modified on

both sources, and displays these to the user Uses the rsync algorithm Designed to be robust in the event of a program or

system crash or a communication failure

DIFFSYNC

Uses open source software – rsync, inotify Uses PHP scripting language Built on top of a MySQL 5.1 triple-master replicated

database system running in chrooted jail for security Custom SSH-based VPN for three independent point-

to-point TUN links between the servers Apache-based web server for verification running in

chrooted jail for security phpMyAdmin used for database management

DIFFSYNC FOR THREE SERVERS

Akimbo

Location: Orlando, FLIP: 72.189.195.130CPU: Pentium III 800 MhzStorage: 20GB, 128 MBOS: Gentoo Linux Philomena

Location: Ft. Lauderdale, FLIP: 65.111.166.49CPU: AMD 2000+Storage: 40GB, 256 MBOS: Gentoo Linux

Podice

Location: Fayetteville, NCIP: 216.117.160.235CPU: Pentium 4 2.8 GhzStorage: 80GB, 512 MBOS: Gentoo Linux

CREATION OF DIRECTORY & FILE~/TestArea $ mkdir 1 Array( [0] => Array ( [dir] => ./TestArea/ [event] => CREATE,ISDIR [file] => 1 ) [1] => Array ( [dir] => ./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [2] => Array ( [dir] => ./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 ))

~/TestArea $ nano ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => CREATE [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => X ) [3] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X ))

MOVE DIRECTORY & FILE

~/TestArea $ mv ./1/X ./1/Y Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => MOVED_FROM [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => MOVED_TO [file] => Y ) )

~/TestArea $ mv ./1 ./2 Array( [0] => Array ( [dir] => ./TestArea/ [event] => MOVED_FROM [file] => 1 ) [1] => Array ( [dir] => ./TestArea/ [event] => MOVED_TO [file] => 2 ) )

ATTRIB DIRECTORY & FILE

~/TestArea $ chmod 777 ./1 Array( [0] => Array ( [dir] => ./TestArea/ [event] => ATTRIB [file] => 1 ))

~/TestArea $ chmod 777 ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => ATTRIB [file] => X ))

DELETE DIRECTORY [4] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => ) [5] => Array ( [dir] => ./TestArea/ [event] => DELETE,ISDIR [file] => 1 ) [6] => Array ( [dir] => ./TestArea/1/ [event] => DELETE_SELF [file] => ) [7] => Array ( [dir] => ./TestArea/1/ [event] => IGNORED [file] => ))

~/TestArea $ rm –r ./1

Array( [0] => Array ( [dir] => ./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN,ISDIR [file] => ) [2] => Array ( [dir] => ./TestArea/1/ [event] => DELETE [file] => X ) [3] => Array ( [dir] => ./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 )

DELETE FILE

~/TestArea $ rm ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => DELETE [file] => X ) )

EDIT FILE

~/TestArea $ nano ./1/X Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => ACCESS [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ))

Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X ))

COPY FILE~/TestArea $ cp ./1/X ./1/Y Array( [0] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] => ./TestArea/1/ [event] => CREATE [file] => Y ) [2] => Array ( [dir] => ./TestArea/1/ [event] => OPEN [file] => Y ))

Array( [3] => Array ( [dir] => ./TestArea/1/ [event] => ACCESS [file] => X ) [4] => Array ( [dir] => ./TestArea/1/ [event] => MODIFY [file] => Y ) [5] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => Y ) [6] => Array ( [dir] => ./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ))

DIFFICULTIES ENCOUNTERED

Must have a kernel module installed on the host server for inotify events to be recorded

Problems detecting changes made to newly-created directories within the specified inotify watch point

MySQL replication is sensitive to binary log changes when one node is disconnected

Accurate prediction and parsing of inotify events for complete analysis of changes made to filesystem

Not enough time in the semester to complete integration of all features into program

CONCLUSION

Fundamentals of DiffSync program prove that file synchronization is possible for servers in remote locations

Additional work will be completed to make this program ready for a production environment

Any questions?

THE END

THANK YOU!