cs589 paper presentation - what is in unison? a formal specification and reference implementation of...
DESCRIPTION
Overview of the paper "What's in Unison? A Formal Specication and Reference Implementation of a File Synchronizer" by Benjamin C. Pierce Jerome Vouillon presented at Oregon State University for "Domain Specific Languages" class on May 20th 2014. Presentation time: 20 minTRANSCRIPT
WHAT’S IN UNISON?A FORMAL SPECIFICATION AND REFERENCE IMPLEMENTATION
OF A FILE SYNCHRONIZER
Presentation type: paper presentationClass: CS 589 – Domain Specific Languages
Presenter: Sergii ShmarkatiukDate: 5/20/2014
2
TARGET DOMAIN: FILE REPLICATION AND SYNCHRONIZATION
File synchronizatio
n
Replication
MirroringDeployment
Cloud storage
3
COMMON USE CASE: CLOUD STORAGE AND SYNCHRONIZATION BETWEEN DEVICES
4
COMMON USE CASE: DEPLOYMENT OF WEB-APPLICATION
development instance production instance
http://localhost/MyApp http://myapp.com
3. upload
2. test
ftp://@myapp.com/var/www/html/
1. edit files
4. test
/var/www/html/MyApp
5
COMMON USE CASE: DEPLOYMENT OF WEB-APPLICATION
development instance production instance
http://localhost/MyApp http://myapp.com
4. test
/var/www/html/MyApp ftp://@myapp.com/var/www/html/
1. edit files
5. test
2. edit files
3. sync
+
6
FILE SYNCHRONIZATION SOFTWARE
Rsync
Beyond Compare
Synchronize It!
Dropbox
Google Drive
Box.net
OneDrive
Manual synchronization
Cloud storage
7
SOFTWARE DEPLOYMENT TOOLS
Puppet
Chef
Capistrano
CFEngine
Ansible
8
UNISON
File synchronization toolCommand line interfaceImplemented in OCaml (DSL) and C
(tool)Available for all major platforms
(UNIX, Win, Mac)Unlike rsync, Unison is not included
into basic UNIX distributions
9
PAPER CONTRIBUTIONS
Paper presents mathematical model, DSL, mechanics and challenges of file synchronization
Authors proved some properties of file synchronization operations using Coq
Authors described the gap between idealistic representation and actual tool implementation
10
QUESTIONSKeying Xu, Chao Peng:What is the semantic domain of Unison? Is it a deep or shallow embedded DSL?
Unison is a deep embedded DSL
11
UNISON: SYNTAX AND SEMANTIC DOMAIN
Current states of the replicas 2 file trees Archive (last
synchronized state)
Synchronized replicas 2 file trees Archive (last
synchronized state)
Syntax Semantic domain
sem
A B
~
A B
=
12
QUESTIONSChao Peng:What is the basic object of Unison?
13
UNISON: BASIC DATA STRUCTURESFILESYSTEM
type name = stringtype contents = stringtype properties = string
type fs = Dir of properties * dContents| File of properties * contents| Symlink of contents| Botand dContents = (name * fs) list
OCaml
14
UNISON: BASIC DATA STRUCTURESUPDATE DETECTION
type prevState = DIR| FILE| SYMLINK| ABSENT type ’a leafUpdate = LeafSame| LeafUpdated of ’a * ’a option
OCaml
15
UNISON: BASIC DATA STRUCTURESUPDATE DETECTION
type updateItem = Same| Updated of updateContent * prevState| Errorand updateContent =UCDir of properties leafUpdate * updateChildren| UCFile of properties leafUpdate * contents leafUpdate| UCSymlink of contents leafUpdate| UCAbsentand updateChildren = (name * updateItem) list
OCaml
16
UNISON: BASIC DATA STRUCTURESRECONCILIATION
type direction = Conflict| LeftToRight| RightToLeft| Equal type transportInstr =Instr of updateItem * updateItem * direction| NoInstr| Problemtype transportInstrTree = Node of transportInstr * transportInstrListand transportInstrList = (name * transportInstrTree) list
OCaml
17
UNISON: BASIC OPERATIONS
• Comparison of two file trees• Description of detected difference
Update detection (buildUpdates)
• Building set of transport instructions
Reconciliation (reconcile)
• Performing transport instructions• Giving user the opportunity to verify
changes
Propagation (propagate)
18
UNISON: SEMANTIC FUNCTIONSFILESYSTEM OPERATIONS
• assoc• lookupPath• reverse• mem• fsKind• remove• deletions• buildUpdates• buildUpdateChildren
19
UNISON: SEMANTIC FUNCTIONSUPDATE DETECTION
• hasErrors• hasErrorsChildren• propagateErrors• noConflictInstr• reconcileNoConflict• reconcileNoConflictChildren• leafDirection• combineDirections• uassoc• reconcile• reconcileChildren• reconcileLeft
20
UNISON: SEMANTIC FUNCTIONSRECONCILIATION
• leafApply• updateArchive• updateArchiveChildren• emptySource• copyRec• copyChildren• copy• checkNoUpdates• replaceRec• replace
21
UNISON: SEMANTIC FUNCTIONSPROPAGATION
• performInstrLeaf• newProps• unchangedProps• performInstrDir• performInstr• propagateLocally• propagate• propagateInChildren
22
QUESTIONSKeying Xu:How do authors deal with modeling gap between the reference implementation and the specification?
Authors describe ‘modeling gap’ limitations in their paper
23
UNISON: THE “MODELING GAP”
Functional program (Ocaml) Returns new replicas
without changing content Written as if it “owns”
filesystems Regards filesystems as
simple, mathematical tree structures
Assumes that all operations can be implemented 111111111111
Treats archive as full-blown filesystem
Imperative program (C) Modifies real filesystems in-
place Runs with live filesystems
123123 Operates on real
implementations of filesystems (POSIX, NTFS, …)
Deals with operations that might be impossible to implement
Stores just a fingerprint of each file’s contents
Reference implementation (DSL)
Real implementation (software tool)
24
QUESTIONSBrent Carmer:What is the connection between Unison and your DSL?Panini Patapanchala:Relation with the version control and what aspects you can take from this paper.
• SCMF-DSL also uses concept of replication
• SCMF-DSL also operates with file trees
• SCMF-DSL also detects file changes to perform such automatic actions as version numbering
25
UNISON VS SCMF-DSL
Unison
SCMF-DSL
Deplo
ym
ent
Replication Version control
Version numberingConflicts
26
UNISON VS SCMF-DSL
Takes into account only latest synchronized state
Allows synchronization only between latest states
Treats both file trees as equal sources of changes (everything is writable)11111111111
There is no defined direction for replication 1111111111111111111
Operates with file trees Replicates contents of
filesystem Might generate incompatible
changes (conflicts)
Saves information about all synchronized states
Allows to roll back to previous states
Treats file trees as primary and secondary (can be writable or read-only)
Operates with certain direction for replication: from primary replica to secondary replica
Operates with version trees Replicates contents of version
control system Does not generate
incompatible changes
Unison SCMF-DSL
27
UNISON VS SCMF-DSL
A B
~
A B
=
sem
Unison: file trees
SCMF-DSL: version trees
sem1
x
2
3
57
8
9
11
x
4
6
x
10
1
x
2
3
57
8
9
11
x
4
6
x
10
12
28
UNISON VS SCMF-DSL
A B
~
A B
=
sem
Unison: file trees, 2 platforms (A, B)
SCMF-DSL: version trees, N platforms (P1, P2, … PN)
sem1
x
2
3
57
8
9
11
x
4
6
x
10
1
x
2
3
57
8
9
11
x
4
6
x
10
12
29
QUESTIONS
Brent Carmer:Does the user ever construct things using the types listed in the reference implementation?
NO
User uses real implementation (tool) instead of reference implementation
(DSL)
30
QUESTIONSAmin Alipour:How can they make sure that function synch is run atomically?Brent Carmer:How do they use Coq to verify their reference implementation?Panini Patapanchala:Maximal runs are unique can you justify the theorem with an example.
31
QUESTIONS
Authors use Coq to prove following properties of their DSL: • Laziness is safe (replication with itself is safe )• Mirroring is a special case (replication with
previously synchronized state o and replica a gives replica a)
• Maximal runs are unique (it is impossible to generate two different synchronizations on the same two replicas a and b)
• Success in the absence of conflicts (if replication does not generate conflicts first time, it won’t generate conflicts next time as well)
32
QUESTIONSAmin Alipour:The paper assumes that there are only two replica of filesystem's. Is that right? If so, how it can scale synchronization to more than two replicas?Rui Qin:How about more than two replicas, does it also work?Panini Patapanchala:I feel the paper explained the base cases for reconciliation and presently the more important problems are the one presented in future scope like multi-replica synchronization for more number of replicas.
NO
Unison works only with pairs of replicas
33
QUESTIONSAmin Alipour:What is the relation of conflict as described in the paper and merge in git?
Chao Peng:Is Unison easily extensible? can you conclude some low-level or high-level aspects of Unison?
34
QUESTIONSPanini Patapanchala:The buildupdate of the implementation is a bit like imperative implementation than functional.
This is partly because of mixed nature of OCaml language -it incorporates functional, imperative and object-oriented paradigms