real-world functional programming @ incubaid

About IncubaidFunctional Programming @ Incubaid

OCamlConclusion

Real-World Functional Programming @Incubaid

Romain Slootmaekers Nicolas Trangez

Incubaid BVBA

{romain,nicolas}@incubaid.com

Twitter: @incubaid

Team Blog: blog.incubaid.com

Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid

http://incubaid.com

http://twitter.com/incubaid

http://blog.incubaid.com


OCamlConclusion

1 About Incubaid

2 Functional Programming @ IncubaidWhy FP?Current Projects

3 OCamlOCaml vs. HaskellCode Teaser

4 Conclusion



OCamlConclusion

Outline

1 About Incubaid



4 Conclusion



OCamlConclusion

Company Overview

Technology incubatorCreating & providing services to startupsNot a VC, collaboration with external partiesMain activities: Datacenter & Cloud Computing



OCamlConclusion

History

Dedigate (2000)acquired by Terremark (2005)Hostbasket (2000)acquired by Telenet (2008)DataCenter Technologies (2002)acquired by Veritas/Symantec (2005)Q-Layer (2007)acquired by Sun Microsystems (2009)



OCamlConclusion

Why FP?Current Projects

Outline

1 About Incubaid



4 Conclusion



OCamlConclusion


Development Benefits

Code qualityCorrectnessEasy refactoringPerformance

Developer performance (Getting Things Done)Developer satisfactionDeveloper quality (hiring!)

Not the average “Enterprise Developer” ;-)Mathematical background or interestsGood at logical, conceptual reasoning

Smaller prototype-to-production cycle



OCamlConclusion


Ideal fit for our Application Domains

Handling large opaque data blobs (DSS)System & application automation (SSO)Highly scalable distributed systemsLarge scale implies constant random failure

Transactions? Where, why, how?ConcurrencyImmutability of data, purity of operations, idempotentactions,. . . allow to reason about systems and theirbehaviour



OCamlConclusion


FP: more than languages

NoteFunctional Programming is not only about languages. Insight inFP concepts and techniques can bring radical improvements tothe architecture and design of software, especially inlarge-scale, highly-unreliable environments.



OCamlConclusion


Outline

1 About Incubaid



4 Conclusion



OCamlConclusion


Amplidata DSS

Next-generation large object storageRedundant storage of petabytes of big filesDispersed storage using rateless codesInitial version: C++Current version: OCaml (90%), C (9%), Asm (1%)More info @ amplidata.com


http://amplidata.com


OCamlConclusion


Amplidata DSS (Cont.)

Math

266666664

a0,0 . . . a0,na1,0 . . . a1,n

. . . . . . . . . . . . . . . . . . .an,0 . . . an,n

an+1,0 . . . an+1,n. . . . . . . . . . . . . . . . . . .an+m,0 . . . an+m,n

3777777752664

x0x1. . .xn

3775 =

266666664

b0b1. . .bn

bn+1. . .

bn+m

377777775

Dataflow

Client codec

Storage0

........

StorageN



OCamlConclusion


Arakoon

Distributed, consistent, persistent key-value storeMulti-Paxos implementationTokyoCabinet backendOCamlOpen Source, see http://www.arakoon.org/

A nursery of two 3-node clusters

A0

A1

A2

Cluster A client0 client1

B1

client2 B0

B2

Cluster B


http://www.arakoon.org/


OCamlConclusion


Baardskeerder

Local, in-process key-value databaseAppend-only B-tree-ish, persistentReplace TokyoCabinet which can’t cope with large valuesBenefits of and doesn’t kill SSD drivesOCaml (future Arakoon backend)Not publicly available yetFollow @incubaid or blog.incubaid.com for updates


http://twitter.com/incubaid

http://blog.incubaid.com


OCamlConclusion


Baardskeerder (Cont.)

Tree Construction

A B C

a b c

0: Value "A"1 : Leaf [ " a " , Outer 0 ]2 : Commit ( Outer 1)3 : Value "B"4 : Leaf [ " a " , Outer 0 ; " b " , Outer 3 ]5 : Commit ( Outer 4)6 : Value "C"7 : Leaf [ " a " , Outer 0 ; " b " , Outer 3 ; " c " , Outer 6 ]8 : Commit ( Outer 7)



OCamlConclusion

OCaml vs. HaskellCode Teaser

Outline

1 About Incubaid



4 Conclusion



OCamlConclusion


Introduction

Objective CamlObjective Categorical Abstract Machine/Meta LanguageDialect of MLDescendents include Microsoft F#, JoCaml (, Scala)Static typing using type inferenceEager evaluation (by default)Emphasis on performance, both in language features aswell as standard library implementation



OCamlConclusion


Timeline

CamlOcaml

F#ML

MirandaHaskell

Standard ML

PrologErlang

CC++

ScalaJava

C#

1970

1970

1980

1980

1990

1990

2000

2000

2010

2010



OCamlConclusion


Compiler and Runtime

Main implementation from INRIA (.fr)Bytecode and native targetsVery fast compilationStraight-forward compiler backend & code generation

Easy to reason about efficiency, predict compiler outputIf necessary (unlike GHC) native assembly can beread/interpreted by humansThanks to this, we were able to debug an obscurenon-reproducible bug in Arakoon once, which turned out tobe a flaw in one of the libraries we use. Details on request!



OCamlConclusion


Concurrency & Parallellism

Using native/system threads (1-to-1 mapping)No parallel runtime (only one OCaml execution thread atall times), a reentrant runtime is WIPLightweight “Monadic Threads” for concurrency using Lwt,Async,. . . See W.L.Harrison, “Cheap (But Functional)Threads”



OCamlConclusion


Applications

Ocsigen: Web application framework, parent project ofLwt: lightweight threadingJs_of_ocaml: OCaml to JavaScript compiler

Mirage: Xen-based exokernelCoq: Proof assistantMLDonkey: EDonkey P2P clientUnison: File synchronizationhaXe compiler: High-level multiplatform language,compiled to JS/SWF/AS3/PHP5/C++/. . .Coccinelle: Semantic patches



OCamlConclusion


Outline

1 About Incubaid



4 Conclusion



OCamlConclusion


Key Differences (1/3)

OCaml evaluation is eager by defaultI once defined (>>) as (>>) a b = a >>= fun _ −> b. Lost quitesome time debugging the unexpected runtime behaviour.No overloading (cfr. Haskell type-classes)

+ : int −> int −> int+. : float −> float −> float (notice the dot)+\ : num −> num −> num

Both structural and referential equality operators: = and ==

Optional & named argumentsHigher-order modules & functors (not the Haskell ones)



OCamlConclusion



OO supportClean semanticsQuirky syntaxType operators / coercions: :#, :>

Polymorphic variants (union types)# [ ‘ Number 1 ; ‘ F loa t 3 .14 ; ‘ Un i t ; ] ; ;− : [ > ‘ F loa t o f f l o a t | ‘Number o f i n t | ‘ Un i t ] l i s t =[ ‘ Number 1 ; ‘ F loa t 3 .14 ; ‘ Un i t ]

No purity restrictions (cfr. IO monad)Side-effects (incl. IO) possible everywhereRecord fields can be mutableIn-place mutable variables (refs) implemented on top of thisStrings are in-place mutable



OCamlConclusion



Sugared syntax: "abc".[1] , [| 1; 2; 3; |].(1) vs. String .get,Array.get

Exceptions are common, also in pure code (e.g. Not_found)Smaller “standard library” compared to Haskell PlatformSyntax extensions

Macros: (IFDEF ... THEN ... ELSE ... END)Comprehensions: [x | x <− [1;2;3]; x<2]Succinctness for libraries

Normal

calc_x ( ) >>= fun x −>calc_y ( ) >>= fun y −>. . .

lwt.syntaxlwt x = calc_x() inlwt y = calc_y() in...(* concurrent evaluation! *)lwt a = f () and b = g () in ...



OCamlConclusion


Gotcha: Value Restriction (1/4)

DefinitionThe “value restriction” is a way to maintain correct typing in thepresence of side-effects: the type of an expression can only begeneralized if the expression is a “syntactic value” (or“non-expansive expression”):

a literal or identifier: 3, ’a’, . . .an abstraction: fun x −> 2 ∗ x, . . .a constructor applied to a syntactic value: Some 1, . . .



OCamlConclusion



Example

# l e t i = fun x −> x ; ;va l i : ’ a −> ’ a = <fun># l e t r = r e f i ; ;

The type of i is ∀α.(α→ α). Without value restriction, the type ofr would be ∀α.(α→ α) ref.Since it’d be polymorphic, it could be used as type(bool −> bool) ref or ( int −> int) ref, so

r := fun ( t r ue | f a l s e ) −> f a l s e ;( ! r ) 0

would both type-check but the last expression would result in aruntime error.



OCamlConclusion



DemoHere’s what happens for real:# l e t r = r e f i ; ;va l r : ( ’ _a −> ’ _a ) r e f = { contents = <fun >}# r := fun ( t r ue | f a l s e ) −> t rue ; ;− : u n i t = ( )# r ; ;− : ( bool −> bool ) r e f = { contents = <fun >}# l e t s = r e f i ; ;va l s : ( ’ _a −> ’ _a ) r e f = { contents = <fun >}# ( ! s ) 0 ; ;− : i n t = 0# s ; ;− : ( i n t −> i n t ) r e f = { contents = <fun >}# l e t t = l e t l = r e f [ ] in fun e −> l := ( e : : ( ! l ) ) ; ( ! l ) ; ;va l t : ’ _a −> ’ _a l i s t = <fun># t 1 ; ;− : i n t l i s t = [ 1 ]# t ; ;− : i n t −> i n t l i s t = <fun># t 2 ; ;− : i n t l i s t = [ 2 ; 1 ]



OCamlConclusion



Work-around using eta-expansion

# l e t i d x = x ; ;va l i d : ’ a −> ’ a = <fun># ( i d 1 , i d "abc" ) ; ;− : i n t ∗ s t r i n g = (1 , "abc" )# l e t id ’ = i d i d ; ;va l id ’ : ’ _a −> ’ _a = <fun># id ’ 1 ; ;− : i n t = 1# id ’ ; ;− : i n t −> i n t = <fun># id ’ "abc" ; ;E r ro r : This expression has type s t r i n g but an expression was expected of type i n t# l e t id ’ ’ = fun x −> ( i d i d ) x ; ;va l id ’ ’ : ’ a −> ’ a = <fun># ( id ’ ’ 1 , id ’ ’ "abc" ) ; ;− : i n t ∗ s t r i n g = (1 , "abc" )# l e t f1 = L i s t .map i d ; ;va l f1 : ’ _a l i s t −> ’ _a l i s t = <fun># l e t f2 = fun l −> L i s t .map i d l ; ;va l f2 : ’ a l i s t −> ’ a l i s t = <fun>



OCamlConclusion


Outline

1 About Incubaid



4 Conclusion



OCamlConclusion


Rsync algo

Implementation of the classic “rsync” algorithmFull source @https://github.com/Incubaid/rsync-demo

Programming “in the large”Expose use of

OOModules, functorsImperative programming, mutability. . .

but not necessarily the best, most advisable approach


https://github.com/Incubaid/rsync-demo


OCamlConclusion


module type WEAK = sigtype tval make : u n i t −> tval from : s t r i n g −> i n t −> i n t −> tval rese t : t −> u n i tval d iges t : t −> i n tval r o t a t e : t −> char −> char −> u n i tval update : t −> s t r i n g −> i n t −> i n t −> u n i t

end

module type STRONG = sigtype tval to_hex : t −> s t r i n gval f i l e : s t r i n g −> tval subs t r i ng : s t r i n g −> i n t −> i n t −> tval w r i t e : out_channel −> t −> u n i tval read : in_channel −> t

end

module SDigest = ( structinclude Digestl e t read i c = Io . read_s t r i ng i cl e t w r i t e oc t = Io . w r i t e _ s t r i n g oc t

end : STRONG)



OCamlConclusion


open Hashopen Signaturemodule Rsync = functor (W:WEAK) −> functor (S :STRONG) −> struct

module MySig = Signature (W) ( S)

class d e l t a _ e m i t t e r s igna tu re new_fn handler =l e t bs = MySig . b lock_s ize s igna tu re inl e t b u f f e r _ s i z e = 8 ∗ bs inl e t b u f f e r = S t r i n g . c reate b u f f e r _ s i z e in

object ( s e l f )val mutable _read = 0val mutable _ f i r s t _ f r e e = 0val mutable _n_free = b u f f e r _ s i z eval mutable _ f i r s t _ t o d o = 0val mutable _prev ious_ac t ion = S t a r t bsval mutable _ f i n i s h ed = f a l s eval mutable _weak_ok = f a l s eval _weak = W. make ( )

method _examine_block b u f f e r o f f s e t leng th =l e t wd = W. d iges t _weak inmatch MySig . lookup_weak s igna tu re wd with

| None −> None| Some bs −>

l e t s t rong = S. subs t r i ng b u f f e r o f f s e t leng th ini f s t rong = MySig . bs_strong bsthen Some ( MySig . bs_index bs )else None



OCamlConclusion

Outline

1 About Incubaid



4 Conclusion



OCamlConclusion

In Retrospect

No regretsProductivity increased, less bugsBugs fixed quickly, features added easilyRuntime issues mainly due to improper usage orconfiguration (the universe is still winning!), except LwtbugsLimited tooling support compared to e.g. JVM(JMX/JConsole, remote debugging, GC tuning,. . . )

No deal-breaker, because less neededWould be some nice projects!



OCamlConclusion

Future

Current focus on OCaml: in general, best known by theteamHaskell for “compilable pseudocode”, once in a whileHaskell for future projects, why not?Introduce devs, architects,. . . of non-FP projects to the FPconcepts and strengths to increase the quality of theirwork: knowledge of FP concepts changes the way youthink about programming, even in non-FP settings



OCamlConclusion

Internships & Jobs

Interested in working on our software and building future-proofscalable computing platforms & looking for an internship or job?Come join us!

[email protected]


mailto:[email protected]

http://www.wehire.be


OCamlConclusion

Questions?


real-world functional programming @ incubaid

Technology

incubaidfunctional programming

incubaidwhy fp

incubaid ocaml conclusion1

incubaid team blog

storagen romain slootmaekers

current projects3 ocamlocaml

c current version

n 3b0230