1

D  in  real  life

By:  Liran  Zvibel,  Weka.IO  CTO

2D  in  real  life

• Defining  the  future  of  software  defined,  scale  out  storage  for  the  cloud-­‐based  datacenter  • VC  backed  company  (NVP,  Gemini),  largest  round-­‐A  of  2014  • Large,  strong  engineering  team,  many  XIV  veterans  • Founded  by  Omri  Palmon,  Maor  Ben-­‐Dayan,  Liran  Zvibel

About  Weka.IO

• Extremely  reliable  • High  performance  data  path  • Complicated  “control  plane”/“Management  code”  • Distributed  nature  due  to  HA  requirements  • Low  level  interaction  with  HW  devices  • Some  kernel-­‐level  code,  some  assembly  • Language  has  to  be  efficient  to  program,  and  fit  for  large  projects

3

Storage  system  requirements

D  in  real  life

• C  codebase  • A  lot  of  auto-­‐generated  code  from  XML  for  RPC,  clustering  code  and  external  APIs  (CLI,  GUI)  - Requires  a  complicated  build  process  - Difficult  to  understand  where  “magic”  code  came  from  • Our  own  implementation  of  Classes/polymorphism  and  templates

4

What  did  we  do  at  XIV?

D  in  real  life

• Data-­‐path  in  C++,  control  path  in  Python  (or  Java,  C#,  Scala,  etc)  - Deciding  whether  to  run  in  the  same  process  or  via  RPC,  both  options  have  downsides  

• All  in  C++  - Outside  the  scope  of  this  talk,  C++  has  many  downsides  writing  very  large  projects,  many  pitfalls  

• Google  Go  - Can’t  disable  GC,  no  generic  programming,  inheritance,  no  C  ABI.  Not  performant  • Mozilla  Rust  - Currently  a  moving  target

5

Some  other  possible  options

D  in  real  life

6D  in  real  life

“Often, programming teams will resort to a hybrid approach, where they will mix Python and C++, trying to get the productivity of Python and the performance of C++. The frequency of this approach indicates that there is a large unmet need in the programming language department.”

“D intends to fill that need. It combines the ability to do low-level manipulation of the machine with the latest technologies in building reliable, maintainable, portable, high-level code. D has moved well ahead of any other language in its abilities to support and integrate multiple paradigms like imperative, OOP, and generic programming.”

— Walter Bright, Co-designer of D

• Created  ’99  by  Walter  Bright,  Andrei  Alexandrescu,  as  a  C++  replacement  

• Work  on  D2  started  2007,  considered  stable  in  late  2010.  

• Modern  programming  language  with  native  efficiency,  but  leveraging  20  years  of  experience  making  mistakes  while  designing  C++  and  Java.  

• Combines  the  power  of  C++,  but  with  the  usability  of  languages  like  Python,  Java  and  C#.  

• Has  several  compilers  :  dmd,  gdc,  ldc

7

What  is  D?

D  in  real  life

• Smaller  community  • Stability  reminds  C++  of  the  early  2000s  • Binary  compatible  to  C,  no  source  compatibility  - We  must  “translate”  h  files  to  d  - Inability  to  inline  c  code  (that  would  otherwise  be  inlined  in  C++)  • Compilation  time  grows  if  many  templates  are  instantiated  across  different  compilation  units  • Missing  keyword  arguments  when  calling  functions  (a  la  python)

8

D  down  sides

D  in  real  life

• range  is  the  D  equivalent  of  an  iterable    • InputRange  requires  3  basic  methods:  - bool empty; - T front; - void popFront(); • foreach(elem; range) • foreach(k, v; dict) • foreach(idx, ref e; array)

9

Ranges  and  Algorithms

D  in  real  life

Output copy(Input, Output)(ref Input source, ref Output sink) if (isInputRange!Input && isOutputRange!(Output, ElementType!Input)) { foreach(c; source) sink.put(c); return sink; }

10

Simple  Algorithm

D  in  real  life

Symbol[] getSymbols(string filename) { return executeShell("nm -S " ~ filename) .output .splitLines .map!” a.split()" .filter!"4 == a.length" .map!((x) => new Symbol(x)) .filter!`a.typeFits(["d", "D", "v", "V"])` .array; }

11

UFCS  and  Pipeline  programming

D  in  real  life

bool binarySearch(T)(T[] input, T value) {    while (!input.empty) {       auto i = input.length / 2;       auto mid = input[i];       if (mid > value) input = input[0 .. i];       else if (mid < value) input = input[i + 1 .. $];       else return true;    }    return false; }

12

Slices

D  in  real  life

• It  supports  raw  pointers  if  required  • Has  native  C  ABI  so  linking  with  existing  libraries  is  easier  than  C++  • Supports  static  local  variables  • Has  Templates  (Generic  programming)  - They  are  much  easier  and  flexible  than  in  C++  - D  supports  compile  time  execution  —  CTFE  • Supports  Class  hierarchies  and  polymorphism  • Statically  typed,  but  with  very  strong  type  inference  • Supports  operator  overloading    • Inline  assembler  support

13

D  is  similar  to  C++

D  in  real  life

• It  supports  Garbage  Collection  - this  is  the  default,  but  you  can  opt-­‐out,  do  pointer  arithmetic  - Compiler  helps  you  verify  GC  is  indeed  not  being  used  - Allows  prototyping  in  a  high  level  way,  then  optimizing  GC  out  where  needed  

• It  has  inner-­‐classes    • It  has  delegates  • Vast  support  for  attributes  • Very  strong  type  introspection/reflection  allowing  “duck  typing”

14

Python  that  compiles?

D  in  real  life

@StatsCategory(“fs_driver_user"):

@RateQuery @StatUnit("Ops/Sec") { @("Number of read operations per second") IncrementalStat READS;

@("Number of write operations per second") IncrementalStat WRITES; } @StatUnit("Microseconds", 0.1) { @("Average latency of READ operations") @MeasuredQuery!`{READS}` TimerStat READ_LATENCY;

@("Average latency of WRITE operations") @MeasuredQuery!`{WRITES}` TimerStat WRITE_LATENCY }

15

Attributes  usage

D  in  real  life

• It  has  modules,  packages  and  imports  system  • Native  Array,  Tuple  and  Associative  Array  types  with  bounds  checking  • Supports  nested  functions  and  closures  - Also  very  strong  anonymous  functions/delegates  (lambda  with  a  closure)  • Has  pure  (enforced!)  functional  programming  subset  - Higher  order  functions:  reduce,  map,  filter,  compose,  curry  - supports  lazy  evaluation  of  function  parameters  

16

Python  that  compiles?

D  in  real  life

auto dumpField(U)(U field) { static if (__traits(hasMember, field, "toJSON")) { return field.toJSON(); } else static if (is(typeof({auto x = JSONValue(field);}))) { return field; } else static if (is(typeof(structToJSON!U))) { return structToJSON(field); } else static if (isArray!U) { return field.map!(dumpField!(typeof(field[0]))).array; } else static if (is(typeof({text(field);}))) { return text(field); } else { return "<unjsonable>"; } }

17

Introspection

D  in  real  life

JSONValue structToJSON(T)(ref T obj) if (is(T == struct) || is(T == class)) { string[string] dummy; auto jv = JSONValue(dummy);

foreach(i, U; typeof(obj.tupleof)) { enum NAME = __traits(identifier, obj.tupleof[i]); jv.object[NAME] = dumpField(obj.tupleof[i]); } return jv; }

18

Introspection

D  in  real  life

struct RobinHashTable(K, V, size_t maxLength) { static if (NumEntries < ushort.max-1) { alias CellIdx = ushort; } else { alias CellIdx = uint; } static if (K.sizeof % 8 < 7) { align(8) struct KV { align(1): K k; ubyte cellData; align(8): V v; } }

19

Compile  time  magic

D  in  real  life

else { align(8) struct KV { K k; align(8): V v; align(1): ubyte cellData; } }

• All  variables  are  initialized  to  default  values  • Contract  programming  • Casts  are  explicit,  with  special  keyword  • Exceptions  support  • mixin  support  • Concurrent  programming  support  - native  thread  local  storage  - native  locks  and  message  passing  mechanisms

20

D  is  for  real  SW  engineers

D  in  real  life

• static  if  • scope(exit/success/failure)  statement  • Unit  testing  support  • -­‐debug  and  -­‐release  default  support  • Versioned  compilation  (special  debug  case)  • @safe/@trusted/@system  code  • Documentation  comments

21

D  is  for  real  SW  engineers

D  in  real  life

void testing(T...)(T values) {    writeln("Called with ", values.length, " arguments.");    // Access each index and each value    foreach (i, value; values) {       writeln(i, ": ", typeid(T[i]), " ", value);    } } void main() { testing(5, "hello", 4.2); } Called with 3 arguments. 0: int 5 1: immutable(char)[] hello 2: double 4.2

22

(Hetero)  Variadic  functions

D  in  real  life

string makeHammingWeightsTable(string name, uint max = 255) {     string result = "immutable ubyte[%s] %s = [ ".format(max+1,name);     foreach (b; 0 .. max + 1) {         result ~= to!string(bitsSet(b)) ~ ", ";     }     return result ~ "];"; } mixin(makeHammingWeightsTable("hwTable")); unittest {    assert(hwTable[10] == 2);    assert(hwTable[0] == 0);    assert(hwTable[255] == 8); }

23

Mixin,  CTFE,  Unittests

D  in  real  life

Questions?Peta    Exa  Zetta      Yotta  Xenna  Weka  (1030)

• Tracing  system  - space  and  cpu  efficient  - keep  track  of  func  enter/exit  - Easy  way  to  keep  logs  • RPC  - Very  efficient  - Easy  to  use  and  declare  • CLI  • Statistics  • Events  (Telemetry)    

25

Infrastructure

D  in  real  life