programming in the large (again)

8/12/2019 Programming in the Large (Again)

1/98

PROGRAMMINGin theLARGE

!

B

"

z@markhibbe
https://twitter.com/markhibberd


2/98

!

B

"

z

SYSTEMFORMULATION


3/98

How do we arrive at

Systems?


4/98

Modernizati

Old New


5/98

Productionizat

Prototype Production


6/98

Consolidati

Bigger

Small

Small


7/98

Modularizat

Big

Smaller

Smaller


8/98

Greenfiel

Legacy


9/98

Increment

BaseMore More


10/98

THESE ARENOT SYSTEM


11/98

Any organization that designs a

system will produce a design whostructure is a copy of the organizat

communication structure

Conways L


12/98

Therules andboundaries

of our systems shouldstem fromarchitecture not organisation


13/98

How the systeminteracts

What is the systemspurpose

Macro Architec

Domain Architec

How the system isbuiltMicro Architec


14/98

Systems as theunit of work

Notunit of work as asystem


15/98

!

B

"

z

SYSTEMSof

SYSTEMS


16/98


17/98

CROSS LANGUAGE CODEAS AN EXAMPLE


18/98

Code search the JVM wrong w

Pick Framework


19/98

Scala + Play

Pick Language + Framework



20/98

UI API

FILESYSTEM


Scala + Play


21/98

UI API

DB INDEXER


Scala + Play


22/98

SYSTEMS, NOTSUB-SYSTE


23/98


24/98

Code search as Systems D o m a i n A r c h

i t e c t u r e

INDEXING SEARCH


25/98

Code search as Systems

INDEXING SEARCH

CODE

CTAGS

CTAGS

application/htmlapplication/search.v1+json


26/98


INDEXING SEARCH

CODE

CTAGS

CTAGS

application/htmlapplication/search.v1+json

M a c r o A r c

h i t e c t u r e


27/98

STANDARDFORMATS HGIVE USAUTONOMY


28/98


INDEXING SEARCH

Scala

Embedded Server

OS for logging

Bourne Shell

Exuberant CTAGS

Deploy as git hook

OS for logging

OS for services


29/98


INDEXING SEARCH

M i c r o A r c h

i t e c t u r e

Scala

Embedded Server

OS for services

OS for logging

Bourne Shell

Exuberant CTAGS

Deploy as git hook

OS for logging


30/98


INDEXING SEARCH

M i c r o A r c h

i t e c t u r e

Scala

Embedded Server

OS for services

OS for logging

Bourne Shell

Exuberant CTAGS

Deploy as git hook

OS for logging

MOREAUTONO


31/98

RULEARCHITECTURE IS ANOTBOXES


32/98

Signs offailure Systems arenot autonomous

Systemsshare a domain model

Systems need to beevolved in stepSystems need to bebuilt by single tea

Systemsshare a data base


33/98


If you cant add a person to your project then you are

failing worse than you think

again

An Asid


34/98

!

B

"

z

EVOLUTION


35/98

Thinking ahead is not about avoidchange

Thinking ahead is about letting uschange atdifferent rates fordifferent problems


36/98

Indexing V2

INDEXING SEARCH

Scala

Embedded Server

OS for logging

Haskell

Static Executable

OS for services

OS for logging

OS for services


37/98

INDEXING SEARCH

Scala

Embedded Server

OS for logging

Haskell

Static Executable

OS for services

OS for logging

OS for services

M i c r o A r c h

i t e c t u r e

Indexing V2


38/98

now vslaterpriorities shift over time,can't let early tractionsacrifice long term speed

An Asid


39/98

INDEXING SEARCH

Scala

Embedded Server

OS for logging

Haskell

Static Executable

OS for services

OS for logging

OS for services

Indexing V2

DIFFERENT SYSTEMS SAMERULES

DIFFERENT SYSTEMS DIFFERENTRULES

An Asid


40/98

M a c r o A r c

h i t e c t u r e

Indexing V3

INDEXING SEARCH

CODE

CTAGS

CTAGS

application/htmlapplication/search.v1+jsonSuper-Zippy Format


41/98

M a c r o A r c

h i t e c t u r e

Search V2

INDEXING SEARCH

CODE

CTAGS

Super-Zippy Format

application/htmlapplication/search.v1+jsonSuper-Zippy Format


42/98

Notice how easy it is to think evolution in thelarge


43/98

Rates of change

domainmacro

micro


44/98

Rates of change

domainmacro

microFreedom to choose tools and technology by succeeding atdomain andprotocols


45/98

HOW DO SYSTEMSTHINK HOW DO SYSTEMSCOMMUNIC

HOW DO SYSTEMSCHANGE


46/98

Implications at

Scale


47/98

The concept ofmicro-services aregreat

An Asid

The concept ofsmall libraries aregreat

But they arenot enough withou

thinking in syst


48/98

Larger Systems of Systems


49/98

How long does it take to get1 lineof code toproduction


50/98

Larger Systems of Systems


51/98


52/98

Mistaking Modules For Syste


53/98

Mistaking Modules For Syste


54/98

Implication of AutonomyUSERS


55/98

Implication of (lack of) AutonUSERS


56/98

!

B

"

z

SYSTEMSin

ACTION


57/98

CMSs hold customer data hostage

CONTENT ANALYSIS & DATA M

Free data

Analyse dataDerive data

CMSs tied to horrible platforms

GoalsConstraints


58/98


Prototype Production1


59/98



JSP Java


60/98



1 JSP Java


61/98



Smaller Smaller

1

2 Java1 JSP


62/98



Smaller Smaller

1

21 JSP Java

JavaScala


63/98

WHAT I IMAGINDATA

WEB

CORE LEECH PORTAL



64/98

WEB / CORE / DATA / LEECHWEBSHERE

REALITY SETTI


65/98

CAUGHT THINKING

SMALMODULES- LIBRARIES- TECH


66/98

We shouldleverage architecture chang rather than force new technolo


67/98

CONTENTANALYSIS IDENTITY

CONTENTSTORE

QUERYENGINE

A Better Appro


68/98

Play 1.1 App

Our Starting Point

bl


69/98

Play 1.1 App

Enabling Moves

IDENTITY

bli hi l & A


70/98

Play 1.1 App

ProtobufsThrift

HTTP

Establishing Rules & Autonomy

sbt-assemblyOS packageContainers

IDENTITY

Protocols

Deployment

Id if i D d


71/98

Play 1.1 App

Identifying Demand

CONTENTSTORE

IDENTITY

V lid i


72/98

Play 1.1 App

Validation

CONTENTSTORE

IDENTITY

R


73/98

Play 1.1 App

Repeat

CONTENTSTORE

IDENTITY

CONTENTANALYSIS


74/98


75/98

architecture is aneveryday task


76/98

good architecture means neverhaving torewrite


77/98

All editors are bad, really bad

ONLINE CONTENT EDITIN

Something aaSAPIsReady to chan

Small team, tight deadline

GoalsConstraints

Lots of ideas, not sure what would


78/98

Spelling Identity Templates

Editor CustomerUI

D o m a i n A

r c h i t e c t u r

e


79/98


Editor CustomerUI

M a c r o A r c

h i t e c t u r e

HTTP / JSON


80/98

C Scala Scala


Editor CustomerUI

M i c r o A r c h

i t e c t u r e

Scala/JS Scala

Postgres

Postgres Postgres

syslog

jail per service

rc.d

1 executable

Minimal stack


81/98

Editor was working well


IncrementalMultiple VersiAdd not modif

Needed to add more services

GoalsConstraints

System need to keep running


82/98

Spelling Identity Templates Content

Editor Analysis CustomerUI


83/98



HTTP / JSON


84/98



Scala/JS Scala JS

C Scala Scala Scala

multilog

jail per service

daemon tools

1 executable

Minimal stack


85/98

Demand for standalone features


QuickMinimal OverhMaximum Reu

Not tied to our editor

GoalsConstraints

Run in parallel to current system


86/98




87/98

UGLY BITS


88/98



Glue

Glue Glue Glue

GlueGlue

Glue


89/98


Editor Analysis CustomerUIConfig


90/98

ARCHITECTURE SHOULD HARULES FOR HOW WESHARE COD


91/98

!

B

"

z

RECAP


92/98

RULEARCHITECTURE IS ANOTBOXES


93/98

good architecture means neverhaving torewrite


94/98

Thinking ahead is not about avoidchange

Thinking ahead is about letting uschange atdifferent rates fordifferent problems


95/98


Notunit of work as asystem


96/98

The thing aboutreal systems is

AUTONOM


97/98

!

B

"

z

FIN

@markhibb

References + More Info
https://twitter.com/markhibberd


98/98

References + More Info

Stefan Tilkov's Breaking the Monolith slides vide

Coda Hale and Ryan Kennedy on "Streamie" notification service a

Slide Deck v2v1
https://speakerdeck.com/markhibberd/programming-in-the-largehttps://speakerdeck.com/markhibberd/programming-in-the-large-againhttps://speakerdeck.com/markhibberd/programming-in-the-large-againhttp://vimeo.com/21598799http://vimeo.com/74354372https://speakerdeck.com/stilkov/breaking-the-monolith-1http://www.innoq.com/blog/st/2013/10/on-monoliths/

programming in the large (again)

Documents