true reusable code - devsum2016

Stockholm

Get Social#devsum16

Who am I ?

Sr. Solution Consultant with LastMileLink Technologies, a CitySprint UK Innovation Lab, the leading Same Day UK network, often cited as amongst top 5 in Europe.

: #Lazaredd

:https://uk.linkedin.com/in/lazared

Process Flow.

Topics

• Software Defined “Everything”

• Geo-Spatial Fleet Analysis using Big-Data

• Truly Reusable Code – FlowBaseProgramming

Architect

TADev

CommunityNetwork

Marketplace

TechnologyInfrastructure

Data

Source: Sangeet Paul Chaudry , http://platformed.info/platform-stack/

Community, Network, Marketplace

TechnologyInfrastructure

Data

CommunityNetwork

Marketplace

TechnologyInfrastructure

Data

Defunct or Trailing Hanging On Or Declining Thriving & Growing

Leveraging Data produced by Community, Network & Marketplaceis Key to

Platform Success.

From Pipes to Platforms & API Economy

Platform Security – Emphasis on Data

Platform Security – Data Component

Software Defined “Everything”

Process Flow.

“Software is Eating the World”

Image : Pivotal

Marc Andreessen , WSJ 2011

Cloud Natives

$6B $50B $41B

$25B $33.5B

11

SPEED

UBIQUITY*SCALE

SAFETY

(MOBILE and APIs

Four key patterns

12

Day One Day Two and BeyondDeliver Continuously

DevSecOps and Rugged

How we do it

13

Build Reliable Systemsfrom

Unreliable Components

Invite Chaos

14

What a Cloud Native is NOT:

•Just Configuring Infrastructure

•Just Orchestrating Containers

•Composing Distributed Systems

•Supporting Ad-Hoc General Purpose Automation

Conway’s Law.

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the

organization's communication structure.

Melvyn Conway, 1967

Invoke the Inverse Conway Maneuver

Breaking the Monolith & Scale

How deep is deep enough?

An example - Mantl.io

Image Source: Mantle.io

Infrastructure Metrics, Monitoring and Microservices Simulation

Image Source: Instana, Image: Spigo and SIMIANVIZ

https://github.com/adrianco/spigo

Command and Control

Omakase

omakaseəʊməˈkaseɪ,əʊˈmakəseɪ/noun1.(in a Japanese restaurant) a type of meal consisting of dishes selected by the chef.2."we had the five-course omakase"

Source:Wikipedia

“ Rails is omakase (DHH) - David Heinemeier Hansson “

SPRING CLOUD

23

http://cloud.spring.iohttps://network.pivotal.io/products/p-spring-cloud-services

Omakase Distibuted Systems

Zeta Architecture - Technologies That Work

• Dynamic compute resources• Common storage platform• Real-time application support• Flexible programming

models• Deployment management• Solution based approach• Applications to operate a

business

* This is a pluggable architectureImage Source: courtesy of Jim Scott @ MapR

PolyglotProgramming?

Fleet Geospatial Analysis

Process Flow.

Blue signals a pick-upRed signals a drop-off

Sample of how one driver’s journey looks like

Used for:• Viewing the base unit of analysis

Process Flow.Demand heat map

Demand heat mapHeat map of pickup locations density

Used for:• Optimising resource allocation• Identifying areas for potential expansion

Process Flow.K-means clustering analysis – 40 centres

K-means clustering analysis – 40 centresEmployed the K-means algorithm to identify clusters of pickup points

Used for:• Validating against current service centres map• Identifying areas for potential expansion

Process Flow.K-means 100 centres

Can we be better placed ? Where can we plan the next M&A?

K-means 100 centresHigher granularity clustering

Used for:• Assessing the frequency of pickups for

micro-clusters (e.g. villages, neighbourhoods)

• Directing drivers to hotter waiting areas

Process Flow.Geographical supply & demand

Geographical supply & demandPickup locations shown vs to routes

Used for:• Improving likelihood of parcel pickup while on-route

Demand variation across time

0 3 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230.0

4.5

9.0

13.5

18.0

Expected parcels allocated to cluster 41 (Stevenage)

Time of day

Exp

ecte

d pa

rcel

sDemand variation across time

Used for:• Positioning couriers in the right place at the right

time

For each demand cluster we calculated the frequency of pickups per hour

Solution Summary

The solution outline

• Data science capabilities of Spark, easy to use with SQL knowledge• Map plotting on Esri– heat mapping, zoom in/out capabilities, real-time• High-performance due to in-memory processing capabilities of Spark• Can work with large data sets due to high performance disk-based

data access in Hadoop File System (HDFS)• Can import data from EDW• All delivered with Bigstep on their high performance Full Metal Cloud

IaaS-Paas

Shout out to Bigstep

Process Flow.Heavy Lifting done by:

Shout out to Bigstep

Process Flow.

Shout out to Bigstep

84 Lines of Scala code in Spark

Process Flow.

FlowBasedProgramming(FBP)

Switch and …

DevSum 2016

Stockholm

Please vote for this session on DevSum App

: #Lazaredd

:https://uk.linkedin.com/in/lazared