brendan o'bra scala by the schuykill

1

Brendan O'Bra - Scala By The Schuylkill 2017

Akka ServicesThe Journey from one Big Problem to Many Small Problems

Brendan O’Bra, Software Developer, Comcast


One slide about me

20+ years as a coder“Burned out” Java/maven dude before Scala

(probably like everyone else in the room)Born , raised and lived in Northern California my entire life (somewhat of a

unicorn in that respect)I actually made a living writing Delphi in the 90’sUsed to have REALLY long hairWelcome to my stream of conciousness…


You know that guy who has the dream project ?

Everything works

The 1st time

At massive scale

All the awesome buzzwords are there?


That’s not me…

All Aboard!Toot Toot!


If there’s a mistake, it’s quite possible I’ve made it…

Things like…

“I’ll get to that on the next sprint” 6 months later, the interest has more than accrued on that Tech. Debt

“I’ll write the tests later”Later never comes. It attacks in production. We have no way to isolate it.

Guess it’s time to write the tests

“I can’t find a thing to do X, I’d better write one of those”Look harder next time, there’s a thing that does that very well


Gumby. WTF is that?

Gumby is: • Comcast’ Cloud Deployment tool • Something that provides a REST API• Provides “cloud abstraction” – one payload can create machines in many different backends

• The thing that led us to Akka ;)


What ? Another one of these things?

4 years ago , tools were much less mature

Comcast needed a tool that would work with all (OpenStack/EC2/Vsphere) cloud types

We had libraries/integrations laying around (written in java) that already worked with openstack/vpshere. These libraries provided “scope narrowing” to provide ease of use to our customers, the app developers


The History Of Gumby

Play! App built in Maven

2014 2015 2016 2017

Spray/Akka built w/SBT :The Monolith is born

Oh crap, we needmicroservices

Oh crap, how do weMove to microservices

uLIth!

Microservices

Many, Many releases

ELK Cassandra

AppDynamics

PagerDuty

Docker

Our Story Begins Here


Act I

A Science Experiment goes Production


In the beginning, there was the Gumby Monolith

One big pile of (mainly) scala code, with one and only one release artifact (all modules run in one JVM)


And all was good…

We made itWe deployed it (one gigantic instance per

tenant)It did (most) of what it was supposed to do


Monolithic Gumby Runtime Architecture

Gumby


But..

There were only 2 developers – we were slow

Customers wanted it to do new stuff + adoption rate was increasing

The product was a bit slow too...

Coupling was high

Blast Radius was large


And…

Each new tenant required a separate instance

Config for each instance was maintained by Gumby developers

Each new instance needed to have it’s config maintained

Upgrades were a real pain

It seemed like all we were doing was support


We needed to scale our team

So we useradd new_awesome_dev (s)Everyone started to git push origin feature/awesome_featureTests! Features! Yes!And then…

The collisions and “blast radius” events started. Coupling was killng us. We started to lose our way....


We were now here


But we knew we needed to be here:


The solution to this quandry was…

Microservices


Microservices – It’s all about developers!• Small, loosely coupled applications

• Well defined and relatively small in scope

• Emphasis on interfaces/APIs as collaboration ”points”. Could be REST, but does not need to be – depends on use case – make it messaging based

• “Micro” is a relative term


MicroServices == Small Changes

Allow developers to operate in the small:•Small changes•Small tests•Small interdependency•Small collateral damage to other code•Only interactions with collaborators are via API (ideally)


But, of course…


We are DevOps


What the Devs in DevOps really want is…


DevOps and Microservices : BFFs

Because there are smaller “pieces”…• There are more things to release–And more things to deploy–And more things to configure

AKA:


For which there is only one cure


We realized we need to get from...


Here: Gumby Single Tenant Architecture


TO here: Gumby Multitenant ”Logical” Architecture


Look at those arrows…


So, how do we go Monolith -> Microservices, without Greenfielding?


Cloud

Cloud-Init

Auth

Datastore

Openstack

EC2VMWare

Azure

UI

DNS Bakery

Spray

Job Control (sharded)




2014 2015 2016 2017




uLIth!

Microservices

Many, Many releases

ELK Cassandra

AppDynamics

PagerDuty

Docker


And now….

A Message from the Ops side of the brain


Ops view of managing a monolith


Ops View of managing microservices


Make a microlith (uLith)


The Lith: A means to an end

The uLith is all Gumby Microservices composed into one (not so) Microservice – the microservices still use “proper” interfaces to collaborate, but just happen to run in one JVM

This allows us to iteratively move from the monolithic codebase to microservices, without having to “green field” a top-bottom refactor of entire product, while simultaneously adding critical new functionality such as multitenancy and self service

Several uLiths run in a cluster, providing HA/H-Scalability (work can be distributed across cluster)


Shatter the Monolith!

Cloud

Cloud-Init

Auth

Datastore

Openstack

EC2

VMWareAzure

UIDNS

Bakery Job Control


Define Microservices based on Identified services

Modularize code into modules that encapsulate each microservice:-External messages (messages that are used to to collaborate with *this*

service) are identified-Internal messages are left in each module


Lexicon – the common module

Identify and isolate common objects into common message module

This is a compromise for sure, as it creates a bottleneck around the common module

In addition to having a authoritative “dictionary” of messages, circular dependencies are reduced/eliminated

Also could possibly provide a convenient dependency for external integration


Externalize EXTERNAL dependencies into common dependencies “module”

Unifying versions of things such as Spray, Akka, etc. becomes VERY important

This is a great place for an SBT Plugin ;)


Spray nice to together

- Use interfaces + system.actorOf(…) to make the services collaborate

- val jobHandlerActor = system.actorOf(JobRequestHandlerFactory.props(imageJobService, machineJobPersistentService, machineJobProxy, imageJobProxy, userAuthActor ))


Compose them into one thing (using git submodule and a build.sbt)

lazy val projectRoot = Project(id = "gumby-ulith", base = file(".")) .settings(ulithCommonSettings: _*) .aggregate( projectAcmLib,

lazy val projectAcmLib = Project(id = "gumby-acm-lib", base = file("gumby-acm/gumby-acm-lib")) .settings(ulithCommonSettings: _*) .settings(libraryDependencies += phantomDsl, libraryDependencies += phantomConnectors, libraryDependencies += phantomTestKit) .dependsOn(projectAcmCommon,…


Mainval restServiceProps = RestApiService.props(

machineHandlerFactory = machineHandlerFactoryActor, machineJobPersistentService, userAuthService = userAuthActor, machinePersistenceActor, buildMachineJobProps, imageHandlerFactoryActor, jobHandler = jobHandlerActor, imageJobHandler = imageJobHandlerActor, machineJobHandler = machineJobHandlerActor, jobDetailsHandler = machineJobHandlerActor, userMetadataLookup = userMetadataLookup, acmUserHandlerFactoryActor = acmUserHandlerFactoryActor, resourceAuthorizationService = resourceAuthorizationServiceActor, acmTenantHandlerFactoryActor = acmTenantHandlerFactoryActor )

import sslConfiguration._ logger.info(s"loaded ssl config") IO(Http) ! Http.Bind(system.actorOf(restServiceProps, "ulith-rest-service"), interface = "0.0.0.0", port = config.getInt("app.http_port"))


Auth Cloud Datastore Cloud-init UI

VMWareOpenstackEC2Bakery DNS Azure

Common Messages

Common Dependencies


Eh?

Seems stupid, right?


It is (more than) a little silly… But...

It creates a pattern in which:Modularity is increasedCoupling is DECREASEDCohesion is increasedDevelopers can think in a more “service oriented” fashion without having to

take on full Ops burden all at onceIndividual microservices can be individually peeled of and deployed

independently (outside of uLith)All modules can be released independentlyChange is incremental


There are a few side effects…


Swarm

Seed Seed Seed

ULith ULith ULith ULith ULith

ULithULithULithULith ULith




2014 2015 2016 2017




uLIth!

Microservices

Many, Many releases

ELK Cassandra

AppDynamics

PagerDuty

Docker

We resume here


Next Stop, (real) Microservices


Cloud

Cloud-Init

Auth

Datastore

Openstack

EC2

VMWareAzure

UIDNS

Bakery

Microlith



Swarm

Seed Seed Seed

Auth Cloud Model Cloud-init UI

VMWareOpenstackEC2Bakery DNS Azure


A few development challenges

Singleton can become Singleton(s) in Akka

Cluster Sharding can be hard to understand for newbs (and not so newbs)

There’s a database (Cassandra) behind it – we’re just a bunch of dopey devs, we don’t know how to admin that stuff

Akka has it’s own gotchas (closing over sender, type “sensitivity” with messaging,Cluster Startup Order,split brain)

Much consternation about serialization choices


In the end…

Cluster early, cluster oftenDocker ASAPDon’t be afraid of writing SBT pluginsImplicit conversions are like atomic weapons – they have a time and a placeDon’t fight ivy, because it will winBe strong in your resistance to using exclude(…) in sbtDon’t be a Scala bigot - use Scala to set an exampleScala *really* does help with hiring – not JUST attracting people, but attracting

the *right* peoplePractice EVERYTHING. ALL THE TIME. Cassandra. Networky things. Docker at

all points in pipelineConsider sbt test git hooks on dev machines


Tools/Library Shoutouts

Phantom – kickass Cassandra DSL/library https://github.com/outworkers/phantomAmmonite – Just like a scala worksheet, but way better https://github.com/lihaoyi/AmmoniteSbt-release – accept it’s patterns, and your releases will be something you

won’t really have to care about anymore https://github.com/sbt/sbt-releaseSbt-extras – command line abstraction for the myriad of sbt options https://github.com/paulp/sbt-extras


Contact me!

[email protected]@brendanobra

mailto:[email protected]