brendan o'bra scala by the schuykill

Brendan O'Bra - Scala By The Schuylkill 2017

Akka ServicesThe Journey from one Big Problem to Many Small Problems

Brendan O’Bra, Software Developer, Comcast

One slide about me

20+ years as a coder“Burned out” Java/maven dude before Scala

(probably like everyone else in the room)Born , raised and lived in Northern California my entire life (somewhat of a

unicorn in that respect)I actually made a living writing Delphi in the 90’sUsed to have REALLY long hairWelcome to my stream of conciousness…

You know that guy who has the dream project ?

Everything works

The 1st time

At massive scale

All the awesome buzzwords are there?

That’s not me…

All Aboard!Toot Toot!

If there’s a mistake, it’s quite possible I’ve made it…

Things like…

“I’ll get to that on the next sprint” 6 months later, the interest has more than accrued on that Tech. Debt

“I’ll write the tests later”Later never comes. It attacks in production. We have no way to isolate it.

Guess it’s time to write the tests

“I can’t find a thing to do X, I’d better write one of those”Look harder next time, there’s a thing that does that very well

Gumby. WTF is that?

Gumby is: • Comcast’ Cloud Deployment tool • Something that provides a REST API• Provides “cloud abstraction” – one payload can create machines in many different backends

• The thing that led us to Akka ;)

What ? Another one of these things?

4 years ago , tools were much less mature

Comcast needed a tool that would work with all (OpenStack/EC2/Vsphere) cloud types

We had libraries/integrations laying around (written in java) that already worked with openstack/vpshere. These libraries provided “scope narrowing” to provide ease of use to our customers, the app developers

The History Of Gumby

Play! App built in Maven

2014 2015 2016 2017

Spray/Akka built w/SBT :The Monolith is born

Oh crap, we needmicroservices

Oh crap, how do weMove to microservices

uLIth!

Microservices

Many, Many releases

ELK Cassandra

AppDynamics

PagerDuty

Docker

Our Story Begins Here

A Science Experiment goes Production

In the beginning, there was the Gumby Monolith

One big pile of (mainly) scala code, with one and only one release artifact (all modules run in one JVM)

And all was good…

We made itWe deployed it (one gigantic instance per

tenant)It did (most) of what it was supposed to do

Monolithic Gumby Runtime Architecture

There were only 2 developers – we were slow

Customers wanted it to do new stuff + adoption rate was increasing

The product was a bit slow too...

Coupling was high

Blast Radius was large

And…

Each new tenant required a separate instance

Config for each instance was maintained by Gumby developers

Each new instance needed to have it’s config maintained

Upgrades were a real pain

It seemed like all we were doing was support

We needed to scale our team

So we useradd new_awesome_dev (s)Everyone started to git push origin feature/awesome_featureTests! Features! Yes!And then…

The collisions and “blast radius” events started. Coupling was killng us. We started to lose our way....

We were now here

But we knew we needed to be here:

The solution to this quandry was…

Microservices

Microservices – It’s all about developers!• Small, loosely coupled applications

• Well defined and relatively small in scope

• Emphasis on interfaces/APIs as collaboration ”points”. Could be REST, but does not need to be – depends on use case – make it messaging based

• “Micro” is a relative term

MicroServices == Small Changes

Allow developers to operate in the small:•Small changes•Small tests•Small interdependency•Small collateral damage to other code•Only interactions with collaborators are via API (ideally)

But, of course…

We are DevOps

What the Devs in DevOps really want is…

DevOps and Microservices : BFFs

Because there are smaller “pieces”…• There are more things to release–And more things to deploy–And more things to configure

For which there is only one cure

We realized we need to get from...

Here: Gumby Single Tenant Architecture

TO here: Gumby Multitenant ”Logical” Architecture

Look at those arrows…

So, how do we go Monolith -> Microservices, without Greenfielding?

Cloud-Init

Datastore

Openstack

EC2VMWare

DNS Bakery

Job Control (sharded)

2014 2015 2016 2017

uLIth!

Microservices

Many, Many releases

ELK Cassandra

AppDynamics

PagerDuty

Docker

And now….

A Message from the Ops side of the brain

Ops view of managing a monolith

Ops View of managing microservices

Make a microlith (uLith)

The Lith: A means to an end

The uLith is all Gumby Microservices composed into one (not so) Microservice – the microservices still use “proper” interfaces to collaborate, but just happen to run in one JVM

This allows us to iteratively move from the monolithic codebase to microservices, without having to “green field” a top-bottom refactor of entire product, while simultaneously adding critical new functionality such as multitenancy and self service

Several uLiths run in a cluster, providing HA/H-Scalability (work can be distributed across cluster)

Shatter the Monolith!

Cloud-Init

Datastore

Openstack

VMWareAzure

Bakery Job Control

Define Microservices based on Identified services

Modularize code into modules that encapsulate each microservice:-External messages (messages that are used to to collaborate with *this*

service) are identified-Internal messages are left in each module

Lexicon – the common module

Identify and isolate common objects into common message module

This is a compromise for sure, as it creates a bottleneck around the common module

In addition to having a authoritative “dictionary” of messages, circular dependencies are reduced/eliminated

Also could possibly provide a convenient dependency for external integration

Externalize EXTERNAL dependencies into common dependencies “module”

Unifying versions of things such as Spray, Akka, etc. becomes VERY important

This is a great place for an SBT Plugin ;)

Spray nice to together

- Use interfaces + system.actorOf(…) to make the services collaborate

- val jobHandlerActor = system.actorOf(JobRequestHandlerFactory.props(imageJobService, machineJobPersistentService, machineJobProxy, imageJobProxy, userAuthActor ))

Compose them into one thing (using git submodule and a build.sbt)

lazy val projectRoot = Project(id = "gumby-ulith", base = file(".")) .settings(ulithCommonSettings: _*) .aggregate( projectAcmLib,

lazy val projectAcmLib = Project(id = "gumby-acm-lib", base = file("gumby-acm/gumby-acm-lib")) .settings(ulithCommonSettings: _*) .settings(libraryDependencies += phantomDsl, libraryDependencies += phantomConnectors, libraryDependencies += phantomTestKit) .dependsOn(projectAcmCommon,…

Mainval restServiceProps = RestApiService.props(

machineHandlerFactory = machineHandlerFactoryActor, machineJobPersistentService, userAuthService = userAuthActor, machinePersistenceActor, buildMachineJobProps, imageHandlerFactoryActor, jobHandler = jobHandlerActor, imageJobHandler = imageJobHandlerActor, machineJobHandler = machineJobHandlerActor, jobDetailsHandler = machineJobHandlerActor, userMetadataLookup = userMetadataLookup, acmUserHandlerFactoryActor = acmUserHandlerFactoryActor, resourceAuthorizationService = resourceAuthorizationServiceActor, acmTenantHandlerFactoryActor = acmTenantHandlerFactoryActor )

import sslConfiguration._ logger.info(s"loaded ssl config") IO(Http) ! Http.Bind(system.actorOf(restServiceProps, "ulith-rest-service"), interface = "0.0.0.0", port = config.getInt("app.http_port"))

Auth Cloud Datastore Cloud-init UI

VMWareOpenstackEC2Bakery DNS Azure

Common Messages

Common Dependencies

Seems stupid, right?

It is (more than) a little silly… But...

It creates a pattern in which:Modularity is increasedCoupling is DECREASEDCohesion is increasedDevelopers can think in a more “service oriented” fashion without having to

take on full Ops burden all at onceIndividual microservices can be individually peeled of and deployed

independently (outside of uLith)All modules can be released independentlyChange is incremental

There are a few side effects…

Seed Seed Seed

ULith ULith ULith ULith ULith

ULithULithULithULith ULith

2014 2015 2016 2017

uLIth!

Microservices

Many, Many releases

ELK Cassandra

AppDynamics

PagerDuty

Docker

We resume here

Next Stop, (real) Microservices

Cloud-Init

Datastore

Openstack

VMWareAzure

Bakery

Microlith

Seed Seed Seed

Auth Cloud Model Cloud-init UI

VMWareOpenstackEC2Bakery DNS Azure

A few development challenges

Singleton can become Singleton(s) in Akka

Cluster Sharding can be hard to understand for newbs (and not so newbs)

There’s a database (Cassandra) behind it – we’re just a bunch of dopey devs, we don’t know how to admin that stuff

Akka has it’s own gotchas (closing over sender, type “sensitivity” with messaging,Cluster Startup Order,split brain)

Much consternation about serialization choices

In the end…

Cluster early, cluster oftenDocker ASAPDon’t be afraid of writing SBT pluginsImplicit conversions are like atomic weapons – they have a time and a placeDon’t fight ivy, because it will winBe strong in your resistance to using exclude(…) in sbtDon’t be a Scala bigot - use Scala to set an exampleScala *really* does help with hiring – not JUST attracting people, but attracting

the *right* peoplePractice EVERYTHING. ALL THE TIME. Cassandra. Networky things. Docker at

all points in pipelineConsider sbt test git hooks on dev machines

Tools/Library Shoutouts

Phantom – kickass Cassandra DSL/library https://github.com/outworkers/phantomAmmonite – Just like a scala worksheet, but way better https://github.com/lihaoyi/AmmoniteSbt-release – accept it’s patterns, and your releases will be something you

won’t really have to care about anymore https://github.com/sbt/sbt-releaseSbt-extras – command line abstraction for the myriad of sbt options https://github.com/paulp/sbt-extras

Contact me!

Brendan_obra@comcast.com@brendanobra

brendan o'bra scala by the schuykill

Software

brendan carter presentation

water balance of a rain garden - schuykill action network

mongodb + scala - meetupfiles.meetup.com/1652053/scala +...

brendan copia

performance analysis methodology - brendan · pdf...

reflection - st brendan

scala 55 / scala 100 / scala 30 - molplast...und metall sind...

javascript at ten years brendan eich brendan@mozilla

brendan sears toyota

brendan o’connor salesforce

brenna & brendan

brendan stewart

portfolio | brendan sullivan

scala days 2018 you are a -...

fishworks - brendan gregg

brendan kayes thesis

brendan d’souza

brendan filice's sunsets

founded by brendan

brendan twohig