how and why we evolved a legacy java web application to scala... and we are still alive!

How and why we evolved a legacy java application to scala

And we are still alive !24/06/2015

I am

@karesti

The Software Dream

Software is more like Madonna

FACTS

Web applications get old (very) fast

Continuous small refactoring does not avoid long-term technical debt

From strach Refactoring

Continuous Dilemma

2014

French Job Search Website Launched in 2000

2008

Problems in 2014

High Cost Adding New FunctionalitiesHigh Technical Debt

Coupled Architecture

S O A Spaghettis Oriented Architecture

Lack of real KPI

In a 100% Linux Environment

Monolithic Architecture

DAO

ServiceBatch

MVC

RDMS

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

Jobs

External App

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

Jobs

External App

Connected User BoardCV, Mail Alert, Newsletter

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

Jobs

External App

Data

Mailing App

Connected User BoardCV, Mail Alert, Newsletter

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

Jobs

External App

Data

Mailing App

External App

Connected User BoardCV, Mail Alert, Newsletter

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

Jobs

External App

Data

Mailing App

External App

Mobile Version

Connected User BoardCV, Mail Alert, Newsletter

DAO

ServiceBatch

MVC

RDMS

Anonymous UserJob Search, Detail, Newsletter

Jobs

Jobs

External App

Data

Mailing App

External App

Mobile Version

Connected User BoardCV, Mail Alert, Newsletter

DAO

ServiceBatch

MVC

RDMSExternal

App

Connected User BoardCV, Mail Alert, Newsletter

Anonymous UserJob Search, Detail, Newsletter

Non connectedAlerts, Newsletter

Jobs

Jobs

External App

Data

Mailing App

Mobile Version

Partners

How do we fix this

Target

Where do we start

DAO

ServiceBatch

MVC

RDMSExternal

App

Jobs

Jobs

External App

Data

Mailing App

Mobile Version

Partners

Connected User BoardCV, Mail Alert, Newsletter

Anonymous UserJob Search, Detail, Newsletter

Non connectedAlerts, Newsletter

Mars – September 2014

User Board

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

User BoardUser Board Front-End

REST API

Mongo

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

Where do we start ?User Board Front-End

REST API

Mongo

Partners

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

Where do we start ?User Board Front-End

REST API

Mongo

Partners

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

Where do we start ?User Board Front-End

REST API

Mongo

Partners

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

Where do we start ?User Board Front-End

REST API

Mongo

Partners

External App

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

Where do we start ?User Board Front-End

REST API

Mongo

Partners

External App

DAO

ServiceBatch

MVC

RDMS

Data

Mailing App

Where do we start ?User Board Front-End

REST API

Mongo

Partners

External App

Data

Batch

API First

Macro

Operation

Focus on technical choices

Main Language

• Growing and Solid Community

• Powerful Frameworks and Utilities

• JVM

Backend

• REST oriented• Template Type Safe• Hot Reloading• “Simplifies” Scala• Reactive programming

Batch System

• Actors

• Scale Up – concurrency

• Scale Out – remoting

• Fault Tolerance

Front End

• Sass built in with play

• AngularJS, popular, community, experience

• Foundation, solid and easy CSS framework

Database

• Flexible schema• Document oriented makes sense with CVs• Low transactions• Application code rules the database schema– Already the case with SQL Server

• No DBA*– We rule the database as dev

Migrating Data Challenge

Status

• +10 years of candidate data

• Crucial for business

• Cannot fail really, C A N O T F A I L !!

Strategy

• Start migration as soon as possible– Started in April – Mai

• Migrate data incrementally

• Verify as much as possible

• Legacy ID

Akka Actors

• One actor per data

• Concurrence execution when possible

• Handle updates for the crucial moment => between SQL Server Stop and MongoDB Up

Madrid MUG

This section is specially dedicated to the Madrid MUG Members

SQL Model

• 8 tables for the user profile• Complex joins• SQLServer => Lost in a Linux World• Devs => Backup, dump …

(Very) Simplified Schema

MongoDB Model

• 1 document / profile• Object <> Document• Simplified model– Option Scala

• Using Reactive Mongo Driver• Using Jongo in Java

Mongo Collections

Model choices

• ++ reads / -- writes• Stable Reference data *– Sometimes sectors can change…

• Low and non risky transactions– Ex : user deletes account

Testing

Unit Testing

• Using Specs framework

• Using Mockito – Not as useful as in Java

• Some tests are not necessary – Constructors, Builders … Scala Type Safe and

Immutability

Testing the Front END

• Unit testing JS with Karma

• Selenium – Very fragile tests– Proxy Nightmare– Endless navigator problems– Just vital tests after production

API Tests are CRUCIAL

Testing the REST API

• No Mocking MongoDB

• Using Embedded Mongo

• Start and Stop Mongo once for every test– DRY data is a hard part

https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo

Always thinking on KPI

Build Measure Learn

SCALA USER GROUP

This section is specially dedicated to the Madrid Scala User Group.Thank you to Nouhoum Traoré who spoke about it at scala.io in Paris

Front-end

● Client API

● No DB Access

● Mostly Javascript Code

Front-end

● 9 % Scala● 26.2 % CSS

●64.8 % JS

Le frontend : asset pipeline

pipelineStages := Seq(rjs, digest, gzip)

Le frontend : asset pipeline

curl http://keljob.com/assets/js/e454f1013e30b783818c8efaf3a8e3a5-startup.js

HTTP/1.1 200 OKCache-Control: public, max-age=31536000Content-Length: 171997Content-Type: application/javascript; charset=utf-8Date: Tue, 14 Oct 2014 22:39:23 GMTETag: e454f1013e30b783818c8efaf3a8e3a5Last-Modified: Wed, 08 Oct 2014 12:35:10 GMT

Compressing content

import play.api.mvc._import play.filters.gzip.GzipFilter

object Global extends WithFilters(new GzipFilter()) {

...}

API : links on JSON

implicit val AccountWrites = new Writes[Account] {

override def writes(account: Account): JsValue =

Json.obj(

"id" -> account.id,

"email" -> account.email,

"creationDate" -> account.creationDate.toString(),

"links" -> Json.obj(

"self" -> routes.AccountDetailCtrl.get(account.id).url,

"alerts" ->

routes.JobAlertDetailCtrl.getAlertsOf(account.id).url,

"cv" -> routes.CvDetailController.getCvOf(account.id).url

)

)

}

Single responsibility Object : Controllers@Singleton

class AccountValidationController @Inject() (

accountValidator: AccountValidator) extends Controller

{

def validateAccount(token: String) = Action.async

{ request =>

accountValidator.validate(token).map {

case Some(account) => Ok(Json.toJson(account))

case _ =>

UnprocessableEntity(Json.toJson(InvalidToken))

}

}

Single responsibility Object : Servicesclass AccountAuthenticator @Inject() (...)

class AccountValidator @Inject() (...)

class AccountCreator @Inject() (...)

class AccountSettingsUpdater @Inject() (...)

Single Responsibility Object : Actors

import akka.actor._

...

class CvExporter(...) extends Actor { def commonBehavior(): Receive = ??? def deleteBehavior(): Receive = ??? def updateBehavior(): Receive = ???

def receive = commonBehavior orElse updateBehavior orElse deleteBehavior

}

Error Handling in Services

sealed trait NewsletterError

case object InvalidNewsletterActivationToken extends

NewsletterError

case object NewsletterUpdateError extends NewsletterError

class NewsletterActivator @Inject() (...) {

def activate(code: String): Future[Either[NewsletterError, Boolean]]

= ???

}

Error handling in controllers

class NewsletterActivationController (

newsletterActivor: NewsletterActivator) extends Controller {

def activate(token: String) = Action.async { request =>

newsletterActivor.activate(token).map {

…

case InvalidNewsletterActivationToken => ???

case NewsletterUpdateError => ???

…

}.recoverApiError("Oops !!!")

}

}

Error handling in controllers

implicit class ApiErrorRecover(result: Future[Result]) {

def recoverApiError(message: String) =

result recover {

case NonFatal(e) => InternalServerError(

Json.toJson(SimpleError(message))

)

}

}

The Team

The (original) Team

• 3 Developers and a Product Owner– Legacy + Backend + Scala– Full-Stack– Java + Backend + MongoDB

• From 5-10 years of experience• People who are able to leave their confortable

coding zone• Want to communicate

Project Management Method

SCRUM

KANBAN

Programing MF

(SOME) DIFFICULTIES

DEFINING THE INITIAL SCOPE

MVP

MVP

Maximal Viable Product

DEALING WITH NO TECHNICAL PEOPLE

Demo

We have a situation …

Y

FIF Pattern

Fancy Interface First

SCALA

Warning ! This section might contain some trolls

Personal journey to Scala

SIMPLE ??? Build Tool

"de.flapdoodle.embed" % "de.flapdoodle.embed.mongo" % "1.46.0” "org.mongodb" %% "casbah" % "2.5.0"

Reactive Futures …

Implicits

accountFuture.filter(_.isDefined).map(_.get)

Cake Pattern

DI Framework vs Cake Pattern

@Singletonclass LoginController @Inject() (accountAuthenticator: AccountAuthenticator) extends Controller

trait LoginController { this: Controller with UserServiceComponent =>

object LoginController extends LoginController with Controller with MongoDbUserServiceComponent

Loving Case Class And Constructors

case class Account( id: Option[BSONObjectID] = None,

email: String, creationDate: DateTime,

optin: Boolean = false, optoutScenario: Option[DateTime] = None, source: String = Account.DEFAULT_SOURCE,

deletionDate: Option[DateTime] = None)

Account(“chucknorris@gmail.com”, creationDate = creationDate,

optin = true)

After 2-3 months as happy as being at Machu Picchu

Loving both …

September – December 2014

Once in production

– Creating accounts more easily– Parsing CV on upload– Can apply with the CV – Follow applications

• And other awesome stuff built fast and furiously !

January – March 2015

Search and Relooking

Relooking and positioning

BatchDataMailing

App

2015Web Front-

End

User REST API

Mongo

Partners in WIP

External App Data

Search REST API

Elastic Search

Batch

SEOMost Important Challenge

Challenge 2 : Performance

• Gatling – Play!

• Comparing performance