Stairway to the cloud or can we take the highway?

Taivo Liik

Who Am I

Taivo.Liik@Playtech.com

Current Situation

• 3000 physical servers• 25 000 Virtual Machines• Over 100 deployable application

servers

Current Situation (2)

• Developments happen across the Globe• New versions and fresh features every

week• How to scale them easily?• How to cut down the cost?

Current Approach

• Moving more and more to micro services approach• New micro services are developed constantly• Needs to:

– Be scalable– Be discoverable– Have connectivity to different locations– Be highly available– Be configured– Have ability to store the logs for troubleshooting– Have up to date middleware

Drawing a parallel

Platform as a Service

• Requirements– Easy deployment– Run and scale applications– Possibility to make fast application upgrades

• There are several cloud products to accomplice it.• Do we need:– Open sourced?– Commercial?– Public?– Private?

Open sourced or

commercial?

Public or

private?

What did we choose?

• Open sourced

• Private

Cloud product

• We chose Cloud Foundry as our cloud platform

Drawing a parallel (2)

From VM to cloud

• No permanent storage in Cloud Foundry for applications

• One port opened for application to be accessed outside the cloud

• Flexible scaling – application lifecycle can be very small

• Application can connect to outside world over as many ports as needed

From VM to cloud (2)• Where to store application start-up

configuration?• Where logs can be written?• How application monitoring will work?• How middleware is deployed?• How application deployment changes?• How to know how many application nodes are

running? Do we need to know?

Application logs

• No permanent storage like in VM• Solution needed due–We have more than 100 deployable

application servers– It quite an hassle to read logs from different

data sources

• Push logs to Apache Kafka–Clustered for high availability

Log solution

• Camus jobs are pulling the logs from Kafka to Hadoop Distributed File System (HDFS)

• Reading the logs can be done in single place• On top of Hadoop Hue with Hive and Oozie

Log solution (2)

Keystores & Configuration

• Needs to be stored somewhere • Can be packaged inside the

application package• Can be stored in some site central

location

Keystores & Configuration (2)

• We are using Git

Keystores & Configuration (3)

• Keystores needs to be in a secure location• They are different per site• Needed for making trusted

connection• We are using Vault

Monitoring

• Status was pulled form apps• One port – cannot be wasted• Also having metrics running to the

Graphite on top of Whisper database

Monitoring (2)

• Needs to be highly available• Use Graphite with Influx database• The switch gave us additional tools

like Grafana• Monitoring pulling service can pull

status from Influx DB

The road so far

The road so far (2)

The road so far (3)

The road so far (4)

Service Discovery

• Information about nodes was in the database

• How it works with dynamic scaling?

Middleware

• In virtual machine we had middleware preinstalled• Upgrades took time

Middleware (2)

• manifest.yml• Cloud Foundry middleware in its

blobstore• Custom build packs are supported

Load balancer

• Past–Each VM needed to be in the LB pool–Each VM needed to have firewall hole

opened–DNS records needed to be created

• Now–Define that the application is in the cloud–Cloud infrastructure makes the rest

Security

• Each application is

–In specific org•Inside specific space–Inside application container

Orgs• Orgs can be used for different environments in

single cloud:–Production– Staging– Integration–Testing–Development–Performance– Some other purpose

Did we take the stairway or highway?• We needed to change:–VM based procedures–Middleware and application deployments–Modify existing automation–Keystore storing policy– Error troubleshootings–Network layout–Capacity calculations– Service discovery

Did we take the stairway or highway? (2)

• For single application it isHIGHWAY

• Cutting down the cost infrastructure dependencies• Taking private cloud approach it is

STAIRWAY

Sometimes the stairs have to be taken to get to the highway for faster developments and deployments.

Questions?

Thank You