application deployment at uc riverside

Developer Toolchain at UC Riverside

Mike KennedyEnterprise Architect

Developer Toolchain at UC Riverside

Mike KennedyEnterprise Architect

Application Deployment

About Me

About Me• Mike Kennedy


• At UC Riverside since 1995 (Student then Staff)



• Enterprise Architect




• Interests lie in web services, application architecture, integration, middleware




• Interests lie in web services, application architecture, integration, middleware

• Major projects - New SIS (Banner), UCPath, ServiceNow, New development environment, ESB

Talk Background

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS



• Began effort to modernize: .NET vs. Java




• Ultimately settled on Grails (Groovy/JVM based framework)




• Ultimately settled on Grails (Groovy/JVM based framework)

• Opportunity: Improve on application delivery processes and architecture (my focus today)

Grails

Grails

Grails Application

Tomcat App Server

Java Virtual Machine (JVM)

Operating System

F5

app-prd-1.cis.ucr.edu app-prd-2.cis.ucr.edu

APPPROD

User 1

Tomcat Session Replication

User 2

Database Connection Datab

ase C

onne

ction

HTTP HTTP

Proxy H

TTP Proxy HTTP

Puppet Server

Deployment Server

Configure

Configure

Configure

Source Control Management Server (Git)

ESB

Web Service C

all

Public Network

Private Network

Deploy

Pull

Nagios (Monitoring)

Graphite (Performance

Stats)

Dev

CIS

Commits Code

Starts

Deployments

Writes Scripts

Writes Modules

Writes M

onitoring Checks

Monitor

Col

labo

ratio

n

Grails Application and Support Architecture

https://app.ucr.edu/

Process Improvement

Process Improvement• What are the strengths/weaknesses of current

process?


process?

• What is the target state?


process?

• What is the target state?

• Identify technologies/tools that could help

Legacy Deployment Process

Legacy Deployment Process• Developer writes code and deploys


• Developer has complete control



• Little to no logging




• Potential issues with coordination




• Potential issues with coordination

• Infrastructure (Dev Ops) not in the loop

New Deployment Process Requirements


• Developers must be empowered to work asynchronously from Dev Ops - BUY IN!



• Must be logged and auditable




• Automation! Automation! Automation!





• Implicit notification of events





• Implicit notification of events

• Close collaboration between developers and dev ops teams

Application Delivery Cycle


Develop

Build

Deploy

Monitor


Develop

Build

Deploy

MonitorGraphite

Develop

Develop• Gitlab

Develop• Gitlab

• Git server (Source Control Management)

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests• Open source

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests• Open source• On premise or cloud

Develop

Develop• Slack

Develop• Slack

• Collaboration tool

Develop• Slack

• Collaboration tool• Chat

Develop• Slack

• Collaboration tool• Chat• Integrations galore!

Develop• Slack

• Collaboration tool• Chat• Integrations galore!• Free w/ special EDU pricing for advanced

features

Develop

Developer Git Server (SCM)

Collaboration

Developer

Developer

Develop


Collaboration

Push

Developer

Developer

Develop


Collaboration

Push

Notify

Developer

DeveloperReceive

Develop


Collaboration

Push

Developer

Developer

Develop


Collaboration

Push

Developer

Developer

Pull

Build• Gitlab, Slack


• Jenkins


• Jenkins• Continuous Integration Server (CI) - Builds


• Jenkins• Continuous Integration Server (CI) - Builds• Can be triggered by branch push


• Jenkins• Continuous Integration Server (CI) - Builds• Can be triggered by branch push• We use special branch names - release/x.y.z


• Nexus


• Nexus• Maven artifact repository


• Nexus• Maven artifact repository• Jenkins publishes built wars here


• Nexus• Maven artifact repository• Jenkins publishes built wars here• Can contain other artifacts (jars, zips)

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

file.groovy

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

file.groovyPush

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

file.groovyPush

NotifyReceive

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

file.groovyPush

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

file.groovyPush

Build Webhook

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration file.groovy

Push

Pull

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

Push

Pull

Build WAR

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

Push

Pull

Notify

Build WAR

Receive

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

Push

Pull

Build WAR

Build

Developer

Git Server (SCM)

CI (Build) Server


Collaboration

Push

Pull

Publish

Build WAR

Deploy

Deploy• Slack, Nexus


• Rundeck


• Rundeck• Job scheduler and runbook automation


• Rundeck• Job scheduler and runbook automation• Jobs can be prebuilt and shared


• Rundeck• Job scheduler and runbook automation• Jobs can be prebuilt and shared• Auditable

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

App Server 2

App Server 1

Deploy

Developer



App Server N

Run

App Server 2

App Server 1

Deploy

Developer



App Server N

Run Execute

App Server 2

App Server 1

Deploy

Developer



App Server N

Run Execute

Download WAR

App Server 2

App Server 1

Deploy

Developer



App Server N

Run Execute

App Server 2

App Server 1

Deploy

Developer



App Server N

Run Execute

NotifyReceive

Monitor

Monitor• Slack

Monitor• Slack

• Nagios

Monitor• Slack

• Nagios• Ping test

Monitor• Slack

• Nagios• Ping test• Application availability

Monitor• Slack


• Graphite

Monitor• Slack


• Graphite• Application Metrics

Monitor• Slack


• Graphite• Application Metrics• JVM Performance

Monitor

Monitor• Logging

Monitor• Logging

• ELK stack

Monitor• Logging

• ELK stack• Log Stash (Log Collection)

Monitor• Logging

• ELK stack• Log Stash (Log Collection)• Elastic Search (Log Indexing, Searching)

Monitor• Logging

• ELK stack• Log Stash (Log Collection)• Elastic Search (Log Indexing, Searching)• Kibana (Visualization)

Monitor• Logging


• Splunk Killer?

Monitor• Logging


• Splunk Killer?• Check out Nick Turley’s talk:

“Security Data Analytics Platform” tomorrow at 9am!

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Graphite

Monitor

App Server 2


Collaboration

App Server N

Dev Ops


Monitoring

Performance Metrics

Log Messages

Graphite

Monitor

App Server 2


Collaboration

App Server N

Dev Ops


Monitoring

Performance Metrics

Log MessagesView

Graphite

Monitor

App Server 2


Collaboration

App Server N

Dev Ops


Monitoring

Performance Metrics

View

Graphite

Monitor

App Server 2


Collaboration

App Server N

Dev Ops


Monitoring

Performance Metrics

View

Graphite

Monitor

Monitor

App Server 2


Collaboration

App Server N

Notify

Receive

Dev Ops


Monitoring

Performance Metrics

Receive

View

Graphite

Monitor

Monitor

App Server 2


Collaboration

App Server N

Notify

Receive

Dev Ops


Monitoring

Performance Metrics

Receive

View

Graphite

Monitor

App Server 2


Collaboration

App Server N

Notify

Receive

Dev Ops


Monitoring

Performance Metrics

Receive

View

GraphiteMetrics

Monitor

App Server 2


Collaboration

App Server N

Notify

Receive

Dev Ops


Monitoring

Performance Metrics

Receive

View

GraphiteView

Metrics

Demo Time

Outcomes

Outcomes• Still early


• Buy-in was quick



• Good feedback from developers and dev ops teams




• Many of the tools deployed used for non-development projects (New SIS)




• Many of the tools deployed used for non-development projects (New SIS)

• All tools deployed are free and open source!

QuestionsMike Kennedy

[email protected]

Twitter: @mike_kennedy

application deployment at uc riverside

Software