groovy devops in the cloud
DESCRIPTION
This talk focuses on a set of tools to automate the infrastructure provisioning and testing on Amazon EC2 with the help of Groovy-based tools and libraries. We will explore how to leverage those to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud - elegant and groovy.TRANSCRIPT
Groovy DevOps in theCloud
01
About us02
Luciano Fiandesio
Bio: Developer, speaker, author, kitchen hacker
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/lucianofiandesio
••••
03
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
••••
04
What's this presentation about?
Our take on:
DevOps
Software Provisioning
Continuous Integration
Continuous Delivery
••••
05
Technologies
AWS - http://aws.amazon.com
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
•••••
06
Developers +Operations =
?07
Silos
08
Conflicts09
Risk
10
Agile
11
What is DevOps?
12
Infrastructure as Code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize Open Source tools
Ensure testability
•••
13
Configuration propagation
14
Changes
No Manual Changes!
15
Building a DevOps toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
Full-stack approach
••••
16
Sshoogr
17
Sshoogr Features
Remote command execution
File uploading/downloading
Tunneling
•••
18
Sshoogr Example I
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
19
Sshoogr Example II
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
20
Sshoogr Example III
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.21
Why Groovy?
Groovy is perfect choice for scripting
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
••••
22
Shameless plug
23
Puppet24
Why Puppet?
More mature than competition
Large community
Ruby-based DSL
No need to learn Ruby ;)
••••
26
Puppet example
27
Puppet provisioning
28
Puppet provisioning
29
Puppet provisioning
30
Puppet provisioning
31
Puppet state management
32
Puppet state management
33
Puppet state management
34
Puppet modules
35
Puppet modules
36
Puppet modules
37
Continuous Integration
38
Why Jenkins?
De-facto standard
Stable
There is a plugin for that!
•••
39
Services Partner
40
Gradle Example I
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
41
Gradle Example I
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
42
Gradle Example II
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.43
Jenkins example
44
In the meanwhile...
We started developing complex Puppet modules
Modules needs proper testing
...on different platforms
•••
45
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
46
PuppetUnit
47
PUnit
Simple testing tool for verifying remote server state
Uses sshoogr and JUnit
Reuse reporting features of JUnit and Jenkins
As simple as ...
••••
48
PUnit Example I
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
49
PUnit Example II
@Test
def void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
50
PUnit Example III
@Test
def void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
51
PUnit Example III
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should returne at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.52
PUnit Example III
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
53
PUnit
54
Nextproblem?
55
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
56
Amazon WebServices
57
Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
••••
58
EC2 Console
59
Gramazon
60
Gramazon
Groovy based API for interacting with EC2
Integrates with the rest of the stack
••
61
Gramazon Example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.62
Gramazon Example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
63
Costs considerations
Name Compute Units Memory Hourly Cost
t1.micro 2 0.60 GB $0.02
m1.small 1 1.70 GB $0.06
c1.medium 5 1.70 GB $0.14
m1.medium 2 3.75 GB $0.12
m1.large 4 7.50 GB $0.24
c1.xlarge 20 7.00 GB $0.58
m3.xlarge 13 15.00 GB $0.50
64
Free tier
750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of
memory and 32-bit and 64-bit platform support)
Enough hours to run continuously each month
•
•
65
Security considerations
Host operating system
Individual SSH keyed logins
All accesses logged and audited
Customer-generated keypairs
Stateful firewall
Mandatory inbound firewall, default deny mode
Signed API Calls
Require X.509 certificate or secret AWS key
••••
••
••
66
Imgr
67
Imgr
A tool for building images
Inspired by Packer
••
68
Images?
69
Supports
Shell
Puppet
••
70
Configuration Example
71
The big picture
72
Aetomation
73
Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Automate!
•••••
74
Readingmaterial
75
Groovy 2 Cookbook
76
The Phoenix Project
77
Continuous Delivery
78
Programming Amazon EC2
79
Gradle in Action
80
Technologies to follow
Vagrant - http://www.vagrantup.com/
Docker - https://www.docker.io/
Packer - http://www.packer.io/
Qemu - http://wiki.qemu.org/Main_Page
jclouds - http://jclouds.apache.org/
Cloudbees - http://www.cloudbees.com/
••••••
81
One morething...
82
It's all OpenSource!
83
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••
84
Seekingcontributors!
85
Questions?86
Paldies! Thank you!
87