Proprietary and Confidential

Continuous Integration using Hudson and Fitnesse

Norman Boccone | Lead Engineer | twitter: @dropslowlyJennifer Wong | Lead QE Engineer | twitter: @jenlwong

Vasu Durgavarjhula | Director of Development | twitter: @d_vasu

2Proprietary and Confidential

Overview

► Who we are ?► The Challenge► Tools and Processes► Test Strategy► Conclusion

Proprietary and Confidential 3

• We are a leading provider of information and analytics solutions for life science researchers

• Our goal is to help life science researchers get biological insights from their data quickly and reliably

• Our products are used by thousands of researchers worldwide

Ingenuity Systems

Partial Customer List

Tagline

Experimental Data Search

?

Publishing and Collaboration

Cancer

Enormous volume and complexity of biological

and chemical data…

Enormous volume and complexity of biological

and chemical data…

Hard to synthesize the picture of what that data

means in a broader biological context, and how the pieces work together…

Hard to synthesize the picture of what that data

means in a broader biological context, and how the pieces work together…

Traditional publishing models are good for

learning but not applying knowledge …

Traditional publishing models are good for

learning but not applying knowledge …

Toxicology Biomarkers PharmacogenomicsDiscovery

Use understanding of disease

mechanisms to identify and

validate targets

Elucidate biological mechanisms of drug action and

toxicity

Identify and prioritize novel biomarkers by

understanding role in disease pathways

Understand mechanisms behind

differential response to R(x)

Proprietary and Confidential 4

Scientists Need a Tool to Make Good Decisions From Complex Biological Data

Discover existing relationships and function, as well as inference of relationships that may not have been studied in a particular context yet

The Ingenuity® Knowledge Base Is the core technology behind all of Ingenuity’s products and services

THE INGENUITY KNOWLEDGE BASEANALYSIS

HYPOTHESIS GENERATION

VISUALIZATION

Generate testable hypotheses, build pathways, ability to inference

Get answers by asking detailed and in-depth scientific questions

ENABLES SCIENTIFIC UNDERSTANDING

ECOMMERCE ENABLEMENT

PATHWAY, REAGENT &

GENE SEARCH

► Structured - to capture relevant details

Scientific statements are modeled into Findings using the Ingenuity Ontology

► Expert Review Process - for accuracyFindings go through QC process

► Comprehensive - leverage knowledge in one placeLargest scientific knowledge base of its kind with modeled relationships between proteins, genes, complexes, cells, tissues, drugs, pathways and diseases

► Timely - for access to up-to-date knowledge Findings are added weekly

The Ingenuity® Knowledge Base How is it different?

THE INGENUITY KNOWLEDGE BASE

Proprietary and Confidential 7

The Challenge

Proprietary and Confidential 8

Single product

Proprietary and Confidential 9

multiple products

Ingenuity Product Search

Reagent Portals for BD, Sigma, Thermo Fischer

Ingenuity Variant Analysis

Next Generation Sequencing Analysis Reports

10

Proprietary and Confidential

Large monolithic code base

Proprietary and Confidential 11

Modular Services

UserManagement

ContentService

Master Graph

User Data

Views & Reports

iReport

Single Sign On

Proprietary and Confidential 12

Long release cycles

Proprietary and Confidential 13

Faster release cycles

Proprietary and Confidential 14

Continuous Integration

Speeding up the feedback loop

Stakeholders Developer

Proprietary and Confidential 15

Do all of this while supporting the existing product

Proprietary and Confidential 16

Approach• Move to a Service Oriented Architecture

– Don’t miss Jeff’s talk on Sunday 10:45 am “Learning to Love Services: Experiences from the Transition to a Service-oriented Architecture”

• Develop a test strategy focused on automated testing

• Build a continuous integration system• Adopt agile product management• Improve infrastructure to create and deploy

services• Foster a Dev/Ops culture

Proprietary and Confidential 17

Continuous Integration Workflow

Nightly Build

(Clover)

ApplicationBundle

Run JUnitTests

Hudson Dashboard(JUnit, Fitnesse summary, Code Coverage)

FitnesseBundle

SVN

Deploy Application

Deploy Fitnesse

Run Fitnesse Tests

(Nightly suite)

Fitnesse Wiki(Test history, Details,

Test Case Management)

Link

pu

blish

pu

blish

publish summary

Commit(Test Cases)

Proprietary and Confidential 18

Tools and Processes

Proprietary and Confidential 19

Service Oriented Architecture

Proprietary and Confidential 20

SOA: Before• lots of modules, separate but dependent

– easy to develop on but also easy to add circular dependencies

– hard to keep track of purpose/value of all the modules

• product consisted of bunches of modules put together– reference was typically to latest dev code– to ensure "latest," individual modules of a product

were rebuilt for every release– junit tests were run every time as well

• new product version meant build everything, test everything– builds were slow

Proprietary and Confidential 21

SOA: “Current”• components: module grouped together.  Access

through api module only– less chance of circular dependencies– easier to comprehend modules– more flexibility to change code: just do not change

api• components built/tested separately

– faster builds– less repetitive testing

• some shared modules still exist (they are not part of any component)

Proprietary and Confidential 22

SOA: Conversion Notes• find a logical group of modules• Convert groupput group into hierarchical

structure, hide things behind an api layer• convert outside modules to use apis• see Jeff's talk

Proprietary and Confidential 23

Build System

Proprietary and Confidential 24

Build System: Before• in-house ant scripts

– lots of control but it required lots of resources– no real industry standard

• global scripts/property files, overrides allowed– offered consistency but (busy) people would

override instead of follow pattern– scripts got complicated

• anyone could add a new feature– if anyone wanted a new feature, just tell them to

add it– inefficient additions, different code styles made

scripts hard to read– very little dependency management

• multiple repositories for binary files

Proprietary and Confidential 25

Build System: “Current”• maven

– not much control, and very rigid  (a good thing, in a way)

– steep learning curve– different from previous way of doing things– standard structure; difficult to not follow structure– hierarchical structure easily followed– versions for all dependencies well defined– easy to add new features (after a short burst of

intense agony)– only one repository

Proprietary and Confidential 26

Build System: Conversion Notes• get a maven repository manager (nexus? Turn on

search!)• add antrun plugin to publish artifacts to old

system• choose a simple component to start, make lots of

little submodules (1 module per artifact, plus assembly)

• manually add needed dependencies to repository• build up company parent pom, use hierarchy• Put all plugin management into parent• Key maven plugins: buildnumber, buildhelper,

release, assembly,

Proprietary and Confidential 27

Build System: TODO• cleaner release plugin• continue learning maven• contribute to plugins• Improve internal maven FAQ• Look into git

Proprietary and Confidential 28

Continuous Builds

Proprietary and Confidential 29

Continuous Builds: Before• Hudson (no plugins)

– auto build start for each module, with artifacts published

– manual build start for products– shell scripts to copy artifacts to release dir

Proprietary and Confidential 30

Continuous Builds: “Current”• Hudson

– dev master for auto build of modules/submodules, auto published to repository manager (dev area)

– release master for components/products, auto published to repository manager (release area)

– modules/submodules built on checkin– components/products built on schedule or on

demand– process includes various tests and deployments

(Jen)

Proprietary and Confidential 31

Continuous Builds: Conversion Notes• use Jenkins, not Hudson (more activity for

Jenkins)• startup new, clean servers

– copy/create job that we want• Try to make Jenkins be the portal to everything • fingerprinting• Plugins we use: clover, fitnesse, findbugs,

checkstyle, disk-usage, downstream build view, (custom) dashboard view, promoted builds, ssh slaves

Proprietary and Confidential 32

Continuous Builds: TODO• better job rollup• job templates• more maven/hudson plugins• submit hudson plugin(s) to community

Proprietary and Confidential 33

Deployment: Before• proprietary shell/perl scripts• targeted to install app only (not machine set up)• multiple proprietary property config scripts

Proprietary and Confidential 34

Deployment: “Current”• Puppet

– Configures machines and apps– Work on our machines and EC2

• Use of .erb files everywhere for property configuration

Proprietary and Confidential 35

Deployment: Conversion Notes• modify tar format• moved to one tomcat• used erb files• split properties into app

Proprietary and Confidential 36

Deployment: TODO• standard server configuration types (e.g. “build

machine”, “app machine”, …)• more puppet scripts

Proprietary and Confidential 37

Visibility

Proprietary and Confidential 38

Visibility: Before• cfmap (IT server diagnostic tool)• Hudson status page• FitNesse server

Proprietary and Confidential 39

Visibility: “Current”• cfmap• Hudson status page• Hudson dashboard• extra tests built in to Hudson• FitNesse server• Service status page

Proprietary and Confidential 40

Visibility: Conversion Notes• use Hudson plugins, with drill down• get everything to report in a standard way (junit

xml)

Proprietary and Confidential 41

Visibility: TODO• more widgets for the dashboard• Integrate outside reports into dashboard

– cfmap – status page– sdm? (service discovery manager)

Proprietary and Confidential 42

Test Strategy

Proprietary and Confidential 43

Testing For CI/CD

• Test Automation is key• How and what to automate

Cohn

Mezaros

Crispin

Proprietary and Confidential 44

Test Challenges

• Testability: some products are difficult to test at lower levels

• Legacy Apps: one of our main products was a ‘Legacy App’. Tests used to look like this:

Manual Tests

Silk Tests

Unit

Proprietary and Confidential 45

Test Challenges

• varying levels of coverage– some of our newer components have great

coverage– other components have lower coverage: legacy,

proof of concept

Proprietary and Confidential 46

Solutions

• Test and code refactoring• Legacy Apps: Strangulation approach

– automate new and refactored features– incremental work on tests: reserve time in each

iteration for adding to tests• Change in culture: team ownership of tests and

status• Cycle time for ipa was 2.5 weeks• Now for most of our services, it is 20-40 minutes

Proprietary and Confidential 47

Test Infrastructure and Environments

Proprietary and Confidential 48

Process: Build Promotion

• Build Promotion through Quality GatesDev

• Build succeeds

• Junit tests pass

auto

Continuous Integration (CI)

• Deployed to CI

• Automated

• Functional Tests Pass

auto

Stable

• Deployed to multiple nodes in Stable

auto

Beta

•Deployed to Beta

Proprietary and Confidential 49

Test Environments

CI Environment- Single nodes- functional tests

Stable Environment- Multiple nodes- Deployment Tests- System and performance tests- Manual/exploratory

Service1 Service2

Service3 Service4

Service5 Service6

Service1

Service2

Service3

Service4

Service5

Proprietary and Confidential 50

Test Types

Proprietary and Confidential 51

Test Types: Deployment and Health

• Each of our components has a built-in status page

Proprietary and Confidential 52

Test Types: Deployment and Health

• Status page– Reflects app status– Resource availability: DB Connections, Services

Proprietary and Confidential 53

Test Types: Deployment and Health

• Information is used for:– Deployment– Health Monitoring– Service Compatibility and Dependency

checking

Proprietary and Confidential 54

Test Types: FitNesse

• FitNesse is a wiki-based web server test tool– Helps abstract test definition from technical

implementation– Provides visual reporting and result history tracking

Proprietary and Confidential 55

Test Types: FitNesse

• The FitNesse Server – http://localhost:8080/FitNesse.UserGuide.TwoMinuteExample

http://localhost:8080/FitNesse.UserGuide.TwoMinuteExample?pageHistory

Wiki view Editing the wiki Test Results

Proprietary and Confidential 56

Test Types: FitNesse

• Integration with Hudson- Views, drilldown, configuration

- Test results converted to junit format

View of fitnesse test job in Hudson

Proprietary and Confidential 57

Drill downs on fitnesse test results in Hudson

Proprietary and Confidential 58

FitNesse Plugin Configuration

Convert results to Junit format

Proprietary and Confidential 59

Publish FitNesse Results and Coverage

Proprietary and Confidential 60

Test Types: FitNesse• What we use it for:

– Functional tests– Integration tests– UI Tests (com.jbergin.HtmlFixture)– DB Tests (dbfit)

• What we’ve done with it that is different– Use as execution framework for more complex tests – Extension of fitnesse server for data-driven tests– json fixture – pass in javascript– Execution of Selenium tests

Proprietary and Confidential 61

Test Types: FitNesse

• Lessons learned– FitNesse is good for straightforward verification of

data– To do more, you have to get creative– Fixture and test ownership needs to be a shared

responsibility

Proprietary and Confidential 62

Test Types: Selenium• We fire off Selenium tests in two ways: via

Hudson job or through FitNesse.• Test results converted to junit format for display in

hudson

Hudson job in dashboard:

Hudson job view:

Proprietary and Confidential 63

Test Types: Selenium

• Hudson job configuration: trigger execution on Selenium grid using Ant

Proprietary and Confidential 64

Test Types: Selenium

• Second way we fire off Selenium tests is using a FitNesse fixture: WebTest

Proprietary and Confidential 65

Test Types: Backward Compatibility

• No staging environment• How do we know that when we release a new

service version to prod that it won’t break?• Service version compatibility

– Leverage Service Discover Manager (SDM) and Status pages to check for availability of services

• Jar compatibility– static code analysis could detect changes in

method signature but not underlying object or serialization changes

Proprietary and Confidential 66

What’s Next: Getting to the End of the Road

Proprietary and Confidential 67

What’s Next

• What do we need to do to get to the end of the path to Continuous Integration and Deployment?

• Continue to invest in standardized tools and processes– Maven– Puppet– Jenkins– Metrics and Immune Systems

• Continue to improve our test coverage and practices

• Culture:– Continue to support, reinforce, educate

Proprietary and Confidential 68

The Finish Line

• Continuous deployment• Reduced Cycle Time• Improved quality• Predictable metrics• Reduced risk

Proprietary and Confidential 69

• Slides are posted here: http://www.slideshare.net/jenlwong/ingenuity-svcc-ci-presentation-20111007

• Plugs– Learning to Love Services: Experiences from the

Transition to a Service-oriented Architecture• Speaker: Jeff Green 10:45 AM Sunday

– Ingenuity is hiring: http://www.ingenuity.com/company/careers.html

• Q&A• References

– Todd. Test Automation Pyramid – review. Retrieved September 29, 2011 from: http://blog.goneopen.com/2010/08/test-automation-pyramid-review/

– Humble, Jez, and Farley, David. Continuous Delivery. Boston: Pearson Education Inc, 2011. Print.– Crispin, Lisa, and Gregory, Janet. Agile Testing: A Practical Guide for Testers and Agile Teams. Boston:

Pearson Education Inc, 2011. Print.