AGILE MATURITY LEVELS AND ITS CORRELATION TO INCREASED THROUGHPUTAnil Varre

Lyudmila Mishra

Agenda

Agile Maturity Project Overview Technology Stack Project Goals and Challenges Agile Tool Set Process to Achieve Maturity Conclusion

Agile Maturity

• Core goal of an Agile team should be to build a deployable product in every iteration through all the phases

• Maturity Levels as defined by usLevel 1 = Continuous Build

• Unit Tests and Code Quality Tests with every commit

Level 2 = Level 1 + Continuous Deployment• All components are rebuilt and release candidates are deployed to a

systems integration environment at least once a day.

Level 3 = Level 2 + Continuous Testing• Automated integrations tests, smoke tests, security scans and

performance tests run after every deployment to systems integration environment.

Level 4 = Level 3 + Continuous Release• Automated promotion of release candidates to a UAT/Pre-Prod

environment with regular sign-offs is what defines “Done”.

Project Overview

Nature: Complete Re-Architecture of 10 year old platform

Size : 60+ team size (developers, content,

design, QA) Duration 2 Years Multi-Million Dollar Budget

Team Distribution - Onsite, Offshore, and Specialists/Consultants

Projects Goals

Business GoalsBetter, Faster, Cheaper System and its delivery

Quality GoalsFeasibilityAccountabilityFunctionalityTestability MaintainabilitySimplicityScalability. Extensibility

Project Challenges

Distributed Teams Organization Culture Poor reliability in terms of

Functionality Delivery Time Quality

Varied Technology Stack

Technology Stack

Complex Problem -> Simple Solution

• The Problem Statement:Define a way to deliver HIGH QUALITY, VALUABLE, software

in an EFFICIENT, FAST, RELIABLE fashion.

• The Solution Adopted:Develop a Build, Deploy, Test and Release Process that is fully automated produces release worthy artifacts regularly is repeatable and reliable provides quick feedback on functionality, quality, and

performance increases visibility into the health of the project clearly articulates team velocity and cadence

Tool Set Adopted

• Project Management and Issue Tracking– JIRA

• Code Coverage– JUnit– Cobertura

• Code Review/Quality – PMD– CPD– Jdepend– Java NCSS (Non Commenting Source

Statements)– CCN (Cyclomatic Complexity Number

also know as McCabe Metric)– Veracode

• Code Testing– JUnit– Selenium– Fitnesse– JMeter– DTM Data Generator

• Productivity– Custom Eclipse Settings

• Code Profiling– JProfiler

• Dependency Management– Ivy– Artifactory– Tattletale

• Database Change Management– DBDeploy

• Source Control Management– SVN

• Build And Deployment Tools– ANT– Shell Script Suite– Hudson

• Version Controlled Developer Image

Hudson

• Continuous Integration• Project Health Dashboard• Code Metrics and Quality• Continuous Builds & Deploy• Create New Projects• JBoss Restarts• Migrate Data Bases• Running & Scheduling Batch• Test Data Management in Data Base• Deploying Content• Single - Push Button Deploy• Selenium Regression Tests• Leader Board

Level 1 – Achieving Continuous Build

The Practices Every developer runs pre-commit tests on local

environment. On successful pre-commit tests, developer commits

code to central SCM repository CI server (Hudson) triggers new build when new

revision is detected. CI server repeats units tests, code review and code

analysis, publishes reports for the project and makes it visible to all.

On successful tests, new source packages, binaries and javadocs, etc are published to a common repository so they may used by downstream components.

CI server continues to build downstream components to ensure working software exists

Characteristics of the Build Scripts Common build scripts for all components belonging to

the same technology type Checked in along with same repository as code base. Branched and tagged along with code in order to

guarantee repeatable builds. Continually evolve along with codebase. Implemented incrementally to fulfill goals of a specific

maturity level Executes in stages as the change follows through the

delivery pipeline Complete ideally in 90 seconds, but no longer than 5

mins. Failed builds alert the entire team, including managers

and testers, not just developers.• Chose Ant as the build scripting tool of choice

• Quick and easy to develop customized scripts• Consistent scripting language irrespective of technology used• Easy to maintain by wide variety of team members

Level 2 – Achieving Continuous Deployment

The Practices CI server kicks off Stage 1 build scripts – this time carefully

tagging everything and exporting tagged source for builds. CI server publishes artifacts as release candidates on

successful build and stage 1 tests. CI server executes next stage build scripts to create

deployment packages or installers. The exact same installer is used to for deployment into every

environment – nothing is rebuilt from this point forward for a release candidate.

CI server reconfigures the systems integration environment using reference implementations scripts of all application containers, web servers, etc that are checked in to source control.

CI server deploys/installs the new packages on systems integration server All environment specific configurations are provided as input

to the installer using configuration and property files.

Characteristics of the Deployment Scripts

Minimal deployment instructions with appropriate levels of automation to reduce chances of errors.

Minimal set of deployment tools to reduce need for extensive training and documentation.

Same deployment tools for each component irrespective of technology to maintain consistent instructions with reduced complexity.

Use of standardized reference implementations of software platforms to ensure cohesiveness between all environments.

Use of automated pre-requisite environment verification testing to ensure fewer environment based deployment issues.

Ability to deploy an individual sub-system when possible without having to alter or update other sub-systems.

Ability to deploy entire system by chaining installation of pre-requisite software environments and all sub-systems to ensure ability to create the entire system on a brand new hardware platform easily and quickly.

Ability to use a unified deployment process into each environment to reduce variability between how changes are applied within the promotion path.

Ability to rollback back changes and restore to previous state of environment easily and quickly with simple and few instructions.

Ability to query the differences between the post-deployment state and the pre-deployment state.

Level 3 – Achieving Continuous Testing

The Practices CI server runs stage 2 tests on systems integration server

integration tests, acceptance tests, performance tests

On successful tests, installers are marked as ready to promote and archived to “release staging area”.

On a pre defined schedule CI server promotes the latest release-distribution to a QA Test environment using the exact same deployment packages and automated deployment scripts.

CI server repeats acceptance tests. CI server executes automated regression tests. QA team performs manual exploratory tests

Automation remains in pause mode until next signal from QA team.

Level 4 - Achieving Continuous Delivery

Develop a Deployment Pipeline that satisfies business needs and reflects team cadence

Begin with manual steps and then script EVERYTHING incrementally.

Push Button Release to any environment Use schedulers and chain scripts to force movement of releases along

pipeline

Manage everything in source control environments and their configurations, Development, build, test tools, etc, build and deployment scripts, test scripts, documentation,

An example

Lessons Learned Despite the following we achieved agility:

No formal CSM/CSP/etc. training for staff No dedicated product management per team No dedicated resources for project teams Not all roles staffed

Highly self directed and motivated team is necessary

Tools and best practices are simply adopted – no management mandate is necessary

Think big but act in small incremental steps

Conclusion

12 Releases since project launch on Nov 1st Only one trouble ticket from end users since

release! First attempt with Veracode scan resulted in AAA

rating Performance test results consistently meet

business requirements 100% code coverage all through project lifecycle

Easy to refactor code, extend functionality <5% PMD and CPD violation throughout project

cycle Easy maintenance

Self documented architecture and code Easy to integrate new members into team