testing in a continuous delivery pipeline - better, faster, cheaper

© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1@CoverosGene

Agility. Security. Delivered.

Testing in a ContinuousDelivery Pipeline:

Faster, Better, Cheaper

Gene GotimerSenior Architect


About Coveros• Coveros builds security-critical applications using agile methods.• Coveros Services• Agile transformations• Agile development and testing• DevOps and continuous integration• Application security analysis

• Agile & Security training• Government qualifications• DCAA approved rates and accounting• TS facility clearance

Areas of Expertise


Select Clients


Delivery PipelineProcess of taking a code change

from developers and getting it deployed into production or delivered to the customer

• Stages along the way• Later stages lead• to higher confidence• closer to production


Delivery PipelineDo we have a

viable candidate for production?


Delivery PipelineRequirement

Code

Check-in

Unit Tests

Deploy to Test

Functional Tests

Deploy to Staging

Acceptance Tests

Deploy to Pre-Prod

Quality GateTrigger

Performance Tests

Security Tests Deploy to Prod

More expensive quality gates

Rapid Feedback

No surprises


Goal is to Balance

EarlyRapid

FeedbackNo Late Surprises


Everything Can’t Be First

Do just enough of each type of testing

early in the pipeline to determine if

further testing is justified.


Defining YourDelivery Pipeline


Value Stream• List out steps from developer to production• That is the delivery pipeline• whether manual or automated

• Identify time for each step• execution time• wait time

• Helps show • where bottlenecks are• what should be automated


Pipeline Stages• Not hard-and-fast stages• Gradual change in focus

Code-focused Quality-focused Delivery-focused


Commit Stage

Commit StageRequirement

Code

Check-in

Unit Tests

Deploy to Test

Functional Tests

Deploy to Staging

Acceptance Tests

Deploy to Pre-Prod

Performance Tests


• Code-focused• Rapid feedback• 10 minutes maximum• Developers are waiting


Acceptance Stage

Acceptance StageRequirement

Code

Check-in

Unit Tests

Deploy to Test

Functional Tests

Deploy to Staging

Acceptance Tests

Deploy to Pre-Prod

Performance Tests


• Quality-focused• Is this is a viable candidate for production?


End Game

End GameRequirement

Code

Check-in

Unit Tests

Deploy to Test

Functional Tests

Deploy to Staging

Acceptance Tests

Deploy to Pre-Prod

Performance Tests


• Delivery-focused• Steps that only get done when

we are releasing• Does not begin until you are confident

there will be no surprises


Pipeline StepsCommit Stage

• Compile• Unit tests• Static analysis

Acceptance Stage• Functional tests• Regression tests• Acceptance tests• System integration• Security testing• Performance testing• Exploratory testing• Usability testing

End Game• Security testing• Performance testing• Exploratory testing• Usability testing• Packaging• Printed documentation• Release announcement


Pipeline StepsCommit Stage

• Compile• Unit tests• Static analysis

Acceptance Stage• Functional tests• Regression tests• Acceptance tests• System integration• Some security testing• Performance trend• Early exploratory testing• Basic usability testing

End Game• Mandated security test• Full load and

performance test• Continuing exploratory

testing• Focus group usability

testing• Packaging• Printed documentation• Release announcement

Do just enough testing to determine if further testing is justified.


Example: Performance Testing• Short JMeter test• On development system, no isolation• 10 concurrent users for 10,000 requests• Track the trend• Answers: “Are we getting slower or faster?”

• Full load and performance test• Dedicated environment, no other traffic• Production-sized servers• 1,000 concurrent users for 4 hours• Answers: “What is the sustained capacity and throughput?”


Example: Security Testing• Functional tests run through

OWASP ZAP proxy• During early testing• Piggy-back on existing testing• Answers: “Do we have any XSS

vulnerabilities?”

• OpenVAS system scanning• Weekly in test environment• Looks for open network ports• Looks for software with CVEs• Answers: “Is Nessus likely to find

anything?”

• HP WebInspect application security scanning• By corporate security group• Looks for black-box web

vulnerabilities• Answers: “Do we have any XSS

vulnerabilities?”

• Nessus system scanning• By corporate security group• Looks for open network ports• Looks for software with CVEs• Answers: “Is system compliant?”


Advantages of Earlier Testing• Quicker feedback cycle• Easier to fix problems

that are found• Developer still has

context of changes• Less rework on

defective product• Proactive response,

not reactive


Testing in the

Commit Stage

Code-focused


Testing in the Commit Stage• Code-focused• Developer-centric• Rapid feedback• Developer waits until complete• 10 minutes maximum

Types of Activities• Continuous integration• Compile• Unit tests• Static analysis• Dependency analysis


Unit Testing

• Unit testing is not QA!• Developer tool• Early confirmation of code behavior• Executable documentation• Fearless refactoring


Code Coverage• A tool, not a target• Measures code executed while unit tests running• NOT amount of code tested• Not covered = not tested• Covered = possibly tested


Mutation Testing• Reruns unit tests against modified versions of your code• If tests still pass, code isn’t tested• Tests quality of tests

public int foo(int i) { i--; return i;}

public int foo(int i) { i++; return i;}


Static Analysis• Early detection of coding issues• style issues• duplicate code blocks• declared but unused variables• confusing code• race conditions• SQL injection• resource leaks


Third-party ComponentsOWASP Top 10 2013:

A9-Using Components with Known Vulnerabilities

Scan your third-party libraries

Update proactively, not reactively


Testing in the Acceptance Stage

Quality-focused


Testing in the Acceptance Stage• Quality-focused• Bulk of the pipeline• Until confident that you have a

viable candidate for production

Types of Activities• Functional tests• Regression tests• Acceptance tests• System integration• Some security testing• Performance trend• Early exploratory testing• Basic usability testing


Automated Deployment• Repeatable, reliable deployments• Test that through practice• Same deploy process everywhere• You will find more reasons to deploy


Smoke Testing• After every deployment• Must be quick• Test the deployment,

not the functionality• Focus on • basic signs of life• interfaces between systems• configuration settings


Testing in the End Game

Delivery-focused


Testing in the End Game• Delivery-focused• Steps that only get done when

we are releasing• Tests that are too expensive to

do every build• time• resources• effort

• Don’t start the End Game until you are confident you won’t be surprised

Types of Activities• Non-functional tests• Mandated security testing• Full load and performance test• Continuing exploratory testing• Focus group usability testing• Packaging• Printed documentation• Release announcement


Non-functional Testing• Availability testing• Accessibility testing• Baseline testing• Compatibility testing• Compliance testing• Configuration testing• Documentation testing• Endurance testing• Ergonomics testing• Interoperability testing• Installation testing• Internationalization testing• Load testing• Localization testing• Maintainability testing

• Operational readiness testing• Performance testing• Portability testing• Recovery testing• Reliability testing• Resilience testing• Scalability testing• Security testing• Stability testing• Stress testing• Supportability testing• Testability testing• Usability testing• Volume testing

Image by Andrew Stellman via http://www.stellman-greene.com/2010/02/17/nonfunctional-requirements-qa/


Parallel Testing• Conduct long-running tests in parallel

• Upside: less elapsed time• Downside: no feedback between tests

• Should already be an expectation that these tests will pass


Summary


Summary• Early rapid feedback vs. no late surprises• Do just enough of each type of testing early in the pipeline to

determine if further testing is justified

Code-focused Quality-focused Delivery-focused


Questions?

Gene [email protected]

@CoverosGene

testing in a continuous delivery pipeline - better, faster, cheaper

Software