Using Docker/Software Containers for Automated Testing

Pini Reznik

Software Testing Checklist

1 Create System and Acceptance Tests [ ] 2 Start Acceptance test Creation [ ] 3 Identify test team [ ] 4 Create Workplan [ ] 5 Create test Approach [ ] 6 Link Acceptance Criteria and Requirements to form the basis of acceptance test [ ]

7 Use subset of system test cases to form requirements portion of acceptance test [ ] 8 Create scripts for use by the customer to demonstrate that the system meets requirements [ ] 9 Create test schedule. Include people and all other resources. [ ] 10 Conduct Acceptance Test [ ] 11 Start System Test Creation [ ] 12 Identify test team members [ ] 13 Create Workplan [ ] 14 Determine resource requirements [ ] 15 Identify productivity tools for testing [ ] 16 Determine data requirements [ ] 17 Reach agreement with data center [ ] 18 Create test Approach [ ] 19 Identify any facilities that are needed [ ] 20 Obtain and review existing test material [ ] 21 Create inventory of test items [ ] 22 Identify Design states, conditions, processes, and procedures [ ] 23 Determine the need for Code based (white box) testing. Identify conditions. [ ] 24 Identify all functional requirements [ ] 25 End inventory creation [ ] 26 Start test case creation [ ] 27 Create test cases based on inventory of test items [ ] 28 Identify logical groups of business function for new sysyem [ ] 29 Divide test cases into functional groups traced to test item inventory [ ] 1.30 Design data sets to correspond to test cases [ ] 31 End test case creation [ ] 32 Review business functions, test cases, and data sets with users [ ] 33 Get signoff on test design from Project leader and QA [ ] 34 End Test Design [ ] 35 Begin test Preparation [ ] 36 Obtain test support resources [ ] 37 Outline expected results for each test case [ ] 38 Obtain test data. Validate and trace to test cases [ ] 39 Prepare detailed test scripts for each test case [ ] 40 Prepare & document environmental set up procedures. Include back up and recovery plans [ ] 41 End Test Preparation phase [ ] 42 Conduct System Test [ ] 43 Execute test scripts [ ] 44 Compare actual result to expected [ ] 45 Document discrepancies and create problem report [ ] 46 Prepare maintenance phase input [ ]

47 Re-execute test group after problem repairs [ ] 48 Create final test report, include known bugs list [ ] 49 Obtain formal signoff [ ]

From http://www.softwaretestinghelp.com/testing-checklist/

Software Testing Checklist

1 Create System and Acceptance Tests [ ] 2 Start Acceptance test Creation [ ] 3 Identify test team [ ] 4 Create Workplan [ ] 5 Create test Approach [ ] 6 Link Acceptance Criteria and Requirements to form the basis of acceptance test [ ]

7 Use subset of system test cases to form requirements portion of acceptance test [ ] 8 Create scripts for use by the customer to demonstrate that the system meets requirements [ ] 9 Create test schedule. Include people and all other resources. [ ] 10 Conduct Acceptance Test [ ] 11 Start System Test Creation [ ] 12 Identify test team members [ ] 13 Create Workplan [ ] 14 Determine resource requirements [ ] 15 Identify productivity tools for testing [ ] 16 Determine data requirements [ ] 17 Reach agreement with data center [ ] 18 Create test Approach [ ] 19 Identify any facilities that are needed [ ] 20 Obtain and review existing test material [ ] 21 Create inventory of test items [ ] 22 Identify Design states, conditions, processes, and procedures [ ] 23 Determine the need for Code based (white box) testing. Identify conditions. [ ] 24 Identify all functional requirements [ ] 25 End inventory creation [ ] 26 Start test case creation [ ] 27 Create test cases based on inventory of test items [ ] 28 Identify logical groups of business function for new sysyem [ ] 29 Divide test cases into functional groups traced to test item inventory [ ] 1.30 Design data sets to correspond to test cases [ ] 31 End test case creation [ ] 32 Review business functions, test cases, and data sets with users [ ] 33 Get signoff on test design from Project leader and QA [ ] 34 End Test Design [ ] 35 Begin test Preparation [ ] 36 Obtain test support resources [ ] 37 Outline expected results for each test case [ ] 38 Obtain test data. Validate and trace to test cases [ ] 39 Prepare detailed test scripts for each test case [ ] 40 Prepare & document environmental set up procedures. Include back up and recovery plans [ ] 41 End Test Preparation phase [ ] 42 Conduct System Test [ ] 43 Execute test scripts [ ] 44 Compare actual result to expected [ ] 45 Document discrepancies and create problem report [ ] 46 Prepare maintenance phase input [ ]

47 Re-execute test group after problem repairs [ ] 48 Create final test report, include known bugs list [ ] 49 Obtain formal signoff [ ]

From http://www.softwaretestinghelp.com/testing-checklist/

Waves of Innovation

Testing Complex Applications

6

Cargo Delivery Pipeline

Shipping Goods

8

Shipping with containers

9

Scalability with Containers

Start a “VM” in milliseconds by removing OS.

VM Container

What is Docker?

Batteries Included but Replaceable

• Docker Machine• Docker Compose• Docker Swarm• Notary• Kitematic• Docker plugins• runC and Open Container Project• Docker Trusted Registry (Enterprise Hub)

Containers Eco-system

• Orchestration• Networking• Persistent Storage• Minimal OS• Containers State• More ...

Orchestration - ApacheMesos

13

Orchestration - Kubernetes

Networking for Containers

15

Persistent Storage

Minimal OS

State of Containers

18

Software Testing Challenges

• Creating clean environments• Multi-machine Setup• Test data injections• Environment tear down• Environment freezing• Testing deployments/upgrades• More ...

Creating Clean Environments

$ time docker run busybox echo hellohello

real 0m0.279suser 0m0.012ssys0m0.004s

Multi-machine Setups

docker-compose.yml

$ docker run -d -v /var/voume_with_small_test_dataset:/data my_image

$ docker run -d -v /var/voume_with_large_test_dataset:/data my_image

$ docker run -d -v /var/voume_with_prod_dataset:/data my_image

Test Data Injection

Dependencies Injection

zookeeper:image: jplock/zookeeper:3.4.5

master:image: redjack/mesos-

master:0.21.0links:- zookeeper:zookeeper

zookeeper:image:

jplock/zookeeper:4.0.snapshot

master:image: redjack/mesos-

master:0.21.0links:- zookeeper:zookeeper

Tear Down Environments

$ time docker kill 78239d7854ed78239d7854ed

real 0m0.377suser 0m0.008ssys0m0.008s

Freezing Test Environments

- Multiple test environments- Hand over from testers to developers- Sharing environments- store broken containers as images- store state with CRIU

Testing Deployments/Upgrades

Bottom Line - Benefits for Testing

• Unittest– New development environment in a container for each

build with tests in milliseconds

• System tests– Fully functional runtime environment in a second

• End-to-end tests– Multi-machine complex systems in couple of seconds

The Future? Cloud Native Applications.

Pre-DevOps

DevOps

Post-DevOps

Future - Cloud Native Applications

Shipped

Mantl

Future - Software Supply Chains

Demo

Full CD Setup in 30 seconds

Questions?