from 3 weeks to 30 minutes – a journey through the ups and downs of test automation

From 3 weeks to 30 minutes – a journey through the ups and downs of test

automation

Who am I?

• Peter Thomas– Chief Software Engineer– Operations IT, UBS Investment Bank

• Developer (mostly)• I do some architecture• I have done Testing• I talk a lot (Mentor/Coach)

• From the dark side (consulting) but getting better

Where did we start?

• Existing mainframe legacy application• 3 week manual PPT testing cycle• 12 week delivery cycle

What did we want to do?

• Belief there was a better way to deliver software

• Incremental development to deliver business value quickly

• Address the rapidly changing business landscape with flexibility in delivery

• Build quality into the solutions• Deliver the software rapidly, but in a cost

effective manner• Put the fun back into software delivery

New York London Kiev Hyderabad Hong Kong

2M trades per day 100 billions settling per day in all major currencies50+ exchanges across EMEA and APAC15 scrum teams/120 people9 applicationsProduction releases every 2 weeks

New York London Kiev Hyderabad Hong Kong

200 commits per day1000 artefacts updated per day1 commit every 5 minutes peak

New York London Kiev Hyderabad Hong Kong

24 Build Targets60+ Test Targets800 Automated Functional Tests10, 000 Unit/Integration Tests7, 000 Behavioural Tests

But……..

Our tests were…..

ComplicatedObscure

Random failuresSlow to run

Difficult to fix

“The TDD rut”

ComplicatedObscure

Random failuresSlow to run

Difficult to fix

Test the Right Thing and

Test the Thing Right

When all you have is a hammer, everything looks like

a nail

Why do you test?

Why do you test?

• Because TDD tells me so?• Because (insert favourite method

here) says I should?• So I meet the 80% coverage metric?

Why do you test?

• To accept the solution• To understand and document the solution• To prove its not broken• To find the unknown unknowns• To help us design and build new features• To help us explore what is really needed• To show it won’t crash under load, to

show it is secure (to test the ‘ilities)…?

Why do you test?

Agile testing Quadrants – Lisa Crispin, Janet Gregory

Testing Purposefully

The Right Thing At The Right Level

UnitComponent

System

The Right Thing At The Right Level

UnitComponent

System

•Tests a single class with no dependencies•If dependencies like Spring Context, Database used then called Unit Integration•Tests technical correctness and robustness•Very specific, a failing test indicates an issue in a specific class•Difficult to perform on poor quality code•Very fast to run, should run on the developer’s desktop in the IDE

The Right Thing At The Right Level

UnitComponent

System

•Tests a group of components which are integrated to perform a business relevant function•Can test technical or business correctness, but should be expressed in Domain concepts•Specific, a failing test indicates problems in that component•Easier to perform on poor quality code, provided component boundaries are clear•Can be quick to run, doesn’t need the full application, should run on developers desktop

The Right Thing At The Right Level

UnitComponent

System

•Tests a system at its boundaries as a ‘black box’•Primarily testing for business correctness•Not Specific, a failing test could be caused anywhere in the system flow•Easy to perform on legacy applications, requires little code modification•Slow to run, can be fragile, may not run on developers desktop

What We Wanted

What We Had

Unit tests which weren’t really Unit TestsEnd to End tests when unit tests would have been sufficientDuplicate and redundant End to End tests

The TDD Cycle

TDD?@Test public void shouldBeEmtpyAfterCreation() {   ReportObject aTrm = new

ReportObject();  assertNull(aTrm.getEligibleTrm());   assertNull(aTrm.getProcessedEvent());

  assertNull(aTrm.getPayloadXml()); }

@Test public void shouldCorrectlySetAttributesViaConstructor() {   ReportObject aTrm = new ReportObject(eligibleObject, REPORTABLE_XML);   assertEquals(eligibleObject, reportableTrm.getEligibleTrm());   assertEquals(REPORTABLE_XML, reportableTrm.getPayloadXml()); }

@Test public void shouldCorrectlySetFieldsViaSetters() {   ReportObject aTrm = new ReportObject();  aTrm.setEligibleTrm(eligibleObject);   aTrm.setProcessedEvent(child);   aTrm.setPayloadXml(REPORTABLE_XML); 

assertEquals(eligibleObject, aTrm.getEligibleTrm());   assertEquals(child, aTrm.getProcessedEvent());   assertEquals(REPORTABLE_XML, aTrm.getPayloadXml()); }

The Hollow Egg

The Hollow Egg98 Tests2.5K LOC

30 Tests200 LOC

RSpec model

Outside In - The TDD Spiral

Make the Intent Clear

How to achieve acceptance without showing your IDE or

log file to the users

Unit Test Naming?

testProcessError()

whenWorkItemIsManuallyAssignedThenClientRuleShouldBeSetToManualOverride()

shouldAllowAnActioningWorkItemToBeUpdated()

Test Data Nightmare

What Do You Demo?

What Do You Demo?

Executable Specification

Improve Testing Stability

Avoiding the Broken Windows syndrome

Separate Progress & Regression Tests

Speed-up Through Parallelism

Identify Unstable Tests

Quarantine Unstable Tests

Avoid External Dependencies

Introduce Fakes

Avoid Time-Dependent Tests

Test Isolation

Asynchronous Testing Headache

Don’t!

• Does your test need to be asynchronous?

• 80/20 rule?• Create synchronous test runner

harness

Asynchronous Testing using Events

So…?

Treat your Tests Like you Treat your Code

“it’s just a test class” is not an excuse

Clean Code applies to tests too

Think about Why You are Testing

Specification tests for internal quality

Business tests for external quality

Think about Who You are Testing For

More people are interested in your tests than you may think

Zero Tolerance to Instability

“It runs OK on my machine” is not a valid response

Interested in a career at UBS?peter.thomas@ubs.com

@peterrhysthomaspeterrhysthomas.wordpress.com