who will test your tests?
TRANSCRIPT
WHO WILL TEST YOUR TESTS ?
Yahya Poonawala @meetykp Priti Biyani @pritibiyani
2 https://flic.kr/p/9uZhDS 2
TESTS Code works
Reduce Bugs
Reduce cost of change
Allow Refactoring
Faster development
Reduce fear
3
NOW WHEN YOU HEAR ….
4
Oh, it failed?
Don’t worry. Just re-trigger it!
5
Oh, it failed?
Strange; it passed on my machine!
6
Oh, it failed?
Run it on agent 007! It’s an agent issue!
7
Oh, it failed?
I know. It fails at 12AM!
8
9
Now, again,
WHY DO WE WRITE TESTS?
10
Tests prove that the code actually works
Tests prove that the code sometimes works
11
Tests reduce bugs
Tests itself might have bugs
12
Tests reduce the cost of change
Change? Are you kidding?
13
Tests helps in faster development.
Testing is slowing me down L
14
NON-DETERMINISTIC
TESTS
A.K.A “Flaky Tests”
15
16 http://www.trovatten.com/long-game/
17
18
http://tom-doherty.com/wp-content/uploads/2014/02/craigslist-waste-time.jpg
19
ILLUSION OF SUCCESS!
20
ILLUSION OF FAILURE!
21
22
23
DO NOT DELETE YOUR FLAKY TESTS!!
24
CAUSES OF FLAKINESS
25
Thread 1:: !" " if( name is not Yahya ) { !" " " " " "… !" " " " " "fputs( name, file ) !" " " " " "… !" " " " "} !!Thread 2:: !" " Initialize name to Yahya!!
CONCURRENCY
27
Initialize lock; !!Thread 1:: !" " Acquire lock !" " " " "if( name is not Yahya) { !" " " " " ". . . !" " " " " "fputs( name, file ) !" " " " " ". . . !" " " " "} !" " " " "Release lock !!Thread 2:: !" " Acquire lock !" " " " " Initialize name to"Yahya!" " " " " Release lock !!!
TIME
29
http://imgs.xkcd.com/comics/bug.png
EXTERNAL SYSTEM CALLS
30
UNORDERED COLLECTIONS
31
32 https://www.vardot.com/sites/default/files/2.jpg
! !# pseudo-code ! !!! !makeAsyncCall; ! !sleep(2000); ! !readResponse; !
ASYNCHRONOUS WAITING
34
!!"make asynchronous call; !!"While (response received?) !" "if timeout then !" " " "throw Test Timeout Exception !"!" else !" " "sleep(10) ! . . . !!"read response !!!
RESOURCE LEAKS
36
# //pseudo-code !FILE_NAME = “content.txt”!!// Test 1:!!"FileReader.read(FILE_NAME).to be empty !!// Test2: Write to the file!!"data = “Priti” !"FileWriter.write(FILE_NAME, data) !"FileReader.read(FILE_NAME) == data !!// Test 3: Updates the file!!"append_data = “Biyani” !"complete_data = “Priti Biyani” ! FileWrite.append(FILE_NAME, append_data) ! FileReader.read(FILE_NAME) == complete_data !!
TEST ORDER DEPENDENCY
38
INPUT / OUTPUT
39
40
WHAT’S YOUR STRATEGY?
41
STOP CALLING YOUR BUILD FLAKY!!
42
Identify Flaky tests
Quarantine
Plan Fix Tests
Monitor
43
IT’S NOT ALWAYS YOUR TESTS
44
DO NOT GET OVERWHELMED BY FAILING TESTS
45
REMEMBER!
46
LOVE YOUR TEST CODE !
47
There is no such thing as FLAKY TEST!
If its flaky, its not a test ! 48
AND THE FUTURE…
49
REFERENCES • An Empirical Analysis of Flaky tests • Eradicating Non-Determinism in Tests – Martin
Fowler
• No more flaky tests on the GO team - Pavan Sudarshan
• Top 12 reasons to write unit tests
• http://tech.just-eat.com/2015/03/30/how-to-fix-flaky-tests/
50
