conducting experiments in software industry
TRANSCRIPT
Natalia Juristo University of Oulu
& Universidad Politécnica de Madrid
FiDiPro Finland Distinguished Professor Program
Project start, January 2013
Experimental Software Engineering
Industrial Laboratory (ESEIL)
MISSION
!"#$%#$&#'()%*$%)(+%!"#$$%&'%(%)*%,*$-.,)%+/0+1#2+$)&%#$%)(+%&*34"1+%#$-.&)15%
%
6$,1+"&+%&*34"1+%#$-.&)15%"4"1+$+&&%"7*.)%)(+%)(%*)$&%((%*8%+/0+1#2+$)&%%
9.&(%8*14"1-%)(+%%+,%-./%&0#$1,#-#2.'/1"00:#+-%)*%3&45$%2'%1'%&%-#64&1"7*.)%
&*34"1+%-+;+:*02+$)%
!"#$%#&'$()*$'+$,-.$*+/0$3&$'+$1+&#$*+/0$3-$'"#$Ƴ#$+2$+'"#0&
6)117)08$+2$'"#$&53#-9,5$7#'"+.$'+$:#-#0)'#$"3:"$;/)13'*$$8-+(1#.:#$<+-./59-:$#=>#037#-'&$%03-:&$'"#$&53#-9,5$7#'"+.$'+$?+@()0#$A-:3-##03-:$
!"#$%&'$()*+&*+),$+-(./+01/++
)-+%$2$1.+314*$5$6$3)+%$.17-(*,&#*+
8(9$%*)1(9+),$+,&99$(+'$3,1(&*'*+137(:+;$,&(9+),$+;$,12&-%+0$+-;*$%2$+
<9$(7=/+!"#$"%&'(+#.1/&(:+1+%-.$++
&(+*->01%$+9$2$.-#'$()+1(9+),$++)*++'),*+(+;$)0$$(+),$'+
!"#$"%%&$"#'($)*$+,$"%)'!"#$"%&'($)(*+(),-.%/$0&"s and market-speak to 1%($2"&*$".'$&3)"*,"4$"."4#%&%$".'$&.2-*3",-.$
Fact-based management
-*$%"./*0,,1'20,$('/"($"#)')345,('3%,+'64'$(%".71'
5))4&+"0&4&6#$+-.'&,-.%/$7("8.(%%(%$".'$%6*(.96:%$-2$%-;7"*($'(<(4-)3(.6$6(+:.-4-9&(%$
$
=10>(+,<($-)&.&-.%$+".$4("'$6-$3&%6"8(.$)(*+(),-.%$-2$*("4&6#$
What Use Are Experiments for Industry?
!"#$%&'$()*+,%$+,+#-.$%/01+)--1+/-%+$2,10,3(4+)$56(-1-4&$*+
7 !"#$%&'$()*+#%-2&8$+,(+-9:$532$+5-'#,%&*-(+%,)6$%+)6,(+*09:$532$+-#&(&-(*+
;6$+%$*01)*+-/+$"#$%&'$()*+*6-018+,11-.+&(80*)%<+)-+7 =,*$+8$5&*&-(*+-(+-9:$532$+4%-0(8*+
> ?$*01)*+/%-'+,(+$"#$%&'$()+,%$+2,1&8+,)+5-'#,(<+1$2$1+
7 @(5%$,*$+5-()%-1+-(+*-A.,%$+8$2$1-#'$()+> @8$(3/<+%$1$2,()+2,%&,91$*+-/+*-A.,%$+)$56(-1-4&$*+
1. 2.
What an Experiment Is Not
!"#$%#&'$#()#&*'#+%$,"-./0$!"#$%&$'"!()*&+$$1*%"$-%"#&$,*'*/2&$%#&',$$
3+#4,"-%$#()#&*#+5#$6 78$-+5#$1#+%$92//--+*+:;$6 <2++-%$9#$:#+#&2/*=#0$%-$-%"#&$02>,?$-%"#&$)#-)/#$
@.9A#5BC#$&#)#2%#0$#()#&*#+5#$6 7D2./$"2,$2$/-%$-E$#()#&*#+5#$'2+2:*+:$)&-A#5%,;$6 <2+$:#+#&2%#$&./#,4-E4%".'9,?$*,$5-+F+#0$%-$D2./?$*,$,.9A#5BC#$
<-//#5%#0$02%2$E&-'$)&-A#5%,$6 73.&$0#)2&%'#+%$,%-&#,$02%2$E&-'$2//$)&-A#5%,;$6 8+$/$)&-A#5%,?$C2&*29/#,$52++-%$9#$5-+%&-//#0$2,$+#5#,,2&>$%-$.+0#&,%2+0$1"2%$%"#$
52.,#,$2+0$#G#5%,$2&#$$
A SE Experiment
Task i
A
B
M1
M1
Task i
M2
M2
Software process
Software process
Task i
Task i
Task i
Task i
What is Needed to Conduct an Experiment?
! "#$%&%'()%*+#, -.%#/(*%#$%&%'()%*+0#1.%#/(*%#*%'234'%#*%+5'1+#
! 6&%*#780#29#)(++24'%#
! "#$%&%'()%*+#5:$%*#1.%#+3/%#;(:$2<(:+#, =(:>:%$#2:#3#*((/0#5+2:?#+3/%#2:9*3+1*5;15*%0#@#
! "#$%&%'()%*+#$(2:?#1.%#+3/%#13+A#5:$%*#+15$B#, C#13+A#1B)2;3''B#13A%+#31#'%3+1#1D(#.(5*+#1(#;(/)'%1%#
! E%)%:$2:?#(:#1.%#%F)%*2/%:13'#$%+2?:#D%#;3:#+)'21#)3*<;2)3:1+#(*#%&%*B4($B#$(2:?#21#1D2;%#
Train people in
exchange
Researchers provide free training
Companies provide participants for the experiment
Our TDD Experiment
TRAINING EXERCISES EXPERIMENTAL TASKS
DAY 1 Reminder of Unit Testing concepts
3 Exercises to warm up
CONTROL TASK (Test-last)
DAY 2 Test-driven Development
3 TDD Exercises
DAY 3 2 TDD Exercises TREATMENT TASK (TDD)
Quality @ F-Secure TDD Experiments
• The mean for ITL is 54% of correctly implemented user stories
• The mean for TDD is 76%
• Subjects applying TDD get a 17% improvement in quality compared to subjects applying ITL
Quality @ F-Secure TDD Experiments
• The observed difference between ITL and TDD is not significant
EFFECT SIZE HYPOTHESIS
TESTING
Productivity @ F-Secure TDD Experiments
• The mean for ITL is 23% • The mean for TDD is
greater at 48%
• Subjects applying TDD show a 24% improvement in productivity compared to subjects applying ITL
• The observed
differences between ITL and TDD is significant
Productivity @ F-Secure TDD Experiments
Other Data Analysis
Dissecting TDD: Post-hoc analysis
! "#$%&'()#$*#+,-.*/.,011,%2%3%#(,3*+.(,4%,(.%,)/56%,7&*#/*72%,/)8'*#+,*37&96%$,:8)2*(;,< =)#9,/;/2%',< 0%'(,>&'(?,(.%,/9$%,< @%A)/(9&*#+,)B%&,&)(.%&,(.)#,$%'*+#,>&'(,
! =)#9C$%6%2973%#(,.)',39'(,4%)&*#+,9#,*37&96%$,:8)2*(;,,,< 0*#;,DC3*#8(%,$%6%2973%#(,/;/2%',< E,8#*(,*#/&%)'%,*#,:8)2*(;,A9&,%6%&;,FGDH,&%$8/59#,*#,/;/2%,'*I%,
! =%%$,(9,4%,/9#>&3%$,-*(.,)#,%J7%&*3%#(K,
Schedule: Our TDD Experiment
TRAINING EXERCISES EXPERIMENTAL TASKS
DAY 1 Testing tool Concepts
2 Tool Exercises 1 Motivational exercise (ITL)
BASELINE TASK (Your Way)
DAY 2 Slicing 2 Slicing Exercises CONTROL TASK (ITL)
DAY 3 Test-driven Development
3 TDD Exercises TREATMENT TASK (TDD)
Replications of Our TDD Experiment !"#$%&'$()*+,-$.&/(,01,#%$2#3.),,
4 5*6*,7,58(&),.$9(/,: ;2<$=>%$,?@,.&)$.A,2B,CD,#*%E=&#*().,: 8FG,?<#*&(A,2B,@0,#*%E=&#*().,
4 HII,7,JK$.),.$9(/,: !+$L)%3M&),2B,N,#*%E=&#*().,
!"#$%&'$()*+,-$.&/(,C1,=%3..,7,.+&=&(/,)%*&(&(/,4 5*6*,7,58(&),.$9(/,
: H3'#*(O,P,?!.)3(&*A,: 8FQ,?<#*&(A,: !<F!,?!=>*R3%A,
!"#$%&'$()*+,-$.&/(,@1,!-C,7,S3>%T*O,=3()%3+,)%$*)'$(),,4 HII,7,U(O,.$9(/,
: !%&=..3(,
4 FO)V3(,7,WW,: ;2<$=>%$,
Quality: TL vs TDD
F-Secure Elektrobit Univ. Politecnica Valencia
TL < TDD (42%) Java 31 Students No need more subjects
TL = TDD (32%) C++ 9 Professionals Need more subjects
TL = TDD (17%) Java 24 Professionals No need more subjects
Productivity: ITL vs TDD
ITL < TDD (41%) Java 31 Students No need of more subjects
ITL < TDD (21%) C++ 9 Professionals No need of more subjects
ITL < TDD (24%) Java 24 Professionals No need of more subjects
F-Secure Elektrobit Univ. Politecnica Valencia
!"#$$%$&'#$%()*+,+-'+.%/)012%3+%(4+%2&%(+5+$&6+,.7%286+%&9%.&:;#,+<%
=>>%$+#(.%2&%)"6,&5+"+-2.%)-%?4#$)28%#-(%6,&(4'@5)28%
A&-B.)0-)C'#-2%()*+,+-'+.%9&,%?4#$)28%!)0-)C'#-2%()*+,+-'+.%9&,%6,&(4'@5)28%%
!2#3$+%,+.4$2.%9&,%()*+,+-2%)-9,#.2,4'24,+.%D#5#%E%FGG%DH-)2%E%I2+.2%%
We Are Now Recruiting !
! "#$%&#$'(()*+,$-(&$.(/0%+*#1$*+2##3$*+$&#.#*4*+,$(5&$.(5&1#$(+$677$8 "#$9*''$&5+$2:#$#;0#&*/#+2$%1$0%&2$(-$2:#$2&%*+*+,$
<&##$=$3%>1$2&%*+*+,$(+$677$?(5&$'(.%'$#;0#&*/#+2$'21$
@*3#$50$2($<%.2AB%1#3$C%+%,#/#+2$
