evaluation in practice: artifact-based requirements engineering and scenarios in smart mobility...

© Fraunhofer IESE

EVALUATION IN PRACTICE: Artefact-based RE and Scenarios in Smart Mobility Domains Constanza Lampasona, Philipp Diebold, Jonas Eckhardt, Rolf Schneider

8th International Symposium on Empirical Software Engineering and Measurement 19 September 2014 - Torino, Italy

© Fraunhofer IESE

2

Study Goal

Evaluate

the artifact-based RE approach with respect to usability (G1),

its tool realization with respect to perceived usefulness (G2) and

perceived ease of use (G3),

and the acceptance of the scenario in general (G4) and in the project (G5), as well as its impact on the companies’ operational (G6) and strategic (G7) work.

© Fraunhofer IESE

3

Study Design Initial and Final Design

Study 1:

Structured review of scenarios

Study 2 and 3:

Sub-project meetings

Study 4:

Survey of EA-Plugin users

Par

t 1:

R

esu

lts

Par

t 2:

To

olin

g

Par

t 3:

C

on

cep

ts

the artifact-based RE approach with respect to usability

(G1)

its tool realization with respect to

perceived usefulness (G2) and

perceived ease of use (G3)

the acceptance of the scenario in general (G4) and in the project (G5), as well as its impact on the companies’ operational (G6)

and strategic (G7) work

Online Survey

Paper Survey

© Fraunhofer IESE

4

http://detroitsportsrag.net/wp-content/uploads/2014/02/Results1-1024x675.jpg

© Fraunhofer IESE

5

Results Artefact-based RE

20%

9%

11%

10%

11%

22%

22%

40%

45%

30%

22%

44%

67%

30%

45%

50%

44%

22%

11%

10%

10%

22%

0% 50% 100%

AQ6 (N=9, IDK=4)

AQ5 (N=9, IDK=4)

AQ4 (N=10, IDK=3)

AQ3 (N=11, IDK=2)

AQ2 (N=10, IDK=3)

AQ1 (N=9, IDK=4)

strongly disagree disagree neither agree nor disagree agree strongly agree

N Mdn Stand. test

results p-values

9 4 1.933 0.026*

10 4 1.897 0.029*

11 3 1.000 0.158

10 3 0.107 0.457

9 4 2.530 0.005*

9 4 1.933 0.026*

* significant at a 0.05 level

the artifact-based RE approach with respect to usability

(G1)

The created artifacts are reproducible.

The reference model provided all necessary elements to specify the requirements.

It was clear with which elements to model which kind of information.

The perceived productivity was high.

The reference model allows modeling all contents important for a requirements spec.

The approach is clear and understandable.

© Fraunhofer IESE

6

Results EA-Plugin

its tool realization with respect to

perceived usefulness (G2) and

perceived ease of use (G3)

08%

08%

08%

08%

15%

15%

08%

23%

08%

08%

15%

15%

08%

08%

23%

23%

23%

15%

15%

23%

31%

23%

23%

23%

46%

38%

46%

46%

54%

38%

31%

23%

31%

31%

38%

08%

15%

08%

15%

08%

15%

08%

08%

08%

0% 50% 100%

... enabled me to accomplish tasks morequickly.

... improved my performance inrequirements modeling.

... enhanced my effectiveness in modelingrequirements.

... made it easier to model requirements.

I find the EA-plugin useful for modelingrequirements.

Learning to operate the EA-plugin was easyfor me.

I found it easy to get the EA-plugin to dowhat I wanted to do.

The interaction with the EA-plugin wasclear and understandable.

I found the EA-plugin flexible to interactwith.

It would be easy for me to become skillfulat using the EA-plugin.

I find the EA-plugin easy to use.

strongly disagree disagree neither agree nor disagree agree strongly agree

N Mdn Stand. test

results p-values

11 4 1.350 0.088

11 3 0.513 0.304

11 3 1.406 0.080

10 3.5 0.791 0.214

9 4 1.081 0.139

10 4 1.897 0.029*

* significant at a 0.05 level

11 4 1.414 0.078

10 3.5 1.134 0.128

11 4 2.111 0.017*

11 4 1.706 0.044*

11 4 2.309 0.010*

I find the EA-plugin easy to use.

It would be easy for me to become skillful at using the EA-plugin.

I found the EA-plugin flexible to interact with.

The interaction with the EA-plugin was clear and understandable.

I found it easy to get the EA-plugin to do what I want to do.

Learning to operate the EA-plugin was easy for me.

I find the EA-plugin useful for modeling requirements.

… made it easer to model requirements.

… enhance my effectiveness in modeling requirements.

… improve my performance in requirements modeling.

… enable me to accomplish tasks more quickly.

© Fraunhofer IESE

7

Results Scenarios

21%

5%

29%

28%

16%

23%

32%

56%

61%

49%

16%

10%

21%

23%

SQ4 (N=38, IDK=1)

SQ3

SQ2 (N=38, IDK=1)

SQ1 (N=39)

13%

7%

7%

6%

10%

42%

30%

33%

27%

14%

28%

17%

19%

27%

20%

28%

36%

29%

44%

33%

47%

44%

21%

8%

5%

SQ10 (N=24, IDK=5, Missing=1)

SQ9 (N=27, IDK=2, Missing=1)

SQ8 (N=15, IDK=1)

SQ7 (N=15, IDK=1)

SQ6 (N=36, IDK=3)

SQ5 (N=39)

6%

5%

5%

24%

27%

15%

17%

8%

5%

11%

44%

50%

38%

44%

41%

21%

18%

26%

29%

20%

38%

36%

38%

58%

53%

49%

8%

11%

26%

14%

SQ18 (N=34, IDK=5)

SQ17 (N=30, IDK=9)

SQ16 (N=34, IDK=5)

SQ15 (N=36, IDK=3)

SQ14 (N=37, IDK=2)

SQ13 (N=38, IDK=1)

SQ12 (N=38, IDK=1)

SQ11 (N=35, IDK=4)

strongly disagree disagree neither agree or disagree agree strongly agree

the acceptance of the scenario in general (G4) and in the project (G5), as well

as its impact on the companies’ operational (G6) and strategic (G7) work

In g

en

era

l

The scenario is a reasonable view of the future.

The scenario is relevant for society in future.

The scenario could be used to communicate complex processes to the general public.

The scenario could be used to prioritize actions for implementation.

AR

AM

iS p

roje

ct

The scenario could be used for guiding the development of multicore technology.

The scenario could improve communication within the project.

How relevant is the scenario for SP3 in ARAMiS?

The scenario could have been used as a base for the work performed in SP3.

How relevant is the scenario for SP4 in ARAMiS?

The scenario could have been used as a base for the work performed in SP4.

pa

rtic

ipa

nts

’

com

pa

ny

& w

ork

The scenario is relevant for industry.

The scenario is relevant for research.

The scenario examines possible developments that could have an impact on my organization.

The scenario could be useful for my company.

The scenario could help my company prepare for possible changes.

The scenario could help my company find directions for future decisions.

The scenario could bring advantages to my work.

The scenario could be useful for my work.

© Fraunhofer IESE

8

http://www.beainstitute.org/wp-content/uploads/2013/03/pmo-lessons-learned.jpg

© Fraunhofer IESE

10

Lessons Learned - Empirical

http://eofdreams.com/data_images/dreams/telephone/telephone-01.jpg

© Fraunhofer IESE

11


http://www.stegman.com/site/wp-content/uploads/2012/05/Man-with-loudspeaker.jpg

© Fraunhofer IESE

12


http://bilder.bild.de/fotos-skaliert/philipplahm_39884949_mbqf-1405715143-36890868/2,w=650,c=0.bild.jpg

© Fraunhofer IESE

13

Take-Aways Empirical Work

https://www.bigis-schatzkiste.de/brautschmuck-zuchtperlen/images/product_images/info_images/Feder-4324.jpg

© Fraunhofer IESE

15

Take-Aways Requirements Engineering

http://www.landeswelle.de/assets/images/128825/h450_mar_w800/lupe-hand.jpg

© Fraunhofer IESE

18

Philipp Diebold M. Sc. @p_diebold Process Management Dep. MPE

Fraunhofer IESE Fraunhofer-Platz 1 | D-67663 Kaiserslautern Phone: +49 (0) 631-6800-2183 Fax: +49 (0) 631-6800-9-2183 Email: [email protected]

Thank you

Questions?

mailto:[email protected]

evaluation in practice: artifact-based requirements engineering and scenarios in smart mobility...

Science