survey on requirements engineering tools
DESCRIPTION
REFSQ Industry Track 2011TRANSCRIPT
Survey on Requirements Engineering Tools 1 /26
Survey on Requirements Engineering ToolsREFSQ Industry Track 2011
Juan M. Carrillo de [email protected],Joaquín Nicolás,
José L. Fernández Alemán,Ambrosio Toval
Universidad de Murcia, Spain
Christof EbertVector, Stuttgart, Germany
Aurora VizcaínoUniversidad de Castilla-La
Mancha, Spain
Essen, Germany, March 29, 2011
Survey on Requirements Engineering Tools 2 /26
Contents
1 Introduction
2 Related work
3 Classification framework for RE tools’ capabilities
4 Research methodologyResearch goalsInstrumentationExperimental procedure
5 ResultsParticipantsExperimental resultsDiscussionThreats to validity
6 Conclusions and future workConclusionsFuture work
Survey on Requirements Engineering Tools 3 /26
Introduction
Introduction
Benefits from automated support to RERE tools and support to the RE processRE tools and developer’s expectationsWhat are the RE tools’ desirable features?How are they supported by current RE tools?Description of the state-of-the-art on RE tools
Current RE tools identificationFramework selection and adaptationSurvey conduction
Updated overview on RE tools’ capabilities and potentials
Survey on Requirements Engineering Tools 3 /26
Introduction
Introduction
Benefits from automated support to RERE tools and support to the RE processRE tools and developer’s expectationsWhat are the RE tools’ desirable features?How are they supported by current RE tools?Description of the state-of-the-art on RE tools
Current RE tools identificationFramework selection and adaptationSurvey conduction
Updated overview on RE tools’ capabilities and potentials
Survey on Requirements Engineering Tools 3 /26
Introduction
Introduction
Benefits from automated support to RERE tools and support to the RE processRE tools and developer’s expectationsWhat are the RE tools’ desirable features?How are they supported by current RE tools?Description of the state-of-the-art on RE tools
Current RE tools identificationFramework selection and adaptationSurvey conduction
Updated overview on RE tools’ capabilities and potentials
Survey on Requirements Engineering Tools 4 /26
Related work
Related work
Surveys on RE
[Liu et al., 2010][Winkler and von Pilgrim, 2010][Carlshamre et al., 2001][Johansson et al., 2001][Benslimane et al., 2007]
Surveys on RE tools
[Zowghi and Coulin, 2005][Alenljung and Persson, 2008][Hall, 2008][Gregoriades and Sutcliffe, 2005][Maiden et al., 2006][Portillo Rodríguez et al., 2010]
Survey on Requirements Engineering Tools 5 /26
Classification framework for RE tools’ capabilities
Regulations and guidelines
ISO/IEC TR 24766:2009 Technical Report (TR) of Type 2Set of capabilities that the RE tools should supportSix major categories
Requirements elicitationRequirements analysisRequirements specificationRequirements V&VRequirements managementOther tool capabilities
Supplements ISO/IEC 14102:2008 International StandardEvaluation of CASE tools
Survey on Requirements Engineering Tools 6 /26
Classification framework for RE tools’ capabilities
RE tools’ capabilities
ISO/IEC TR 24766 tool capabilities
Category AmountRequirements elicitation 37Requirements analysis 36Requirements specification 16Requirements verification and validation 34Requirements management 17Other tool capabilities 17Total 157
Survey on Requirements Engineering Tools 7 /26
Research methodology
Research goals
Goal/Question Metric (GQM) framework
Goal/Question Metric (GQM) framework[Basili and Rombach, 1988]GQM template [Basili et al., 1999]
Goal: To investigate the state-of-the-art on RE tools using aquestionnaire aimed at software vendorsQuestion: Do current RE tools address industry challenges?Metric: Capabilities of RE tools (questionnaire scores)
Survey on Requirements Engineering Tools 8 /26
Research methodology
Instrumentation
Instrumentation I
Databases hosting RE tools lists (July, 2010–August, 2010)
Database AmountIan Alexander 67Alarcos Research Group 7INCOSE 34Ludwig Consulting Services 40Qaguild 7Volere 71@WEBO 41Total sample size after discarding invalid tools 94
Survey on Requirements Engineering Tools 9 /26
Research methodology
Instrumentation
Instrumentation II
DESMET [Kitchenham, 1996] method for evaluating softwareengineering methods and tools
Feature Analysis—Survey
146-items questionnaire6 ISO TR 24766 categories of features plus modelling plustraceability157 ISO TR 24766 features → 126 questions126 technical questions + 20 general, administrative questions
Web-based survey using LimeSurvey
Survey on Requirements Engineering Tools 9 /26
Research methodology
Instrumentation
Instrumentation II
DESMET [Kitchenham, 1996] method for evaluating softwareengineering methods and tools
Feature Analysis—Survey
146-items questionnaire6 ISO TR 24766 categories of features plus modelling plustraceability157 ISO TR 24766 features → 126 questions126 technical questions + 20 general, administrative questions
Web-based survey using LimeSurvey
Survey on Requirements Engineering Tools 9 /26
Research methodology
Instrumentation
Instrumentation II
DESMET [Kitchenham, 1996] method for evaluating softwareengineering methods and tools
Feature Analysis—Survey
146-items questionnaire6 ISO TR 24766 categories of features plus modelling plustraceability157 ISO TR 24766 features → 126 questions126 technical questions + 20 general, administrative questions
Web-based survey using LimeSurvey
Survey on Requirements Engineering Tools 10 /26
Research methodology
Experimental procedure
Experimental procedure
Questionnaire and survey system preparation:September, 2010–November, 2010Tool representatives filled in the survey:December 20, 2010–January 15, 2011Follow-up email and deadline extension to February 7, 2011
Survey on Requirements Engineering Tools 11 /26
Results
Participants
Participants I
38 participants out of 94 candidates invited (40.42%)
Acclaro DFSSAligned ElementsAvenqo PEPBlueprintBright Green ProjectsCaliber RMCameoRequirements+
CASE SpecCognition CockpitCradleG-MARCinteGREATIRQAjUCMNav
Leap SEMacA&D/WinA&DMKS IntegrityPACEPolarionRequirementsPsodaQFDcapture
Survey on Requirements Engineering Tools 12 /26
Results
Participants
Participants II
QPackRaQuestRational DOORSReqManReqtifyRequirementsComposer
RTIMERequisiteProRMTrakRommanaScenario PlusSpiraTeam
TestTrack RMTopTeam AnalystTraceCloudTrackStudioVisibleThread On-premise/On-demand
Survey on Requirements Engineering Tools 13 /26
Results
Experimental results
Administrative information I
Year of first release Year of last release
0
1
2
3
4
5
6
7
8
19
75
19
77
19
79
19
81
19
83
19
85
19
87
19
89
19
91
19
93
19
95
19
97
19
99
20
01
20
03
20
05
20
07
20
09
20
11
No. of tools
0
5
10
15
20
25
30
2008 2009 2010 2011
No. of tools
Survey on Requirements Engineering Tools 14 /26
Results
Experimental results
Administrative information II
Platform required Type of license
Windows
Web-based
Windows/ UNIX/Linux
Windows/Mac OS/
UNIX/Linux
Windows/Mac OS
Windows/Linux
Windows/Mac
OS/Linux
Proprietary and not-
free
Proprietary and free
Open-source and
free
Open-source and
not-free
Survey on Requirements Engineering Tools 15 /26
Results
Experimental results
Administrative information III
Cost per individual license Amount of licenses in use
0
2
4
6
8
10
12
14
16
18
No answer Less than 100
100 to 500 501 to 1000
More than 1000
Cost
0
2
4
6
8
10
12
14
No answer 0 to 100 101 to 1000
1001 to 10000
More than 10000
No. of licenses
Survey on Requirements Engineering Tools 16 /26
Results
Experimental results
Correlation between variables
11 variables—one for each category of features plus global scoreplus cost per individual license plus amount of licenses in useBivariate correlation tests (Pearson’s Correlation Coefficients)Correlation is significant at the (**) 0.01 level/(*) 0.05 level(1-tailed)
Strong direct correlation between each distinct category of featuresCost per individual license–analysis (0.336*), modelling (0.404*),traceability (0.329*), specification (0.545**) and global score (0.358)Number of licenses in use–other tool capabilities (0.513**), cost perindividual license (0.243) and global score (0.183)
Survey on Requirements Engineering Tools 16 /26
Results
Experimental results
Correlation between variables
11 variables—one for each category of features plus global scoreplus cost per individual license plus amount of licenses in useBivariate correlation tests (Pearson’s Correlation Coefficients)Correlation is significant at the (**) 0.01 level/(*) 0.05 level(1-tailed)
Strong direct correlation between each distinct category of featuresCost per individual license–analysis (0.336*), modelling (0.404*),traceability (0.329*), specification (0.545**) and global score (0.358)Number of licenses in use–other tool capabilities (0.513**), cost perindividual license (0.243) and global score (0.183)
Survey on Requirements Engineering Tools 16 /26
Results
Experimental results
Correlation between variables
11 variables—one for each category of features plus global scoreplus cost per individual license plus amount of licenses in useBivariate correlation tests (Pearson’s Correlation Coefficients)Correlation is significant at the (**) 0.01 level/(*) 0.05 level(1-tailed)
Strong direct correlation between each distinct category of featuresCost per individual license–analysis (0.336*), modelling (0.404*),traceability (0.329*), specification (0.545**) and global score (0.358)Number of licenses in use–other tool capabilities (0.513**), cost perindividual license (0.243) and global score (0.183)
Survey on Requirements Engineering Tools 16 /26
Results
Experimental results
Correlation between variables
11 variables—one for each category of features plus global scoreplus cost per individual license plus amount of licenses in useBivariate correlation tests (Pearson’s Correlation Coefficients)Correlation is significant at the (**) 0.01 level/(*) 0.05 level(1-tailed)
Strong direct correlation between each distinct category of featuresCost per individual license–analysis (0.336*), modelling (0.404*),traceability (0.329*), specification (0.545**) and global score (0.358)Number of licenses in use–other tool capabilities (0.513**), cost perindividual license (0.243) and global score (0.183)
Survey on Requirements Engineering Tools 17 /26
Results
Experimental results
Technical information I
Tools’ scores
18
12
1
4
0
16
10
54
0
10
17
45
0
7
12
56
4
17
6
23
1
12
89
32
12
14
6
1 1
11 11
5
3
1
0
2
4
6
8
10
12
14
16
18
20
Very high High Medium Low Very low
Elicitation Analysis Specification Modelling V&V Management Traceability Other tool capabilities
Survey on Requirements Engineering Tools 18 /26
Results
Experimental results
Technical information II
Global score (level of accomplishment of the entire ISO/IEC TR24766)—only calculated for those tools participating in all categories
0
2
4
6
8
10
12
Very high High Medium Low Very low
Global
Survey on Requirements Engineering Tools 19 /26
Results
Discussion
Scenario 1: Requirements elicitation and V&V
Storing and managing templates for elicitation (57%), elicitationchecklists (60%), prioritization forms (57%)Providing OMG ReqIF/RIF compatibility (26%)Generating exception reports on verification/validation plancases↔ verification/validation procedures (57%/60%)Providing standard format for interfacing to verification/validationtools (57%/57%)Implementing some built-in requirements checks (50%)Both: Cockpit, Cradle, QPack, ReqtifyElicitation: MKS Integrity, Polarion RequirementsV&V: Aligned Elements, CASE Spec, G-MARC, IRQA, PACE,ReqMan, TraceCloud
Survey on Requirements Engineering Tools 19 /26
Results
Discussion
Scenario 1: Requirements elicitation and V&V
Storing and managing templates for elicitation (57%), elicitationchecklists (60%), prioritization forms (57%)Providing OMG ReqIF/RIF compatibility (26%)Generating exception reports on verification/validation plancases↔ verification/validation procedures (57%/60%)Providing standard format for interfacing to verification/validationtools (57%/57%)Implementing some built-in requirements checks (50%)Both: Cockpit, Cradle, QPack, ReqtifyElicitation: MKS Integrity, Polarion RequirementsV&V: Aligned Elements, CASE Spec, G-MARC, IRQA, PACE,ReqMan, TraceCloud
Survey on Requirements Engineering Tools 19 /26
Results
Discussion
Scenario 1: Requirements elicitation and V&V
Storing and managing templates for elicitation (57%), elicitationchecklists (60%), prioritization forms (57%)Providing OMG ReqIF/RIF compatibility (26%)Generating exception reports on verification/validation plancases↔ verification/validation procedures (57%/60%)Providing standard format for interfacing to verification/validationtools (57%/57%)Implementing some built-in requirements checks (50%)Both: Cockpit, Cradle, QPack, ReqtifyElicitation: MKS Integrity, Polarion RequirementsV&V: Aligned Elements, CASE Spec, G-MARC, IRQA, PACE,ReqMan, TraceCloud
Survey on Requirements Engineering Tools 20 /26
Results
Discussion
Scenario 2: Requirements modelling and specification
Providing storage and display of BPMN (44%), goal models(39%), SysML artifacts (31%), DFDs (44%)Checking the document through spell checking, grammarchecking, data dictionaries, and acronym tables (60%)Generating the output of the specification in a finished form (65%)Complete loop between RE tool and formatted document (39%)Both: Cockpit, Cradle, PACEModelling: ReqtifySpecification: G-MARC, inteGREAT, MKS Integrity, QPack,DOORS, TraceCloud, VisibleThread
Survey on Requirements Engineering Tools 20 /26
Results
Discussion
Scenario 2: Requirements modelling and specification
Providing storage and display of BPMN (44%), goal models(39%), SysML artifacts (31%), DFDs (44%)Checking the document through spell checking, grammarchecking, data dictionaries, and acronym tables (60%)Generating the output of the specification in a finished form (65%)Complete loop between RE tool and formatted document (39%)Both: Cockpit, Cradle, PACEModelling: ReqtifySpecification: G-MARC, inteGREAT, MKS Integrity, QPack,DOORS, TraceCloud, VisibleThread
Survey on Requirements Engineering Tools 20 /26
Results
Discussion
Scenario 2: Requirements modelling and specification
Providing storage and display of BPMN (44%), goal models(39%), SysML artifacts (31%), DFDs (44%)Checking the document through spell checking, grammarchecking, data dictionaries, and acronym tables (60%)Generating the output of the specification in a finished form (65%)Complete loop between RE tool and formatted document (39%)Both: Cockpit, Cradle, PACEModelling: ReqtifySpecification: G-MARC, inteGREAT, MKS Integrity, QPack,DOORS, TraceCloud, VisibleThread
Survey on Requirements Engineering Tools 21 /26
Results
Discussion
Scenario 3: Requirements traceability
Generating reports that compare current and previous versionswhen a source document is updated (55%)Tracing across the tools’ boundaries (55%)Tracing text to graphics (52%), graphics to graphics (47%),elements within graphics (42%), tables and cells within a table(36%)Generating reports of traceability attributes (60%)Best: CASE Spec, Cockpit, Cradle, G-MARC, inteGREAT, ReqtifyClose to the best: Avenqo PEP, IRQA, Polarion Requirements,QPack, TopTeam Analyst
Survey on Requirements Engineering Tools 21 /26
Results
Discussion
Scenario 3: Requirements traceability
Generating reports that compare current and previous versionswhen a source document is updated (55%)Tracing across the tools’ boundaries (55%)Tracing text to graphics (52%), graphics to graphics (47%),elements within graphics (42%), tables and cells within a table(36%)Generating reports of traceability attributes (60%)Best: CASE Spec, Cockpit, Cradle, G-MARC, inteGREAT, ReqtifyClose to the best: Avenqo PEP, IRQA, Polarion Requirements,QPack, TopTeam Analyst
Survey on Requirements Engineering Tools 22 /26
Results
Discussion
Scenario 4: Other features
Providing Eclipse support (42%)Providing data federation (28%)Providing an open data model (18%)
Survey on Requirements Engineering Tools 23 /26
Results
Threats to validity
Threats to validity
Internal validity (causal relationships)Mortality percentage (7.32%)Commitment of the RE tools representativesTruthfulness of the answers (Hawthorne effect)Triangulation techniqueQuestionnaire planning and design
External validity (generalizations)Participants are representative of the RE tools’ community
Survey on Requirements Engineering Tools 23 /26
Results
Threats to validity
Threats to validity
Internal validity (causal relationships)Mortality percentage (7.32%)Commitment of the RE tools representativesTruthfulness of the answers (Hawthorne effect)Triangulation techniqueQuestionnaire planning and design
External validity (generalizations)Participants are representative of the RE tools’ community
Survey on Requirements Engineering Tools 24 /26
Conclusions and future work
Conclusions
Conclusions
The RE process is well-covered by current RE toolsConnection between the scores accomplished by the RE tools ineach category of featuresMore expensive tools offer better requirements analysis,specification, modelling and traceability supportNo association between the cost per individual license and theglobal scoreMore extended tools are stronger in ISO/IEC TR 24766 other toolcapabilitiesThe amount of licenses in use is not associated with the cost perindividual license nor with the global score
Survey on Requirements Engineering Tools 24 /26
Conclusions and future work
Conclusions
Conclusions
The RE process is well-covered by current RE toolsConnection between the scores accomplished by the RE tools ineach category of featuresMore expensive tools offer better requirements analysis,specification, modelling and traceability supportNo association between the cost per individual license and theglobal scoreMore extended tools are stronger in ISO/IEC TR 24766 other toolcapabilitiesThe amount of licenses in use is not associated with the cost perindividual license nor with the global score
Survey on Requirements Engineering Tools 24 /26
Conclusions and future work
Conclusions
Conclusions
The RE process is well-covered by current RE toolsConnection between the scores accomplished by the RE tools ineach category of featuresMore expensive tools offer better requirements analysis,specification, modelling and traceability supportNo association between the cost per individual license and theglobal scoreMore extended tools are stronger in ISO/IEC TR 24766 other toolcapabilitiesThe amount of licenses in use is not associated with the cost perindividual license nor with the global score
Survey on Requirements Engineering Tools 24 /26
Conclusions and future work
Conclusions
Conclusions
The RE process is well-covered by current RE toolsConnection between the scores accomplished by the RE tools ineach category of featuresMore expensive tools offer better requirements analysis,specification, modelling and traceability supportNo association between the cost per individual license and theglobal scoreMore extended tools are stronger in ISO/IEC TR 24766 other toolcapabilitiesThe amount of licenses in use is not associated with the cost perindividual license nor with the global score
Survey on Requirements Engineering Tools 25 /26
Conclusions and future work
Future work
Future work
Support to GSD settingsConcrete RE capabilities
ElicitationModellingTraceability...
Dissemination of results
Survey on Requirements Engineering Tools 25 /26
Conclusions and future work
Future work
Future work
Support to GSD settingsConcrete RE capabilities
ElicitationModellingTraceability...
Dissemination of results
Survey on Requirements Engineering Tools 25 /26
Conclusions and future work
Future work
Future work
Support to GSD settingsConcrete RE capabilities
ElicitationModellingTraceability...
Dissemination of results
Survey on Requirements Engineering Tools 26 /26
Thank you for your attention
Thank you!Any questions?
Survey on Requirements Engineering Tools 26 /26
Thank you for your attention
Thank you!Any questions?
Survey on Requirements Engineering Tools 27 /26
Additional information
Score calculation
Participation of the tool t in the category c :
participant (t , c) ={
true, NA(t , c) ≥ 0.5 · NQ(c);false, otherwise.
NA(t , c): number of answers of the tool t in the category cNQ(c): number of questions of the category cScore s of the tool t in the category c :
score(t , c) =(∑NQ(c)
q=1 score(t ,q))·4
NQ(c)
score(t ,q) ∈ {0,1}: score of the tool t in the question q
∀s ∈ [0,4], discretisation(s) =
Very low, s ∈ [0,0.5];Low, s ∈ (0.5,1.5];Medium, s ∈ (1.5,2.5];High, s ∈ (2.5,3.5];Very high, s ∈ (3.5,4].
Survey on Requirements Engineering Tools 27 /26
Additional information
Score calculation
Participation of the tool t in the category c :
participant (t , c) ={
true, NA(t , c) ≥ 0.5 · NQ(c);false, otherwise.
NA(t , c): number of answers of the tool t in the category cNQ(c): number of questions of the category cScore s of the tool t in the category c :
score(t , c) =(∑NQ(c)
q=1 score(t ,q))·4
NQ(c)
score(t ,q) ∈ {0,1}: score of the tool t in the question q
∀s ∈ [0,4], discretisation(s) =
Very low, s ∈ [0,0.5];Low, s ∈ (0.5,1.5];Medium, s ∈ (1.5,2.5];High, s ∈ (2.5,3.5];Very high, s ∈ (3.5,4].
Survey on Requirements Engineering Tools 27 /26
Additional information
Score calculation
Participation of the tool t in the category c :
participant (t , c) ={
true, NA(t , c) ≥ 0.5 · NQ(c);false, otherwise.
NA(t , c): number of answers of the tool t in the category cNQ(c): number of questions of the category cScore s of the tool t in the category c :
score(t , c) =(∑NQ(c)
q=1 score(t ,q))·4
NQ(c)
score(t ,q) ∈ {0,1}: score of the tool t in the question q
∀s ∈ [0,4], discretisation(s) =
Very low, s ∈ [0,0.5];Low, s ∈ (0.5,1.5];Medium, s ∈ (1.5,2.5];High, s ∈ (2.5,3.5];Very high, s ∈ (3.5,4].
Survey on Requirements Engineering Tools 28 /26
Bibliography
Bibliography I
Alenljung, B. and Persson, A. (2008).DESCRY: a method for evaluating decision-supportingcapabilities of requirements engineering tools.In Proc. of the 14th Int. Conf. on Requir. Eng.: Foundation forSoftware Quality, pages 52–57. Springer-Verlag.
Basili, V. and Rombach, H. (1988).The TAME project: towards improvement-oriented softwareenvironments.IEEE Trans. Softw. Eng., 14:758–773.
Basili, V. R., Shull, F., and Lanubile, F. (1999).Building knowledge through families of experiments.IEEE Trans. Softw. Eng., 25:456–473.
Survey on Requirements Engineering Tools 29 /26
Bibliography
Bibliography II
Benslimane, Y., Cysneiros, L. M., and Bahli, B. (2007).Assessing critical functional and non-functional requirements forweb-based procurement systems: a comprehensive survey.Requir. Eng., 12:191–198.
Carlshamre, P., Sandahl, K., Lindvall, M., Regnell, B., and Dag,J. N. (2001).An industrial survey of requirements interdependencies insoftware product release planning.In Proc. of the Fifth IEEE Int. Symp. on Requir. Eng., pages84–92, Washington, DC, USA. IEEE Computer Society.
Gregoriades, A. and Sutcliffe, A. (2005).Scenario-based assessment of nonfunctional requirements.IEEE Trans. Softw. Eng., 31:392–409.
Survey on Requirements Engineering Tools 30 /26
Bibliography
Bibliography III
Hall, R. J. (2008).A method and tools for large scale scenarios.Autom. Softw. Eng., 15:113–148.
Johansson, E., Wesslén, A., Bratthall, L., and Höst, M. (2001).The importance of quality requirements in software platformdevelopment - a survey.In Proc. of the 34th Annual Hawaii Int. Conf. on Sys. Sci.,volume 9, Washington, DC, USA. IEEE Computer Society.
Kitchenham, B. (1996).DESMET: a method for evaluating software engineering methodsand tools.Tech. Report TR96-09, Dept. of Computer Science, University ofKeele.
Survey on Requirements Engineering Tools 31 /26
Bibliography
Bibliography IV
Liu, L., Li, T., and Peng, F. (2010).Why requirements engineering fails: a survey report from China.In Proc. of the 2010 18th IEEE Int. Requir. Eng. Conf., pages317–322. IEEE Computer Society.
Maiden, N., Seyff, N., Grunbacher, P., Otojare, O., andMitteregger, K. (2006).Making mobile requirements engineering tools usable and useful.In Proc. of the 14th IEEE Int. Requir. Eng. Conf., pages 26–35.
Portillo Rodríguez, J., Ebert, C., and Vizcaíno, A. (2010).Technologies and tools for distributed teams.IEEE Softw., 27:10–14.Winkler, S. and von Pilgrim, J. (2010).A survey of traceability in requirements engineering andmodel-driven development.Softw. and Sys. Modeling, 9:529–565.