open-source formative evaluation process in remote software maintenance
DESCRIPTION
The EU funded FastFix FP7 project aims to provide a monitoring control platform for remote software maintenance. This presentation explains the formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed. This process allows for a continuous integration and enhancement of the platform. As a result, the development process is steadily monitored and evaluated, and early feedback is provided to the designers, developers and architects. The feedback came from the two industrial end user partners: Txt e-solutions and Prodevelop and will help to improve the overall FastFix design methodology and the related tool set.TRANSCRIPT
![Page 1: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/1.jpg)
Work Plan FastFix Open-Source Formative Evaluation Process in Remote Software Maintenance Alessandra Bagnato, TXT e-solutions – Corporate Research Javier Cano and Christophe Joubert, Prodevelop
Industrial Track at RCIS 2012, Sixth International Conference on Research Challenges in Information Science 2012 “Be Fit, Fast and Alert when testing, maintaining or solving bugs of your Future Internet applications” FITTEST & FastFIX & ALERT & Io.Test Joint Workshop
May 18th 2012
![Page 2: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/2.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
2/49
![Page 3: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/3.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
3/49
![Page 4: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/4.jpg)
1.- PROJECT OVERVIEW
European project FASTFIX (ICT- 258109, June 2010 - February 2013) develops a monitoring control platform for remote software maintenance. Within this project, we have been working on a formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed.
4/49
![Page 5: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/5.jpg)
Project Overview
5/49
WP 1 Project Management and Quality Assurance WP 2 FastFix conceptual framework WP 3 Context observation and User modeling WP 4 Event correlation WP 5 Fault replication WP 6 Patch generation and self-healing WP 7 Performance, security and privacy
WP 8 Trials and validation WP 9 Dissemination and exploitation
Formative Evaluation is part of
WP8
![Page 6: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/6.jpg)
Formative Evaluation
6/49
PURPOSE to ensure that the goals of the
under-development software
are being achieved and to improve the
software, if necessary, by means of
identification and subsequent
remediation of problematic aspects.
![Page 7: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/7.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
7/49
![Page 8: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/8.jpg)
2.- FORMATIVE EVALUATION PROCESS
1. Analysis
2. Metrics
3. Tools
4. Testing
5. Monitoring
6. Reporting
8/49
![Page 9: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/9.jpg)
FORMATIVE EVALUATION PROCESS
Analyze the project and
development process.
Select metrics to collect.
Select tools to perform metrics collection.
Define tests to execute. Define which tool will
collect what.
Define who is responsible for
monitoring.
Select the reports to create
from the data.
9/49
![Page 10: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/10.jpg)
Analyze the project
Find the basic natures that define the project. In the case of FastFix :
Strong focus in research
Distributed teams
Development of new
Software Artifacts
Integration into common
platform
10/49
![Page 11: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/11.jpg)
Analyze development process
Analyze the development process to use. Try and find points that are worth monitoring. In the case of FastFix :
Iterative process:
Requirements change as development advances.
Requirements and issues appear at each iteration.
Testing is to start early and happen often.
11/49
![Page 12: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/12.jpg)
Select interesting metrics
Find interesting metrics to be gathered. In the case of FastFix :
tickets opened, solved, closed, overdue, etc.
committers activity, evolution of code base, regular activity.
builds and tests stability.
reports on ranked issues found in source code quality.
12/49
![Page 13: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/13.jpg)
Find ‘invisible’ metrics
There exist metrics that indicate a good development of the project but are not directly visible :
Issues discussed
among partners.
Detection of issues before they promote to a problem.
Responses and
actuations on comments
from others.
Honoring deadlines.
13/49
![Page 14: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/14.jpg)
Select tools that integrate into the development process. In the case of FastFix :
Monitorization target Monitoring tool
Source code quality. Static code analysis.
Components integration to form the
FastFix platform. Continuous integration.
Tests execution. Automated builds and tests
execution.
Platform integration in target
applications.
Integration trials after internal
releases.
Open and resolved issues. Issues statistics reports.
Points to hook the formative evaluation
14/49
![Page 15: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/15.jpg)
Non-invasive tools to gather metrics
Automate the process of metrics gathering as much as possible :
Integration into existing tools.
Periodic jobs in build servers.
Periodic scripts that collect information.
Periodic scripts that generate reports.
Tools invisible until notifications are necessary
Automated tools collect metrics silently
15/49
![Page 16: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/16.jpg)
Start testing early, test often
The best formative evaluation is the one that includes all stakeholders in the development : • Include end users as soon as possible. • Iterative testing with small increments. • Make release cycles short.
Project execution
Testing and Trials
M0. Project starts.
M18. Testing starts.
M33. Project ends. Internal Releases
16/49
![Page 17: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/17.jpg)
Monitor evolution, watch for deviations
Monitor metrics gathered periodically (daily) :
Look for deviations
• I.e: continuous integration failing last week.
Take Corrective
actions
• I.e: review build system logs to correct errors.
Use automated
notifications
• I.e: set up automated notifications to responsibles of latest changes when build fails.
17/49
![Page 18: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/18.jpg)
Create reports
Collect all metrics in a single report
• LOC
• # of tests
• Code quality
Highlight areas to improve
• Build stabilty
• Tests
Highlight good areas
• Code quality
• Dependencies management
18/49
![Page 19: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/19.jpg)
Benefits
The overall benefit of the process is to
correct problems and to include new
requirements before the production
phase of the software project and during
the whole life of the project.
19/49
![Page 20: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/20.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
20/49
![Page 21: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/21.jpg)
Development: sprints from backlog.
Produces an increment of the platform.
3.- AGILE DEVELOPMENT PROCESS (SCRUM)
24 h
Product backlog
Sprint backlog
Sprint
30 days
Product increment
21/49
![Page 22: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/22.jpg)
Development: sprints from backlog.
Produces an increment of the platform.
Testing of produced platform. Gathering
of metrics.
Evaluation: Bug and enhancement issues
stored in backlog. Metrics analyzed.
24 h
Productbacklog
Sprintbacklog
Sprint
30 days
Productincrement
Evaluation definition
FastFixIntegration in target system
Test executionon FastFixed
system
Datacollection
Metrics aquisiton
Feedback
Formativeevaluation
Summativeevaluation
Continuousevaluation
Metricsaquisitionrevision
Changerequests
Newrequirements
Bugs
Development
Integration
Evaluation
24 h
Productbacklog
Sprintbacklog
Sprint
30 days
Productincrement
24 h24 h
Productbacklog
Sprintbacklog
ProductbacklogProductbacklog
SprintbacklogSprint
backlog
Sprint
30 days30 days
Productincrement
Productincrement
Evaluation definition
FastFixIntegration in target system
Test executionon FastFixed
system
Datacollection
Metrics aquisiton
Evaluation definitionEvaluation definition
FastFixIntegration in target system
FastFixIntegration in target system
Test executionon FastFixed
system
Test executionon FastFixed
system
Datacollection
Datacollection
Metrics aquisitonMetrics aquisiton
Feedback
Formativeevaluation
Summativeevaluation
Continuousevaluation
Metricsaquisitionrevision
Changerequests
Newrequirements
Bugs
FeedbackFeedback
FormativeevaluationFormativeevaluation
SummativeevaluationSummativeevaluation
ContinuousevaluationContinuousevaluation
Metricsaquisitionrevision
Metricsaquisitionrevision
ChangerequestsChange
requests
Newrequirements
Newrequirements
BugsBugs
Development
Integration
Evaluation
Agile development process
22/49
![Page 23: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/23.jpg)
Agile development at a glance
A simplified view of the development process followed in FastFix:
Sprint planning
Development
Product increment
Testing
Evaluation
23/49
![Page 24: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/24.jpg)
Specific tools
The formative evaluation hooks into the development process using standard tools:
Sprint planning
Development
Product increment
Testing
Evaluation
Code repository analysis
Code quality analysis
Continuous integration server
Metrics collection
Issues management
Trac Subversion
Sonar
Hudson
Jmeter
24/49
![Page 25: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/25.jpg)
Development: sprints from backlog.
Produces an increment of the platform.
Testing of produced platform. Gathering
of metrics.
Evaluation: Bug and enhancement issues
stored in backlog. Metrics analyzed.
AGILE DEVELOPMENT PROCESS
25/49
![Page 26: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/26.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
26/49
![Page 27: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/27.jpg)
4.- OPEN SOURCE SUPPORT TOOLS
• FOCUS ON OPEN SOURCE
• CRITERIA
• SELECTED TOOLS
27/49
![Page 28: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/28.jpg)
Focus on Open Source
WHY OPEN SOURCE TOOLS ? As a research project with focus on Open Source: • Can deliver industrial level performance. • Free as in beer and speech. • Minimum investment. • Great customization. • Huge integration.
28/49
![Page 29: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/29.jpg)
Criteria
Building and Testing
Communication channels
Quality
Issues
Project Management
Source Code
Alternatives
Consortium selects among all alternatives according to: • Capabilities of each tool • Maturity of each tool • Integration with other tools • Familiarity among partners
29/49
![Page 30: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/30.jpg)
Subversion as source code and Maven as project management
Mature, well known source control management. Integrates with lots of other tools.
Mature, well known project management, specially in Java. Integrates with lots of other tools. Simplifies dependency management and build process.
Source Code
Project Management
http://subversion.tigris.org/
http://maven.apache.org/ 30/49
![Page 31: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/31.jpg)
Hudson as continuous integration
Building and Testing
• Off-the-shelf Integration with Subversion and Maven.
• Notifications for failed builds.
http://hudson-ci.org/ 31/49
![Page 32: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/32.jpg)
Sonar as quality analyzer
• Quality analysis with FindBugs.
• Integration with Hudson.
• Reports generation.
Quality
http://www.sonarsource.org/ 32/49
![Page 33: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/33.jpg)
Trac as issue manager
• Integration with SVN.
• Notifications. • Milestone
planning. • Reports
generation.
Issues
http://trac.edgewall.org/ 33/49
![Page 34: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/34.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
34/49
![Page 35: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/35.jpg)
GOAL
• Usability and performance of FastFixed systems not noticeably affected.
INVESTMENT
• FastFix invests part of the effort in tunning the performance of the platform.
PROCESS
• Each iteration release testing collects metrics to evaluate performance.
5.- PERFORMANCE REQUIREMENTS
35/49
![Page 36: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/36.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
36/49
![Page 37: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/37.jpg)
6.- FASTFIX EVALUATION AT WORK
Samples from the deployed tools: • Hudson
• Sonar
• Trac
37/49
![Page 38: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/38.jpg)
FastFix Formative evaluation at work: HUDSON
• HUDSON continuous integration system
schedules builds every night of the whole platform. Notifications in place for interested parties.
38/49
![Page 39: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/39.jpg)
FastFix Formative evaluation at work: SONAR
• SONAR code analysis system analyses code just after every correct build.
SONAR Components
39/49
![Page 40: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/40.jpg)
FastFix Formative evaluation at work: TRAC
• TRAC issues management system is used as a planning tool, to define milestones and their tasks, to manage responsibles for tasks and internal releases.
40/49
![Page 41: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/41.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
41/49
![Page 42: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/42.jpg)
• Lines Of Code
• Developers
• Continuous integration
• Issues management
7.- METRICS UNTIL 15TH MAY 2012
42/49
![Page 43: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/43.jpg)
Lines Of Code : 879742
Metrics until 15th May 2012 Lines Of Code
43/49
Developers: 18
![Page 44: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/44.jpg)
Continuous integration: • Build stability:
Metrics until 15th May 2012 Continuous integration
44/49
Stable 80%
Failed 20%
Unstable 0%
Common
Stable 85%
Failed 0%
Unstable 15%
Client
Stable 75%
Failed 20%
Unstable 5%
Server
• Test stability:
Correct
Failed
Skipped
![Page 45: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/45.jpg)
Issues management: • 282 tickets managed. • 170 closed tickets. • 112 open tickets.
• 29 tickets in the backlog • 6 tickets from internal testing in industrial partners
Metrics until 15th May 2012 Issues Management
45/49
Closed 60%
Other 28% Backlog
10%
Testing 2%
Active 40%
![Page 46: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/46.jpg)
Metrics until 15th May 2012 Code analysis
• Code quality • 34.4% Sonar Rules Compliance Index
• 2547 violations [ 1 blocker, 147 critical, 1610 major, 732 minor, 57 info]
• 23.4% comments, 8.4% duplications, 100% test success
46/49
![Page 47: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/47.jpg)
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
47/49
![Page 48: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/48.jpg)
8.- NEXT STEPS
Execution of trials in
industrial partners with
live production
applications.
Collection of all metrics data, both qualitative
and quantitative.
Reports that summarize
the evaluation of the project.
48/49
![Page 49: Open-Source Formative Evaluation Process in Remote Software Maintenance](https://reader034.vdocuments.site/reader034/viewer/2022042601/5463b223b4af9f623f8b4623/html5/thumbnails/49.jpg)
Thank you for your attention!
Alessandra Bagnato Corporate Research Division - TXT e-solutions S.p.A [email protected] Francisco Javier Cano and Christophe Joubert Prodevelop [email protected], [email protected] FastFix Project Web Site: https://www.fastfixproject.eu