Comparison and analysis of five different open source bug tracking systems

SUBMITTED BY:- GUIDED BY:

Name – RAHUL ROY Sudhanshu Prakash Tiwari

Regd. NO-11005675

Rollno –A59

CSE526

TABLE OF CONTENT:

Introduction

Bug Tracking System

Classification Criteria

Introduction to BugZilla

Introduction to Mantis

Introduction to Jira

Classification based on platform

Classification based on support

Classification based on size and usage

Classification based on reporting

Classification based on tracking

Classification based on other features

Conclusion

INTRODUCTION:

As software projects become increasingly large and complex, it becomes more difficult to properly verify code before shipping. Maintenance activities account for over two thirds of the life cycle cost of a software system, summing up to a total of $70 billion per year in the United States. Software maintenance is critical to software dependability, and defect reporting is critical to modern software maintenance.

Large software development projects require a bug tracking system to manage bug reports and developers who work on fixing them. A ubiquitous example of such a system is Bugzilla, an open-source system first introduced in the development of the Mozilla web browser, but now used in numerous other projects.

Bug tracking systems are particularly important in open source software development, where the team members can be dispersed around the world. Thus, the bug tracking system is used not only to keep track of problem reports and feature requests, but also to coordinate work among the developers.

Many bug-tracking systems, such as those used by most open source software projects, allow users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other software project management applications. Having a bug tracking system is extremely valuable in software development, and they are used extensively by companies developing software products.

In proprietary software, the fixing of bug takes place in a phased manner because each phase has separate team responsible for the work being done in that phase. While in the open source project, anyone can pick up the bug and start working on that. Dependency may become major issue in calculating the time for fixing of the bug. If the bug is identified before the implementation phase then it can be very well resolved by development team or owner of the module. But once the software is released and implemented, it becomes very difficult to fix the bug.

It has become a cumbersome job for the moderator to find a suitable developer who can fix a particular bug because the description of the reported bug may not generally provide complete information. Many organizations generally rely on email with or without attachment/feedback based bug reporting system and these bugs will be maintained in spreadsheet or in any document editing software. It again becomes very difficult for the owner or developer to track the bug and its progress of fixing.

Bug Tracking System

The bug reporting/tracking system should provide an interactive web based platform for bug reporting and tracking the progress. The system may involve a generic process or specific schedule of bug reporting process. The generic process of the reporting a bug may generally contain the following information:

Title: Title of the bug.

Description: Detailed description of the bug including what, where, why, how and when the bug occurs. Actual message that appears during the operation may be included with the actual set of input and expected output.

Version: Specify the version of the project.

Component: Specific component need to be specified.

Screenshot/Attachment: Corresponding screenshot can also be uploaded as a .jpg or .gif file by capturing the actual operation/output/message.

Priority: Priority may be assigned for its urgency.

Severity: Specify frequent occurrence and its impact in the system.

Status: Current status of the bug (new, opened, confirmed, closed, etc.). Created by: Name of the person or id already registered with the system who

is reporting the bug.

Assigned to: The bug reporter may also assign bug to specific person, if one knows about a particular person who can solve this problem otherwise assigned by the moderator.

Revision History: If the bug is earlier reported, show the historical changes.

Estimated time: The estimated time may be specified, generally used in case of closed team environment not in the open source environment.

Comments: Any other information that will be helpful in identifying the bug.

Bug Life Cycle

When the bug is first reported, its status must be set to “Unconfirmed” as in the case of bugzilla. After its first review, the status can be updated to “New” or “Assigned” based on the previous bug history databases. Once the assignee has corrected the bug, he or she can update its status as “Resolved” by specifying how it was resolved i.e. “fixed”, “invalid (not a bug)”, “duplicate”, “won't fix”, and the (in) famous “works for me”. A resolved bug, will not be “closed” until someone else will confirm it out or it will be confirmed by the moderator. Once it is confirmed, the bug status becomes “verified” otherwise the status must be set to “Reopen”. It remains in “Verified” state until it is incorporated in the release version of the product and its status will be updated to “Closed”. The closed bug can be reopened with its status as “New” or “Unconfirmed”. There might be many bugs that are currently being fixed and some of bugs that are waiting for resolution set to be “closed”.

Classification Criteria

The bug tracking system varies from general purpose to the specific purpose. The generic system will have many features while the specific purpose systems may have limited features. A suitable tool can be selected based on user’s requirement. The requirement generally varies from the platform, support, size, reporting, tracking and other features of the project. These broad criteria can be further classified or categorized in three or more sub-categories.

Platform refers to license policy, architecture, operating system, web server, back-end database, programming language and clients. Support refers to language, multiple projects, web interface, database, email notification and localization. Size and usage refer to the size of the project, number of downloads, number of releases and number of clients/usage. Reporting refers to form based, email, attachments, web and feedback. Tracking refers to timely update status of the bug, graphs for number of pending bugs, assigned, fixed over a period of time in each category and search facility for new as well as existing bugs to know about their status. The other feature refers to any other feature that is not a part of the above mentioned categories.

FIVE OPEN SOURCE BUG TRACKING SYSTEMS ARE:

BUGZILLAMANTISJIRABUGTRACKER.NET FOSSIL

BugzillaINTRODUCTION:

Bugzilla is a Web-based general-purpose bug tracker tool originally developed and used by the Mozilla project. Released as open source software by Netscape Communications in 1998, Bugzilla has been adopted by a variety of organizations for use as a defect tracker for both free software and proprietary products.

Bugzilla is licensed under the Mozilla Public License. Bugzilla was originally written by Terry Weissman in 1998 for the nascent Mozilla.org project, as an open source application to replace the in-house system then in use at Netscape Communications for tracking defects in the Netscape Communicator suite. Originally written in Tcl, Terry decided to port Bugzilla to Perl before its release as part of Netscape's early open source code drops, with the hopes that more people would be able to contribute to it as Perl seemed to be a more popular language at the time.

Bugzilla 2.0 was the result of that port to Perl, and the first version released to the public via anonymous CVS. In April 2000, Weissman handed off control of the Bugzilla project to Tara. Hernandez. Under Tara's leadership, some of the regular contributors were coerced into taking more responsibility, and Bugzilla development became more community-driven.

In July 2001, facing distraction from her other responsibilities in Netscape, Tara handed off control to Dave Miller, who is still in charge as of June 2007.Bugzilla 3.0 was released on May 10th 2007 and brought refreshed UI, XML-RPCInterface, custom fields and resolutions, mod Perl support, shared saved searches, improved UTF8 support and others.

REQUIREMENTS:

Bugzilla's system requirements include: A compatible database management system; A suitable release of Perl 5; An assortment of Perl modules; A compatible web server; A suitable mail transfer agent or any SMTP server.

Currently supported database systems are MySQL and PostgreSQL. Bugzilla is usually installed on Linux and runs using the Apache HTTP Server, but Microsoft Internet Information Services or any web server that supports CGI can be used.Bugzilla's installation process is command line driven and runs through a series of stages where system requirements and software capabilities are checked.

Anatomy of a Bug:The core of Bugzilla is the screen which displays a particular bug. Bug 1 on Landfill

(http://landfill.bugzilla.org/bugzilla-tip/show_bug.cgi?id=1) is a good example. The labels for most fields are hyperlinks; clicking them will take you to context-sensitive help on that particular field. Fields marked * may not be present on every installation of Bugzilla.

1. Product and Component: Bugs are divided up by Product and Component, with a Product having one or more Components in it.For example, bugzilla.mozilla.org’s "Bugzilla" Product is composed of severalComponents:Administration: Administration of a Bugzilla installation.Bugzilla-General: Anything that doesn’t fit in the other components, or spans multiple Components.Creating/Changing Bugs: Creating, changing, and viewing bugs.Documentation: The Bugzilla documentation, including The Bugzilla Guide.Email: Anything to do with email sent by Bugzilla.Installation: The installation process of Bugzilla.Query/Bug list: Anything to do with searching for bugs and viewing the bug lists.Reporting/Charting: Getting reports from Bugzilla.User Accounts: Anything about managing a user account from the user’s perspective.Saved queries, creating accounts, changing User Interface: General issues having to do with the user interface cosmetics (not functionality) including cosmetic issues, HTML2. Status and Resolution: These define exactly what state the bug is in - from not evenbeing confirmed as a bug, through to being fixed and the fix confirmed by Quality Assurance.3. Assigned To: The person responsible for fixing the bug.4. *QA Contact: The person responsible for quality assurance on this bug.5. *URL: A URL associated with the bug, if any.6. Summary: A one-sentence summary of the problem.7. *Status Whiteboard: (a.k.a. Whiteboard) A free-form text area for adding short notesand tags to a bug.8. *Keywords: The administrator can define keywords which you can use to tag andCategories bugs - e.g. The Mozilla Project has keywords like crash and regression.9. Platform and OS: These indicate the computing environment where the bug wasfound.

10. Version: The "Version" field is usually used for versions of a product which havebeen released, and is set to indicate which versions of a Component have the particular problemthe bug report is about.11. Priority: The bug assignee uses this field to prioritize his or her bugs. It’s a good ideanot to change this on other people’s bugs.12. Severity: This indicates how severe the problem is - from blocker ("applicationunusable") to trivial ("minor cosmetic issue"). You can also use this field to indicate whether abug is an enhancement request.13. *Target: (a.k.a. Target Milestone) A future version by which the bug is to be fixed.14. Reporter: The person who filed the bug.15. CC list: A list of people who get mail when the bug changes.16. *Time Tracking: This form can be used for time tracking. To use this feature, youhave to be blessed group membership specified by the “timetrackinggroup” parameter.Orig. Est.: This field shows the original estimated time.Current Est.: This field shows the current estimated time. This number iscalculated from “Hours Worked” and “Hours Left”.Hours Worked: This field shows the number of hours worked.Hours Left: This field shows the “Current Est.” - “Hours Worked”. This value +“Hours Worked” will become the new Current Est.%Complete: This field shows what percentage of the task is complete.Gain: This field shows the number of hours that the bug is ahead of the “Orig.Est.”.Deadline: This field shows the deadline for this bug.17. Attachments: You can attach files (e.g. testcases or patches) to bugs. If there are any attachments, they are listed in this section. Attachments are normally stored in the Bugzilla database, unless they are marked as Big Files, which are stored directly on disk.18. *Dependencies: If this bug cannot be fixed unless other bugs are fixed (depends on), or this bug stops other bugs being fixed (blocks), their numbers are recorded here.19. *Votes: Whether this bug has any votes.20. Additional Comments: You can add your two cents to the bug discussion here, if you have something worthwhile to say.

MANTIS:INTODUCTION:Mantis Bug Tracker is a free web-based bugtracking system. It is written in the PHPscripting language and works with MySQL, MS SQL, and PostgreSQL databases and awebserver. Mantis Bug Tracker claims support for Microsoft Windows, Mac OS, OS/2, and a variety of Unix operating systems. It is released under the terms of the GNU General Public License.

Mantis Bug Tracker is also distributed with most Linux distributions. The goals of this project are to produce and maintain a lightweight, simple bugtracking system. Additions of complexity/features are modular so that users can be shielded from unwanted clutter. Thus, much of the package has a simple version of a feature along with a more fully developed version.

In the 'core' package, the aim is to have the most important, most used, most time saving portions of a bugtracking system. The product is designed to be easily modifiable, customizable, and upgradeable. Anyone with intermediate PHP and MySQL experience should be able to customize Mantis to suit their needs.

Anatomy of a bug:The bugs are listed in a table and the attributes are listed in the following order:

Priority; Id; Number of bugnotes; Category; Severity; Status; Last updated; Summary.

Each (except for number of bugnotes) can be clicked on to sort by that column. Clicking again will reverse the direction of the sort. The default is to sort by last modification time, where the last modified bug appears at the top.The bug id is a link that leads to a more detailed report about the bug. Depending on what you have set in your Account Preferences you will be sent to the simple or advanced view. The user can also add bugnotes here.

The number in the bugnote count column will be bold if a bugnote has been added in the specified time frame. The addition of a bugnote will make the bugnote link of the bug appear in the unvisited state.

The text in the "Severity" column will be bold if the severity is major, crash, or block and the bug not resolved.

Types of severities:

block - prevents further work/progress from being made crash - crashes the application or OS major - major bug; there is no workaround minor - not trivial, but there is a workaround tweak - needs tweaking text - error in the text trivial - extremely minor issue feature - issue does not describe a problem

JIRAINTRODUCTION:

JIRA is a bug tracking, issue tracking, and project management application developed to by Atlassian Software Systems is an Australian software company specializing in issue tracking and collaboration software. JIRA has been designed with a focus on task achievement, is instantly usable and is flexible to work with.JIRA is available in three editions to accommodate your bug tracking, issue tracking, and project management needs:

JIRA Standard JIRA Professional JIRA Enterprise

The main characteristics of JIRA are: Manage bugs, features, tasks, improvements or any issue A clean and powerful user interface that is easy to understand for both

business and technical users Map your business processes to custom workflows Track attachments, changes, components and versions Full text searching and powerful filtering (customizable, savable, shareable

and subscribe able!) Customizable dashboards and real-time statistics Enterprise previsioning and security Easily extended to and integrated with other systems (including email, RSS, Excel, XML and source control) Highly configurable notification options Runs on almost any hardware, OS and database platform Web service enabled for programmatic control (SOAP, XML-RPC and

RESTinterfaces)

Anatomy of a bug:

JIRA tracks issues, which can be software bugs, feature requests, project tasks, helpdesk tickets, leave request forms, or any other tasks the user want to track.The numbered fields as follows:

Key - a unique identifier for this issue. Type - the type of issue that is reported by the user. Status - the stage the issue is currently at in its lifecycle. Resolution - a record of the issue's resolution. Priority - the importance of the issue in relation to other issues. Assignee - the person to whom the issue is currently assigned. Reporter - the person who entered the issue into the system. Project - the 'parent' project to which the issue belongs. Component(s) (if applicable) — project component(s) to which this issue

relates. Affects Version(s) (if applicable) - project version(s) for which the issue is

(or was) manifesting. Fix Version(s) (if applicable) - project version(s) for which the issue was (or

will be) fixed. Summary - a brief one-line summary of the issue. Environment (if applicable) - the hardware or software environment to

which the issue relates. Description - a detailed description of the issue. Comments - added by people who are working on the issue. Additionally, if the administrator has enabled 'Time-Tracking', the following

fields will appear above the 'Description' field: Original Estimate - the total amount of time required to resolve the issue, as

estimated when the issue was created. (This field cannot be modified once work has been logged on the issue.)

Remaining Estimate - the remaining amount of time required to resolve the issue. This field only appears when work has been logged on the issue.

Time Spent - the sum of all the individual work logs for this issue. This field only appears when work has been logged on the issue.

Classification Based on Platform Characteristics :The comparison criteria based on platform characteristics. In this table, most

of these tools are available in open source environment. Mantis is closed source while its free version is available for download and use. Some of the tools are available as a licensed version for which support can be asked from the owner. Paid version of Jira comes in three different variants as Standard, Professional and

Enterprise based on the size of the project and support. Most of these tools work in web based environment while their early versions were available as client/server based environment.

Trac is developed on wiki based architecture, i.e. anyone can see bugs, fix bugs and edit any part of it. Operating environment for most of the projects are cross-platform, i.e. they can be installed on any machine. BugZilla and GNats are available on Linux while BugTracker.Net is available on windows environment. The clients are browser based, so they are independent of the platform. For web server, most of the tools required apache/tomcat except BugTracker.Net which requires MS-IIS. Most of these tools support MySQL as a backend database except BugTracker.net which requires MS-SQL Server and Fossil requires SQLLite. These tools may vary on the basis of programming language used in coding. Tools are developed in C, Python, and PHP, Perl, Java and C #.Net programming languages.

Tools/Platform BugZilla Jira Mantis BugTrack.Net FossilPlatform Open source/

Free/Proprietary

Proprietary/Free forNon commercial

Free/Paid

OpenSource

Open Source/Free

SystemArchitecture

ClientServer/Web Based

ClientServer/Web Based

Web Based/WAP

Web Based DistributedWebBased

ServerOperatingSystem

Linux Cross-Platform

Windows,Linux, Mac OSOS/2 and others

Windows Linux/Mac OS/Windows

WebServer

Apache,MS-IIS orServer capable to run CGI

ApacheTomcat

Apache,MS-IIS

MS-IIS Apache,MS-IIS

BackendDatabase

MySQL,Oracle andPostgreSQL

Mostly supports allRDBMS

MySQL,Oracle andPostgreSQL

MySQLServer

SQLLite

ProgrammingLanguage

TCL/perl Java PHP ASP.NET C

Client (webBrowser)

Anyone Anyone Anyone Anyone Anyone

Classification Based on Support Characteristics:

The general characteristics for support features are Language, Web Interface, Maintenance support, Email notification and Localization. Most of the tools support multiple languages based on Unicode system. Only Fossil and GNats do not have the support of multiple language features. Again in the same line, most of the tools support web based interface. People are more fascinated to use browser based interface because it is independent of operating system environment. Fossil does not have the complete web based interface.

Maintenance support is also available for free of cost or with a paid service with nominal charges. Whenever, a new bug or update is reported, an email will be automatically sent to all users who are in the registered mailing list with the project. The email will also be sent to all registered users even when there is a small update. Bug localization, i.e. finding the most relevant part of source file hierarchy of software in the bug repository database is another important feature provided by BugZilla, Trac and Mantis while others does not have.

Tools/Support BugZilla Jira Mantis BugTracker.Net FossilLanguageSupport

Yes Yes Yes Yes No

Web Interface Yes Yes Yes Yes NoMaintenanceSupport

Yes Yes Yes Yes Yes

EmailNotification

Yes Yes Yes Yes No

BugLocalization

Yes No Yes No No

Classification Based on Size and Usage Characteristics:

In this category, size of the project, support of multiple projects, number of downloads number of releases and clients per usage are the important parameters for which the statistics can be collected over different time period that may be from the date of first release to the recent date of release.

Tools/ReleaseStatus

Bugzilla Jira Mantis BugTracker.Net Fossil

First Release 1998 2004 2000 2002 2006

Latest Release

2011 2010 2010 2011 2011

Year Wise Number of Releases:

Tools/Year BugZilla Jira Mantis BugTracker.Net

2011 4 1 - -2010 14 2 5 142009 15 1 6 282008 10 1 9 362007 7 5 8 152006 8 3 11 -2005 9 4 10 -

2004 3 1 6 -

Classification Based on Reporting Characteristics :

Tools/Reporting BugZIlla Jira Mantis BugTracker.Net Fossil

Form Based Yes Yes Yes Yes Yes

Email Yes Yes Yes Yes No

Attachments Yes Yes Yes Yes No

Web(Online) Yes Yes Yes Yes Yes

Feedback Yes Yes Yes Yes Yes

A well designed reporting feature will attract users in submitting the bug effectively so the project will not miss any of the failure report instance. For example, in the Eclipse project, the submission of bug reports can be done automatically as well as its usage.

The user will click only once in the checkbox and later one can click on a button to submit the bug to the project that will get updated in the system. Therefore, if the bug reporting system contains all these methods, then it becomes very efficient to know about the bug because user will not hesitate in submitting the bugs if he follows a click and proceed approach. BugZilla, Jira, Mantis and BugTraccker.Net have almost every reporting facility but Trac and Fossil are lagging behind in attachments and email based reporting.

Classification Based on Tracking Characteristics:

The tracking feature will contain subcategories likely timely updates, graphical display and search facility. Most of the systems have these functionalities as shown. This can be further categorized to one more level. Whenever the new updates are received, the registered members will receive an email regarding the updates as well as new release of the system.

The graphical and charting facility is one of the very important features that tell user about the number of pending bugs, number of open bugs, number of closed bugs, time taken by each bugs in the form of interactive charts and estimated time to resolve the bug. To track the bug, the tools have the facility to search for a bug that is already submitted in the system. This will be helpful in identification of duplicate bugs. Current day’s product must have all these facility with dynamic graphical and charting options. The search facility is also one of the important considerations. Generally, the systems have full text search facility on title, description and summary of the bug report.

Tools/Tracking BugZilla Jira Mantis BugTracker.net FossilTimelyupdates

Yes Yes Yes Yes Yes

GraphicalDisplay/Reporting

Yes Yes Yes Yes Yes

SearchFacility

Yes Yes Yes Yes Yes

Classification Based on Other Features:Any other feature may incorporate many more features that are very

important but may not have one special category. These are bug localization, change prediction, browsing of source code repository, version control, and Subversion facility. Some of the systems have the capability of these features but they are not mature.

Overall the above mentioned subcategories can be further sub divided into specific subgroups. It will enable to provide a better picture of the bug reporting and tracking system. It becomes very helpful in choosing a tool that will meet the

user specified criteria and need.

Conclusion:A good automated bug-tracking tool should streamline the process of

reporting, managing and fixing issues. In this paper, authors reviewed some of the bug tracking systems and then present a comparative analysis of different bug tracking system based on different classification criteria. It will help the developers to choose bug tracking tool as per their requirement and constraint.

We have also presented a theoretical framework for developing bug tracking and reliability assessment system (BTRAS) based on different consideration. The proposed BTRAS will assess the reliability of software and give information about bug complexity, bug distribution to developer apart from reporting and tracking.