report - ovum decision matrix on selecting alm

Ovum Decision Matrix: Selecting an Application Lifecycle Management Solution, 201314

Reference Code: IT016001477Publication Date: 29 Jan 2013

Author: Chandranshu Singh, Michael Azoff

SUMMARY

Catalyst

Businesses need clarity on how to best manage their software development environments, and how to exploit the current trends in the industry to their advantage in the context of their organizations. Agile, DevOps, and lean development will remain strong drivers for software development teams as they look to these methodologies to reduce waste and enhance quality in their development environments. Application security has become an increasing cause for concern for most organizations. SaaSbased delivery of application lifecycle management (ALM) solutions has considerably reduced the barrier to entry for development teams and this commoditization of tools to manage largescale commercial software development has created a wave that will only grow stronger with time. This report presents a detailed study of ALM solution vendors along the three dimensions of technology, market impact, and customer satisfaction, to help organizations select an ALM vendor that best suits their requirements.

Ovum view

Throughout the several decades of computer programming, and its subsequent use for developing applications for business, education, entertainment, and so forth, the basic fundamentals of software development have remained the same. Development managers are still trying to manage resources and effort such that software applications are delivered on time and within budget, and meet technical, performance, and business requirements. However, the entire technical and business landscape around this central idea of software development has changed. Development environments have proliferated and become complex, processes have become automated, and new technologies are arriving on the scene in rapid succession. Only a few years ago the enterprise development world was largely divided between Java and .NET. This has all changed as mobile devices have disrupted the world of development platforms, and web and mobile technologies proliferate. Managing development across

Ovum Decision Matrix: Selecting an Application Lifecycle Management Solution, 201314 (IT016001477) 29 Jan 2013

© Ovum. Unauthorized reproduction prohibited

these changing landscapes, at the same time evolving developer work patterns with agile, lean, and DevOps, all make for a complex mix.

In the ALM solutions market, leading players are expanding capabilities in the three hot areas of cloud services, mobile development, and supporting agile and lean practices. Softwareasaservice (SaaS) is becoming better established in the enterprise as confidence in its use grows and the benefits in costs and flexibility are realized. Ovum believes that an SaaS ALM option is now essential for vendors to offer; however, some customers such as government and public sector bodies have constraints on what can be allowed outside the firewall and so having an onpremise option will continue to be in demand.

The leading ALM vendors are increasing solution connectability with open source tools and leading spot solutions that are favored for core development automation, such as continuous testing (Hudson and Jenkins), distributed version control systems (Git), and other software development lifecycle activities. Many vendors in this space are also partnering with ALM hub specialist Tasktop to ease product entry to customers with different ALM usage scenarios within a given organization; typically a customer will have a varied mix of products in place, resulting from the different needs of developers and management, cultural preferences across departments and business units within an organization, and mergers and acquisitions.

Continuous delivery (CD) of software from requirements to production environments is no longer a distant dream. While many organizations are still coming to grips with agile, continuous integration, and continuous testing, early adopters are now moving towards continuous delivery. Ovum expects to see further growth in CD implementations as the benefits in streamlining delivery to production are recognized. CD also brings developer and tester activities back under full operations monitoring, which is good practice from a compliance viewpoint.

Software testing and QA will continue to grow in importance in 2013 – in particular, with organizations that are looking to improve their QA maturity to reduce the cost of testing. Ovum expects that organizations will make a concerted effort across the enterprise to move software quality upstream in the software lifecycle to catch defects early, thereby reduce the cost of testing. Organizations in both public and private sectors were targets of malicious attacks in 2011 and 2012. Ovum believes that businesses as well as public sector enterprises will pay more attention to security testing and incorporate secure development practices.

Key findings

• Organizations are under pressure to cut software development costs. • ALM vendors’ tool capabilities have improved in the face of increasing competition and

decreasing market presence for individual vendors. • Test management and change and configuration management remain specialist areas. • ALM vendors are keen to add delivery management capability to their product portfolio.



• Microsoft and HP have executed their ALM strategies well, outperforming other vendors in the market.

• Agile ALM vendors need to add credible requirements management capabilities. • IBM’s leadership in ALM technology remains unchallenged.

• PTC’s integration of its ALM and product lifecycle management tools is on track, following the MKS acquisition in 2011, and is likely to bear fruit for the vendor in 2013.

MARKET AND SOLUTION ANALYSIS

Ovum Decision Matrix for application lifecycle management

The market for ALM tools remains fragmented along several fault lines. On the demand side, factors such as the organization’s size and the complexity of the IT environment play an important role in determining buying behavior. The bottom end of the market includes many small and mediumsized enterprises (SMEs) and freelance developers, development teams with 10 members or fewer, and consultants. This segment typically has highly skilled developers who can work alone or in small teams, and prefers agile methodologies and open source software. The next stratum is comprised of medium to large organizations that often look at ALM from a costcontainment perspective. ISVs (independent software vendors) are a special case, with presence across all size bands (large, medium, and small); here, developer productivity and tool interoperability remain key issues and, predictably, open source software is used heavily. Large, globally distributed enterprises are at the top of the pyramid; these organizations have existing and ongoing investments in ALM tools and rely heavily on advanced requirementsengineering processes.

The ALM market remains diverse, with large enterprises requiring ALM tools for managing largescale and complex development projects and tending to buy suites for the benefits of tool integration. Customers often opt for bestofbreed tools, including open source, and not just from ALM vendors but from specialist vendors who just cover particular segments of the software development lifecycle (SDLC); and the tools market is broad to meet these various needs. For example, project managers opt for tools such as project portfolio management, project management, and requirements management, while developers opt for change and configuration management and defect tracking as vital tools of their trade, and similarly testers and QA professionals opt for QA solutions. This means the ALM SME market is split into two tiers: the tools for core developers and testers and the tools for their managers. Across this split the support for agile methodologies adds another dimension of options, with agile and lean practices such as kanban now considered an essential part of any ALM system. In addition, the option for SaaS delivery of ALM tools provides additional flexibility for the customer and is increasingly popular, though the market for onpremise is still strong. Finally, the increasing importance of embedded software in products is a great opportunity for ALM vendors to enter into new fields; for example, IBM Rational, PTC, and Polarion have focused strategies for these markets. Figure 1 shows a schematic of the choices organizations make in selecting an ALM solution.



Figure 1: ALM solutions market

Source: Ovum

The Ovum Decision Matrix provides a summary of the market standing of the major ALM vendors based on a quantitative and objective assessment of their technology capabilities, market impact, and customer sentiment (see Figure 2). The Decision Matrix provides vendor selection guidance for enterprises looking to deploy ALM solutions and advises them on whether a vendor should be shortlisted, considered, or explored further. The Ovum Decision Matrix is a threedimensional chart representing vendors’ scores in technology, market impact, and customer sentiment. Technology scores are represented on the xaxis and customer sentiment scores on the yaxis. The size of the bubble represents market impact score relative to the market leader. This edition of the Ovum Decision Matrix includes 10 vendors; every one of these vendors was included in the previous version of Ovum’s Decision Matrix for ALM. This year, ThoughtWorks could not make it to the final set of vendors and has



been excluded from analysis. Also, we have included a section on emerging vendors in the ALM market – vendors that, in our opinion, offer “good enough” functionality at attractive price points.

Figure 2: Ovum Decision Matrix – application lifecycle management, 2013

Source: Ovum

Table 1: Ovum Decision Matrix table – application lifecycle management, 2013

Leaders Challengers Followers

HP Polarion Atlassian

IBM Rational PTC CollabNet

Microsoft Serena Micro Focus

TechExcel

Source: Ovum



Market leaders: HP, IBM, and Microsoft

The Ovum Decision Matrix for software lifecycle management, 2011–12, named HP, IBM, Microsoft, and PTC (which acquired MKS) as market leaders, after a detailed assessment based on the technical strengths of their offerings, their impact on the market, and their customer sentiment ratings. In this edition of the Ovum Decision Matrix, much has changed and much has stayed the same.

First, PTC is no longer a leader in the ALM market despite a high score on the technology dimension. PTC’s low score on market impact and customer sentiment dimensions cost it a place in the leader category. However, Ovum believes that PTC has a compelling ALM solution, and the vendor can regain its standing if it executes well in the market. PTC has been integrating its MKS acquisition, the Integrity solution, with its product lifecycle management tool Windchill over the past 18 months and is on track with the first fruits of integration in requirements, defects/issues, and software component build/release. The engineering personnel and sales forces have also been integrated, and Ovum expects that PTC’s integrated solution will be more than the sum of its parts and help the vendor regain its leadership position in the market.

HP, IBM, and Microsoft remain leaders of the ALM market with considerable distance between them and the rest of the pack. However, there has been considerable internal movement in this group. IBM was the undisputed leader of the ALM market in Ovum’s view and the vendor came out on top in the leader category in several previous editions of the Decision Matrix. In the present edition of Ovum’s flagship vendor benchmarking report, HP and Microsoft have upstaged IBM in terms of customer satisfaction and market impact, reflecting their growing maturity as ALM leaders. While the highlevel parameters of the market haven’t changed, there is a new vitality and freshness about both HP’s and Microsoft’s execution of their market strategy, and Ovum acknowledges these changes.

IBM remains the undisputed leader on the technology dimension, scoring much higher than every other vendor on the list. This is consistent with previous years’ scores for technology. IBM’s ALM offering is and continues to be the most comprehensive and featurerich ALM solution in the market. However, IBM has room for improvement in the wider market beyond its large enterprise and systems engineering sweet spots, and this has led to relatively lower market impact and customer satisfaction scores, particularly in the small company category. IBM has also scored lower than HP on these two dimensions, which indicates that HP, in spite of the turmoil surrounding the company, has set its sights on the IT management software market and has been gaining traction in the market. HP is no longer known just for its leadership in the quality management space but as a complete ALM vendor.

Market challengers: Polarion, PTC, and Serena

The Ovum Decision Matrix chart (Figure 1) can be likened to a threestar system: the ALM market leaders, Microsoft, IBM, and HP, are orbited by three planets, the challengers Polarion, PTC, and Serena. Polarion and Serena Software have maintained their position as middletier ALM vendors from the previous edition of the Decision Matrix. PTC (MKS) was, however, named a market leader in the



previous edition and, as mentioned above, the vendor has lost its position due to a lack of clarity in its market messaging and execution. In Ovum’s view, PTC should be able to regain its position easily once it begins to leverage the combined strength of PTC Windchill and Integrity, the ALM solution. PTC’s focused target market and its position of strength in the embedded software development and product engineering space will hold it in good stead.

Polarion has been growing steadily since its inception, and the vendor has created a strong offering targeted at small and midmarket organizations, particularly in the mechatronics industries. Large enterprise customers, though present, are fewer in number. Although off a small base, Polarion’s growth rate is the highest among all vendors that participated in this report, and coupled with a decent showing in the customer sentiment survey, Polarion has managed to score well in market impact and customer satisfaction dimensions of this assessment. Also, the vendor has the third highest score in technology, behind only IBM and PTC (MKS). Polarion has a strong presence in verticals such as manufacturing, communications and media, and financial services.

Serena has taken a process orchestration approach to marshaling the various tasks and tools in a software development environment. This strategy has paid off and Serena has maintained its position as a challenger (middletier vendor) in the ALM market. Serena’s score in technology is moderate, behind Polarion and PTC. The vendor has also scored lower than Polarion and PTC on market impact. Serena’s revenue growth rate has been flat over the past 12–18 months, leading to a low market impact score. Serena, however, is the category leader in customer sentiment, ahead of both Polarion and PTC.

Market followers: Atlassian, CollabNet, Micro Focus, and TechExcel

Atlassian, CollabNet, and Micro Focus were new entrants to the previous edition of the Ovum Decision Matrix on ALM. Atlassian, CollabNet, and TechExcel have retained their position as followers in this edition of the Decision Matrix, and Micro Focus has joined them, not being able to make the challenger category on account of its poor showing in the customer sentiment survey.

By all accounts, Micro Focus is a leader in the ALM space. The vendor’s poor showing in the Ovum Decision Matrix rankings is due to a poor customer sentiment score as mentioned above, and its poor execution in the market. Micro Focus too suffers from flat revenue growth, like Serena, as well as poor market messaging, like PTC. The market had little clarity on Micro Focus’s plans for the acquired products from Borland and Compuware. The vendor spent over three years integrating the acquired tools together, and this has cost it a leader’s spot. However, Micro Focus has now brought to market an integrated endtoend ALM solution, rebuilding on the Borland product and with a new StarTeam offering as the integration platform, and Ovum expects to see better execution and a coherent gotomarket strategy from the vendor. Micro Focus’s product roadmap for the near to medium term and its accelerated rate of product development has led Ovum to believe that the vendor can deliver better over the next 12–18 months.



Atlassian has taken a partnership approach to its solution by creating an open extensible platform. The vendor relies on thirdparty plugins for its Jira platform for the several missing pieces in its portfolio. This has cost Atlassian on the technology dimension. Other vendors in the follower category – TechExcel and CollabNet – have been executing well in the market and growing their technology portfolios. CollabNet has partnered with UC4 to offer a DevOps solution to be launched in January 2013 and with continuous delivery capability. Ovum expects these vendors to move up the ranks soon as their priorities are set on the latest trends in the ALM market and they have created a significant installed base for themselves. Ovum believes that focus on product development coupled with a smartly executed market strategy will put wind in their sails.

Historical analysis

In this edition of the ALM Decision Matrix, Ovum presents two charts containing historical analysis of the scores on technology and market impact dimensions of the vendors included in this report. The charts have been included as additional material based on availability of data from the previous report. The charts are intended to add a historical perspective to Ovum’s analysis of the ALM tools market.

The most apparent observation from the two charts below, Figures 3 and 4, is that technical capabilities have increased for the entire set, except in the case of TechExcel, while market presence has decreased, except in the cases of HP and Microsoft, whose market presence has increased from the previous edition of this report. The trend indicates that costcontainment pressures in the market have influenced ALM vendors; however, their product capabilities have improved in the face of diminishing returns.



Figure 3: Ovum Decision Matrix – historical analysis, technology (2010–11, 2012–13)

Source: Ovum



Figure 4: Ovum Decision Matrix – historical analysis, market impact (2010–11, 2012–13)

Source: Ovum

Emerging vendors

The ALM market has numerous vendors in the smallercompanysize category that either offer a full endtoend suite or specialize in certain segments of the SDLC. In order to assess the very largest players in the market, we had to exclude smaller vendors that are noteworthy in certain respects. However, they still warrant attention, hence this emerging vendor section (see Table 2 for our vendor selection). In addition to small players, we have a new entrant in ALM in the form of an established enterprise IT operations vendor, ServiceNow. It is quite rare for a vendor in the operations space to move into the development space, so this is another category to watch.

Table 2: Emerging vendors – ALM 2013

Vendor Summary

Hansoft The founders of Hansoft started out in 2002 when they created a projectmanagement tool for video game development out of a need for fast responses and usability, which they found lacking in available tools. This led to the first general release of Hansoft in 2005, which featured a fastresponse user interface, supported by the use of inmemory database technology. Hansoft develops its own technology and has been profitable from day one, without the



need for venture capital. Today, the tool offers application lifecycle management (ALM) capabilities and is oriented to support agile methodologies and processes. Uniquely, it is able to support multiple project types, while others rely on waterfall process Gantt charts or use Scrum or kanban boards. So, for example, by using Hansoft, a large product program with subprojects using different processes can be supported on the same tool, and the project portfolio management features can map both Gantt and kanban representations to a highlevel kanban board.

ServiceNow ServiceNow launched its SaaS offering to provide core IT Infrastructure Library (ITIL) process flow functionality to the IT service management (ITSM) space, and has recently entered the application lifecycle management (ALM) space. Starting out in 2010 with project portfolio management (PPM), ServiceNow followed this by offering support for the software development lifecycle space. ServiceNow is using its core IT operations relationship with its customers as an opportunity to gain entry into other parts of their organizations. ALM is a natural step in ServiceNow’s goal of delivering a "single source of truth" for IT, which will require its offering to span both development and operations.

Tasktop Tasktop started out as the company behind the open source Eclipse Mylyn project for managing application lifecycle management (ALM) tasks. Since then, the company has established itself as an ALM integration hub vendor and ALM solutions market partner. Tasktop's products, Tasktop Dev and Tasktop Sync, are designed to maximize investments that organizations have made and are making in their ALM tooling. With no ALM standard emerging in the industry, Tasktop has become the company that other vendors are happy to work with as their ALM integration partner.

VersionOne VersionOne is dedicated to providing an agile project management tool that scales from small teams to large enterprises across multiple teams and projects. It offers both ondemand and onsite deployment options, with an open source integration platform and free thirdparty connectors. As a result, it is also ideal for large enterprises with homegrown tools that need customized integration. VersionOne is highly active in the agile community. It is the prime mover behind the agile education site www.agilesherpa.org and the AgileChronicles newsletter, and the sponsor of many agile events such as its own AgilePalooza.

Source: Ovum



MARKET LEADERS

Market leaders: technology

Figure 5: Market leaders – technology (solution capabilities)

Source: Ovum

Ovum has split the technology dimension into two sets, namely solution capabilities and ALM components for ease of analysis and representation. The two sets represent strategic and tactical issues faced by software development teams, respectively. The first set (see Figure 5) includes parameters such as ALM platform and architecture support, tool interoperability, change and configuration management and collaboration, project portfolio management, enterprise agile capabilities, and application security. The parameters in this set represent businesses’ highlevel priorities for their entire software development function. Improvement in any one dimension of the first set will cascade down to individual development teams as productivity enhancement and other benefits. The second set (see Figure 6), on the other hand, caters to more operational, tactical requirements of



software development teams. The parameters in this set include project management, requirements definition, requirements management, software assurance, quality management, and delivery management.

Figure 6: Market leaders – technology (ALM components)

Source: Ovum

In the solution capabilities set, not much has changed since Ovum’s previous ALM Decision Matrix. IBM Rational, being the overall leader in the technology dimension, is well represented on all technology parameters. In architecture and platform support, IBM, Microsoft, and PTC are represented; these three vendors have consistently scored well in technology dimensions. Tool interoperability is a key concern for most ALM vendors, and Ovum is pleased to see 7 of the 10 vendors benchmarked in this report represented on this dimension. Change and configuration management and project portfolio management remain specialist areas, as can be seen by the vendors represented on these dimensions. Agile development too has found broad acceptance in the market, and the vendors that invested in this trend have profited from it, making the sector attractive for large enterprise IT vendors also. The enterprise agile dimension is populated by IBM, Microsoft, Polarion, PTC, and Serena Software.



Application security is a big concern for enterprise organizations; however, the vendor community has not responded yet in full measure. Only HP, IBM, Microsoft, and PTC have enterpriseclass capabilities in this regard.

In Ovum’s opinion, ALM is a mature market with constantly shifting boundaries; in the past three years the pace of this change has quickened. ALM vendors are now chasing moving targets such as continuous software delivery. This has been made possible by innovations in technology and development processes. The ALM components market leaders’ spider chart reflects this view.

CollabNet, IBM, Microsoft, Polarion, PTC, Serena, and TechExcel are represented on the project management dimension; the remaining vendors Atlassian, HP, and Micro Focus are only marginally off the top score. Requirements definition and requirements management remain specialist areas, with HP, IBM, Micro Focus, and Polarion taking the lead over the rest of the pack.

The software assurance dimension covers audit and compliance support, along with ALM process integrations such as with IT service desk, application performance management, defect management, and release management. IBM, Polarion, HP, PTC, and Serena are the leaders on this dimension.

Quality management is a mature market segment and HP has preserved its lead in this category. Along with HP, other quality management market leaders include IBM, Micro Focus, and Microsoft. The delivery management segment includes build management, release management, and agile practices such as continuous integration and continuous deployment. In this segment, Micro Focus, PTC, and IBM are the market leaders.



Market leaders: customer satisfaction

Figure 7: Market leaders – customer satisfaction

Source: Ovum

Ovum’s customer satisfaction survey for the ALM Decision Matrix was conducted with a minimum threshold of 15 responses per vendor, and none of the respondents were vendorprovided references (see Figure 7). The sample set voted decisively in favor of Microsoft, recognizing it as the customer satisfaction leader in 7 of the 10 parameters, while coming second in client engagement and third in vertical specialization.

The surveyed parameters can be divided into two sets, customer satisfaction for the product or product set, and customer satisfaction for the vendor organization and its capabilities. Customer satisfaction



related to product/s includes product quality, product integration, customer support, portfolio depth, and product usage, whereas customer satisfaction related to the vendor organization includes service capabilities, vertical specialization, service levels, financial stability, and client engagement. Microsoft was voted the top vendor in four of the five customer satisfaction dimensions related to product/s, namely product quality, product integration, portfolio depth, and product usage. HP was voted the leader in the customer support dimension of the customer satisfaction survey.

In the customer satisfaction dimensions related to the vendor organization, Microsoft was voted the leader in three of the five categories, namely service capabilities, service levels, and financial stability. PTC was voted the leader in vertical specialization, and HP in client engagement. The survey validates the traction seen in the market by vendors such as Microsoft and HP. Microsoft is also the leader in the market impact segment of the Ovum ALM Decision Matrix. These two results indicate that Microsoft has executed its ALM product strategy well and is the favored vendor for application development tools and platforms for existing as well as nextgeneration applications. Microsoft’s rise as the mindshare leader among application development tools and platform vendors has come at IBM’s expense. The enterprise software giant has lost out in the customer satisfaction survey in comparison with its nearest rivals, HP and Microsoft. IBM’s strength in technology has not translated to leading positions in the customer satisfaction and market impact segments of this report.



Market leaders: market impact

Figure 8: Market leaders – market impact

Source: Ovum

The market impact score (see Figure 8) is a weighted sum of scores on six individual dimensions of revenue, revenue growth, geographic spread, size spread, vertical spread, and market recognition. Market recognition is the highest weighted parameter (50%) and is calculated as the percentage of respondents that recognize a given vendor. Other parameters are weighted at 10% each. IBM is the leader on the revenue dimension followed by Microsoft and HP. Polarion is the revenue growth leader followed by HP and TechExcel. In geographical spread, IBM is predictably the market leader, followed by HP and Microsoft. Microsoft leads the pack in size spread of revenue followed by IBM and HP.



HP, IBM, and Microsoft are the largest three vendors in the ALM market and hence they are present on every dimension of market impact except revenue growth. It is understandable that smaller vendors such as Polarion and TechExcel have grown with high doubledigit rates and hence are the leaders on the revenue growth dimension. In vertical spread, Micro Focus is the market leader, closely followed by Atlassian, HP, IBM, and Microsoft. Microsoft is the clear leader on the recognition dimension, followed by HP and IBM.

VENDOR ANALYSIS

Atlassian ALM

Figure 9: Atlassian ALM radar diagrams

Source: Ovum



Atlassian is an emerging ALM vendor with a significant installed base. However, the vendor has the lowest score on the technology scale overall. Ovum’s technology assessment is based on a comprehensive set of 12 parameters and Atlassian scores poorly on several dimensions. Atlassian does not market endtoend ALM solutions, instead relying on partners to create addons and plugins on top of an extensible platform. The vendor has scored poorly in requirements definition, requirements management, project portfolio management, application security, software assurance, and quality management.

However, Atlassian offers an effective solution for managing the development processes and artifacts. In market impact, Atlassian – along with most other vendors – is overshadowed by IBM, HP, and Microsoft. Atlassian, however, has managed to score well on the revenue growth and vertical spread axes. In customer satisfaction, Atlassian scores averagely across all ten dimensions. This validates the increasing mindshare that Atlassian is gathering in the ALM market. The vendor’s strengths in defect and issue management, strong support for a range of development processes, and single shared repository architecture spread across a preintegrated toolset make it a compelling proposition for many organizations.

Recommendation: Follower

Due to its low score in the technology and market impact scales, Ovum has positioned Atlassian in the follower category in this edition of the ALM Decision Matrix. Ovum rated Atlassian in the same category in the previous edition of the Decision Matrix. In Ovum’s opinion, the vendor’s capabilities should not be underestimated simply because it chose not to build capabilities in some areas of software lifecycle management. Atlassian offers teambased collaborative knowledge management, strong defect and issue management, project management, delivery management, and agile capabilities. Its tools are designed to work well in heterogeneous environments and are available on demand. In Ovum’s view, organizations should explore Atlassian’s tool set for its strengths.



CollabNet

Figure 10: CollabNet ALM radar diagrams

Source: Ovum

CollabNet is a key player in the ALM market. The vendor has strong capabilities in several segments of the software lifecycle, such as change and configuration management, enterprise agile, project management, and delivery management. CollabNet offers an SaaSbased platform with an integrated toolset. The vendor loses out in quality management and software assurance, and also scores low in requirements definition and requirements management. CollabNet has an overall score of 71% on the technology scale and is placed eighth after IBM, PTC, Polarion, Microsoft, HP, Serena, and Micro Focus. However, it must be noted that these scores are calculated as a percentage of an endtoend ALM suite, and some vendors do not offer capabilities in all areas of the development lifecycle and thus lose points. CollabNet’s recently announced partnership with UC4 will add DevOps capabilities that will grow its presence in the release management space.



CollabNet has an average customer satisfaction score along with several other vendors. Vendors’ performance in the customer satisfaction survey was quite close to each other in this edition of the report. This is expected in surveys of large samples where values tend to converge around the mean. CollabNet’s performance in the market impact segment has predictably been overshadowed by larger IT vendors. CollabNet has a belowaverage score in vertical spread and an average score in revenue growth and is not present on any other axis on the market impact radar.


CollabNet has been placed in the follower category in this edition of the Decision Matrix. The vendor, however, has a capable ALM solution, which may need to be supplemented with a requirements tool in projects with complex requirementsengineering processes. CollabNet has been investing in product development, and Ovum expects that the vendor will soon jump from the follower to the challenger category. CollabNet’s growing installed base also indicates that the vendor has a strong proposition.



HP

Figure 11: HP ALM radar diagrams

Source: Ovum

HP Application Lifecycle Management (ALM) 11.5 is a comprehensive endtoend solution built out from HP’s leading tools in the QA and testing space. HP’s new ALM platform was launched with version 11 and the halfpoint release offers a number of major enhancements: reduced cycle time and support for iterative and collaborative development; inbuilt social networking; enhanced DevOps support, broadening HP ALM across development and IT operations; support for enterprise mobile development; and ALM offered as SaaS and on premise. The latest HP Application Lifecycle Intelligence layer manages source and build tasks across diverse integrated development environments and source and build control management systems.



HP, in line with a number of other ALM vendors, does not offer a change and configuration management solution, but focuses on integrating with the leading products. HP, again in common with many other ALM vendors, does not make a particular play for the systems space, so it does not offer modeling and architecture support. However, Ovum believes requirements definition is an area that HP should focus on improving. Otherwise, HP makes a clean sweep in the technology dimension of the Ovum Decision Matrix.

Recommendation: Leader

HP has been placed in the leader category in this edition of the ALM Decision Matrix. HP made its first appearance as a market leader in the previous edition of the report, and has maintained its position in this edition. Furthermore, HP has also managed to score higher than IBM in customer satisfaction and market impact dimensions. In Ovum’s opinion, HP’s continued leadership in the quality management segment gives it a position of strength among ALM tool vendors. HP’s ability to leverage this position as a softwarequality leader for making inroads into other areas of the software lifecycle is currently being proven. Ovum recommends that enterprise organizations keep HP on their shortlist of ALM vendors.



IBM Rational

Figure 12: IBM Rational ALM radar diagrams

Source: Ovum

IBM Rational’s solution for collaborative lifecycle management (CLM) is an integrated set of applications that offers complete application lifecycle management (ALM) support. The emphasis on collaboration reflects IBM Rational’s support for agile methodologies which have swept the ALM field. In Ovum’s opinion, IBM Rational is one of a select set of vendors leading the ALM space, with a comprehensive portfolio of tools and depth of features that few others can match. In addition, for large enterprises that use other IBM products such as IBM Tivoli for IT service management, there are additional integration benefits: IBM Rational and Tivoli integration is a major IBM initiative that addresses advances in the DevOps movement.



IBM Rational tools were boosted with the Green Hat acquisition, which allows continuous testing to be performed during development when not all components are yet built or available, through the use of service and application virtualization.

The CLM solution has a breadth that allows IBM Rational to target quite different use scenarios, from embedded and systems engineering to agile processes in enterprise IT. IBM has rearchitected some of its Rational portfolio for its Jazz.Net integration platform; these include Rational Team Concert, Rational Requirements Composer, and Rational Quality Manager, which are aimed at the enterprise agile market. In contrast, the Telelogic heritage, which dominates the systems space, such as DOORS, form a part of the Rational solution for systems and software engineering and have been integrated using the OSLC API. The most recent tool added to this portfolio is Rational Engineering Lifecycle Manager, which allows all artifacts in a project or product to be indexed and traced throughout their lifecycle. Ovum has ranked IBM Rational in the top tier, and for enterprises requiring a solution provider to be around in the next decade, IBM Rational can be reasonably relied on to offer that degree of stability, more so than any rival vendor.


IBM has been placed in the leader bracket in this edition of the ALM Decision Matrix. The vendor has maintained its position as an ALM market leader for several years now. IBM has a comprehensive solution set that caters to organizations and teams of all sizes, across geographies and industry verticals. IBM has scored lower than its previous high scores in market impact and customer satisfaction segments. The market recognition score in particular is lower than expected and can be explained by the distribution in our customer survey sample towards smaller organizations – 57% of our respondents came from companies of 100 employees or fewer (see the Methodology section for further details); IBM Rational’s market sweet spot is the larger enterprise, and this includes small units operating within large enterprises. In Ovum’s opinion, the vendor would do well to tighten its market message to smaller companies, as the lower market recognition score compared with its nearest rivals indicates room for improvement. The overall customer satisfaction score is average, and again, represents an opportunity for IBM to raise its service levels and customer support capabilities. IBM has a large lead on the rest of the pack in the technology segment, and a larger lead in ALM revenues; hence, with some improvements to its messaging and support program, IBM can raise its already excellent position even higher as a leader in the ALM market.



Micro Focus

Figure 13: Micro Focus ALM radar diagrams

Source: Ovum

Ovum has referred to Micro Focus as the sleeping giant of the ALM market. After the acquisition of Compuware’s quality management assets and Borland, Micro Focus spent close to three years integrating those product suites together to make a comprehensive ALM solution that is being marketed under the Borland brand. The vendor has recently completed the integration phase and is now looking to quickly expand its solution capabilities. Micro Focus follows agile development practices internally that enable two to three releases every year for every product in the Borland portfolio.

The vendor has average or aboveaverage scores in almost all technology dimensions. Except for areas such as project portfolio management and application security, Micro Focus offers tool support for all other aspects of software development. Micro Focus has scored well in enterprise agile, quality



management, project management, requirements definition, requirements management, and delivery management segments of the software lifecycle.

Micro Focus’s market impact suffered because the market received little or no communication from the vendor during the time it spent integrating the acquired solutions. Micro Focus has no presence on any market impact dimension except vertical spread, where it is the category leader.

Micro Focus’s customer satisfaction scores are also low compared to other vendors profiled in this report. With belowaverage scores across all 10 dimensions of customer satisfaction, Micro Focus needs to pay special attention to customer support and service levels. Having rearchitected its solution by combining elements of Compuware test tools and Borland, Micro Focus is now in a position to reap the benefits of its labor. However, the vendor needs to execute well in the market to extract returns commensurate to the quality of its product suite.


Ovum has placed Micro Focus in the follower category in this edition of the ALM Decision Matrix. In the previous edition, Micro Focus was in the challenger category, however, with decreasing customer satisfaction scores and low market impact, Micro Focus had to be moved to the follower category. Ovum expects the vendor to execute better over the next 12–18 months to regain its position as a leader in the ALM market.



Microsoft

Figure 14: Microsoft ALM radar diagrams

Source: Ovum

Microsoft’s transformation from a development tools vendor to an enterpriseclass software lifecycle management platform vendor is complete. Microsoft has managed this transition in 6–7 years. The vendor has successfully leveraged the Visual Studio brand and turned it into an enterpriseclass ALM platform. Visual 2012 Ultimate edition, Microsoft’s featurerich development platform, is ideally poised for nextgeneration development for mobile phones, tablets, touch interface desktop applications, nextgeneration web application, and so forth. The tool is equally capable of managing the ongoing development and maintenance of existing missioncritical custom applications. Microsoft has executed well in the market and this is reflected in its customer satisfaction and market impact scores. Despite having the fourth best score in the technology scale, behind IBM, PTC, and Polarion, Microsoft’s leading



scores in market impact and customer satisfaction have catapulted the vendor to one of the strongest positions in the Ovum ALM Decision Matrix.

It remains to be seen how Microsoft will exploit the deep integrations between Visual Studio and Azure. Another aspect that will only be clear with time is the growth in Windows 8 application development. Microsoft has done quite well to retain and grow the large Visual Studio community of developers, testers, project managers, configuration managers, release managers, and operations professionals. This community has often acted as the sounding board for future product development and as an installed base for field testing and early adoption of new Microsoft technologies.


Ovum has placed Microsoft in the leader category in this edition of the ALM Decision Matrix. In the previous edition, Microsoft was in the same category. Microsoft’s superior execution in the market, highquality products that can handle old as well as new technologies, and excellent support network make it a key vendor in the market. Organizations that are predominantly Microsoft shops will do well to stick with the vendor. Java shops should also shortlist Microsoft for its attractive price/performance ratio and nextgeneration development tools and platform.



Polarion

Figure 15: Polarion ALM radar diagrams

Source: Ovum

Polarion offers a complete application lifecycle management (ALM) suite to provide greater visibility into the software development process. It has built its portfolio organically, with some components extending open source ALM point solutions, to create an integrated suite that delivers the benefits of an ALM approach to software development:

• ALM modules are closely integrated, facilitated by a solutionwide shared repository. • ALM is driven by a workflow engine that underpins the whole lifecycle on one platform.

• Agile development is supported, with templates and process support.

Polarion finds that the ALM space is becoming fragmented by vendors who do not provide a truly integrated lifecycle solution, but rather provide segments and label these as ALM. Ovum agrees: The



traceability benefits of ALM are only fully realizable in an integrated solution. Having a single repository ensures there is only one version of the “truth” extant. Polarion has found a huge demand for traceability and forensic reporting in areas of high compliance and regulations such as the US Food and Drug Administration and medical instrument manufacturing.

Polarion has made strides in recent years to expand into the North American market and has grown as a consequence. Its largest market is the systems engineering and mechatronics manufacturing industry using embedded software, and in the EU it has a strong customer base in Germany. Despite scoring exceptionally well in the technology dimension of the Decision Matrix it is still a relatively small player in the global ALM market and it has the potential to grow larger. Given that the product lifecycle management (PLM) market is much larger than the ALM market, Polarion has made it a focus to provide muchneeded ALM expertise to the PLM world.

Recommendation: Challenger

Ovum has placed Polarion in the challenger category in this edition of the ALM Decision Matrix. Polarion was placed in the same bucket in the previous edition of this report. That the vendor has retained its position and repeated its strong showing in technology evaluation is testimony to the capabilities of its product suite. However, the vendor still has got a lot of ground to cover before it can really challenge the market leaders. Polarion focuses on the PLM, manufacturing, and embedded development market, and organizations in these verticals will surely benefit from Polarion’s expertise.



PTC

Figure 16: PTC ALM radar diagrams

Source: Ovum

PTC has the second highest technology score after IBM. The vendor has made a clean sweep of the technology dimensions, scoring highest or above average in all dimensions except quality management, where its score is just below average. PTC is on track in its integration between its ALM solution Integrity and its PLM solution Windchill, and 2012 saw the first fruits of the effort in the new release of Windchill. The behindthescenes integration between the PTC and former MKS R&D teams and similarly for the sales forces will bear fruit in 2013, in Ovum’s opinion. PTC’s gotomarket strategy is also clearly delineated between enterprise IT and engineering. PTC is a big name in the PLM space, and with the MKS acquisition it is well placed to bridge the gap between product engineering and software engineering. The vendor has a capable product, reflected by its strong showing in technology evaluation, and it can grow faster than the market if it executes its vertically aligned strategy well.



PTC scored averagely on all dimensions of the customer satisfaction survey and is hardly represented on the market impact radar. This brings us to the conclusion that the vendor has a competent offering but it will have to execute its strategy well to get ahead in this market.


Ovum has placed PTC in the challenger category in this edition of the ALM Decision Matrix. In the previous edition, PTC (MKS) was a market leader. The change in position may well reflect the need to focus on integration. However, that period is now past and PTC is looking forward to a period of stable growth. With its strength in product engineering, PTC will soon be able to leverage its PLM strengths for ALM deployments and vice versa. Ovum expects that PTC will soon regain its leadership position in the market. Product engineering organizations should shortlist PTC for its strengths in this segment.



Serena Software

Figure 17: Serena ALM radar diagrams

Source: Ovum

Serena Orchestrated ALM is an application lifecycle management solution that spans from demand, develop, to deploy, including the entire application lifecycle. Powered by Serena’s workflow technology, Serena Business Manager, as the underlying integration layer, the solution provides a management “frontoffice” set of services with sets of tools for core development and IT operations. In Ovum’s opinion, Serena has created a highly pragmatic solution set geared towards realworld needs, such as managing the execution of project requests and demands to automating the deployment pipeline and supporting DevOps with release management.

Agile support is now native in this latest release of Serena Orchestrated ALM, where before it was available as a plugin: this provides kanban visualization of user stories as they progress through



development, and allows for lightweight management of requirements solely through agile user stories or for greater governance with requirements manager sitting on top.

Serena has reemerged as a major ALM player from a period of reinvestment followed by a number of years of evolving a comprehensive ALM suite. The technology dimension of the Decision Matrix indicates a solution that fulfills nearly every segment of the lifecycle – the gaps exist in modeling, which is important in the systems space but less so in enterprise IT, and in application security. An area where Serena is staking leadership is in release management and automation, where it has made an important partnership with Nolio, a DevOps player, to offer a comprehensive solution offering continuous delivery.


Serena has been placed in the challenger category in this edition of the ALM Decision Matrix. Serena’s continued strong showing in the technology evaluation has led to its placement in the challenger category. Serena Software placed in the same category in the previous edition of this report. Serena is a traditional ALM platform vendor with endtoend capabilities; the vendor has recently channeled its investments towards process orchestration, DevOps, and release management. Ovum believes that Serena Software is well placed to assume a leadership position in delivery management.



TechExcel

Figure 18: TechExcel ALM radar diagrams

Source: Ovum

TechExcel is placed tenth on the technology scale, just ahead of Atlassian. The vendor does not offer functionality for delivery management and application security. Additionally, it has belowaverage scores in enterprise agile, ALM platform and architecture support, change and configuration management, and requirements management. Having said that, TechExcel doesn’t try to be all things to all people; it has a significant presence in hightech verticals such as media and games, communications, product engineering, aerospace and defense, and manufacturing, and focuses product research and development based on customer demand. TechExcel’s solution DevSuite 9.1, however, is capable of handling complex software development projects. The solution has a significant installed base in the electronic gaming industry and is also suitable for distributed development teams.



From a product perspective, TechExcel DevSuite's focus is on capturing earlystage requirements that are not yet well defined and on capturing development artifacts and making them universally visible throughout the lifecycle of the development process. The core modules are also closely integrated, ensuring that critical tasks can flow seamlessly from planning to tracking, and test management to tracking, or any of these combinations. The focus from a customer perspective is on software development companies, systems integrators, game developers (a strong presence), and corporate IT development teams. The original TechExcel product – DevTrack for issue tracking – has significantly higher market presence and deployment cases than the other DevSuite modules, and provides an entry point for the vendor’s customers to adopt further modules. For TechExcel, its quality management offering is also increasingly a differentiator for winning customer engagements.


With low scores in technology evaluation and market impact, Ovum has positioned TechExcel in the follower category. TechExcel was placed in the same bracket in the previous edition of this report. TechExcel lacks geographic reach and market presence compared to market leaders Microsoft, IBM, and HP, as well as other vendors. However, organizations in the aforementioned verticals should consider TechExcel for its fieldtested capabilities.

OVUM RAINBOW MAP ON ALM 2013

Vendor technology landscape

The Ovum Rainbow Map (ORM) on ALM 2013 is shown in Figure 19. The ORM provides a snapshot view of the technology areas each vendor solution covers. The Rainbow colors go from the blue end, to signify that features are present, to the red end, where they are minimal. Black means the vendor has no offering at all in that category. Features are counted as present when they are available out of the box but can be provided by any product in the vendor’s portfolio. Some vendors make their strength from having every feature available out of one product – this is not reflected in the ORM so we advise checking with the vendor if that is an important feature for you.

The vendors with the most complete feature sets are HP, IBM, Microsoft, Polarion, PTC, and Serena. Support for application security is the single category that is most often missing from a vendor’s ALM suite. This reflects the poor attention that developers generally give to security, often relying on other parties in the organization to deal with security matters. As applications increasingly sit outside the firewall, this misses an important opportunity to protect enterprise assets from malicious activity. The other area that is least addressed in the ALM suites is quality management: there are, of course, bestofbreed players in the testing and QA space and some ALM vendors prefer to integrate with these rather than offer fullfeatured homegrown solutions, and most ALM vendors do offer some features in this category.



One area that has seen a lot of activity recently is delivery management, the result of the influence of DevOps and continuous delivery. Support for enterprise agile development is now very good across all the vendors, reflecting the role ALM plays in managing large and complex projects, which today often use agile and lean processes.



Figure 19: Ovum Rainbow Map on ALM 2013

Source: Ovum



VENDOR SOLUTION SELECTION

Inclusion criteria

The application lifecycle management market has many vendors that offer solutions to customers of all sizes. However, the criteria to be included in this report are based on the ability to offer solutions that span the whole software development lifecycle and offer the benefits of an integrated platform that go beyond a set of point lifecycle solutions. As this vendor solution market is quite large, particularly with the influx of more recent agile processoriented players, Ovum has set guidelines to assess only those vendors that have achieved a measure of market presence.

• The vendor must provide at least three of the following core ALM disciplines: Requirements management, project management, project portfolio management, change and configuration management, QA and test management.

• In addition, to qualify as an ALM solution, there should be support for at least two further disciplines: Requirements definition, architecture/design modeling, defect/task tracking, release management, application performance management, collaboration and knowledge management, project reporting and analytics, and project estimation.

• The solution’s target market should be mid to largesize enterprise customers, typically those with over 1,000 employees.

• The vendor should have ALM license revenues over $30m, or total ALMrelated revenues over $60m, or the vendor should have over 300 individual customer companies.

• The vendor should not be focused solely on ALM in a single vertical industry.

Exclusion criteria

The software development tools market is an enormous one and we do not include any vendor that is solely focused on development tools, as contrasted with managing the SDLC. Some tools cross into the management space; for example, modern integrated development environments (IDEs) are quite sophisticated and allow developers to manage their work environment. However, the IDE is a largely commoditized product, dominated by free open source offerings, and with one exception (Microsoft), ALM vendors do not supply their own IDE but will integrate with the most popular ones.

Ovum has not included vendors who are strong solely in one segment of the SDLC. For example, a number of vendors in this space specialize in agile project management and will integrate and partner for the missing lifecycle segments, but this is insufficient in our opinion for the vendor to be considered an ALM solution provider.



Ovum ratings

• Leader – This category represents the leading solutions that we believe are worthy of a place on most technology selection shortlists. The vendor has established a commanding market position with a product that is widely accepted as bestofbreed.

• Challenger – The vendors in this category have good market positioning and are selling and marketing the product well. The products offer competitive functionality and a good priceperformance proposition, and should be considered as part of the technology selection.

• Follower – Solutions in this category have narrower applicability and may have limitations in terms of the product’s functionality or the vendor’s execution capability. However, they will still be suitable to meet specific requirements, and may be worth exploring as part of the technology selection.

Ovum Interactive Decision Matrix

To access the Interactive Decision Matrix for ALM, an online interactive tool that provides the technology features that Ovum believes are crucial differentiators for leading solutions in this area, please download the Ovum Interactive Decision Matrix tool from the Ovum Knowledge Center.

OVUM DECISION MATRIX METHODOLOGY

Technology analysis

The key report vendors were invited to complete an ALM features matrix (FM), a spreadsheet listing product features Ovum believes are required in an ALM solution. Ovum then applied weights to these entries by individual row and by section. The final ranking of vendors in the Decision Matrix for ALM 2013 technology dimension is based on the scores vendors achieve from this analysis. The criterion for a vendor to answer yes to a feature is that it must be available outofthebox in any product within its range of products. A thirdparty provider, custom integration, or partnership is not sufficient to merit a “yes.” Appendix II reproduces the entries in the FM with the weightings.

Customer satisfaction

Ovum conducts a survey of IT decisionmakers around the world to seek their opinion on enterprise software vendors’ performance across ten parameters. Each vendor is ranked by respondents who are either current or previous customers for a given software category. For past customers, Ovum sets a limit of having used the software solution in the past twelve months. This ensures version accuracy and keeps the responses relevant for further analysis. Ovum does not use any vendorsupplied references for its survey, but instead relies on thirdparty survey platforms and independent panels for its research. This helps keep the survey data free from bias and makes it an important indicator of enduser sentiment for the given enterprise software category. In this instance, Ovum surveyed developers, testers, development managers, and CIOs from a vast array of large, medium, and small enterprise



organizations, across geographies. In all, Ovum received over 15 responses per vendor for 10 vendors across 10 distinct customer satisfaction parameters. The vendors were rated on a scale of 0–10. The first five parameters given below are specific to vendor solutions, and the other five deal with vendor organizations. This gives us an objective way to separate enduser sentiment about the product from end users’ perception of the vendor company.

• Product quality – Respondent’s perception of the quality of the vendor’s product or product suite.

• Product integration – Respondent’s perception of the integration capabilities of the vendor solution with thirdparty solutions as well as with other products in the vendor’s portfolio.

• Portfolio depth – Respondent’s perception of the completeness of a vendor’s portfolio in terms of breadth of coverage and functionality depth.

• Product usage – Respondent’s satisfaction level with the product in daytoday usage scenarios.

• Vertical specialization – Respondent’s opinion of vendor’s capability to meet the requirements of their industry vertical, in terms of vertical specific features and functionality.

• Service capabilities – Respondent’s perception of the quality of professional services (consulting, integration, maintenance, and management) offered by the vendor.

• Service levels – Respondent’s perception of the quality of the vendor’s service level agreements, and its ability to meet them.

• Customer support – Respondent’s perception of the quality of the vendor’s business and technical support offerings, and the vendor’s responsiveness to customer issues and enquiries.

• Client engagement – Respondent’s perception of the vendor’s sales, service, and account teams, and its market channel.

• Financial stability – Respondent’s perception of vendor’s present and future financial health.

Market impact

Ovum analysts use data collected through primary and secondary research to determine a vendor’s global market impact. Market impact is measured across six categories, each of which has a maximum score of 10.

• Revenue – Each vendor’s global ALM revenues are calculated as a percentage of the market leader's. This percentage is then multiplied by a market maturity value and rounded to the nearest integer.

• Revenue growth – Each vendor’s revenue growth estimate for the next 12 months is calculated as a percentage of the growth rate of the fastestgrowing company in the market. The percentage is then multiplied by 10 and rounded to the nearest integer.

• Geographical penetration – Ovum determines each vendor’s revenues in three regions: the Americas; Europe, the Middle East, and Africa (EMEA); and AsiaPacific. These revenues are calculated as a percentage of the market leader's revenues in each region, multiplied by 10, and



then rounded to the nearest integer. The vendor’s overall geographical reach score is the average of these three values.

• Recognition – Survey respondents are asked to select ALM vendors they are aware of. The percentage of the vendor recognition is then divided by the highest percentage of the vendor recognition, multiplied by 10, and rounded up to the nearest integer.

• Vertical penetration – Ovum determines each vendor’s revenues in the following verticals: energy and utilities; financial services; healthcare; life sciences; manufacturing; media and entertainment; professional services; public sector; retail; wholesale and distribution; telecommunications; and travel, transportation, logistics, and hospitality. These revenues are calculated as a percentage of the market leader's revenues in each vertical, multiplied by 10, and then rounded to the nearest integer. The vendor’s overall vertical penetration score is the average of these three values.

• Sizeband coverage – Ovum determines each vendor’s revenues in three company size bands: large enterprises (over 5,000 employees), mediumsized enterprises (1,000–4,999 employees), and small enterprises (fewer than 1,000 employees). These revenues are calculated as a percentage of the revenues of the market leader in each region, multiplied by 10, and then rounded to the nearest integer. The vendor’s overall company sizeband score is the average of these three values.

Customer survey

The customer satisfaction and market recognition data was derived from our primary research customer survey. The respondents totaled 2,820 and represented organization sizes, as shown in Figure 20, and industry verticals, as shown in Figure 21.



Figure 20: Ovum Decision Matrix customer survey: respondents by organization size

Source: Ovum



Figure 21: Ovum Decision Matrix customer survey: respondents by vertical industry

Source: Ovum

APPENDIX I

Summary scores

Table 3 shows all the vendors’ scores. Technology analysis and customer satisfaction are given as a percentage, while market impact scores are normalized to a base 10 – higher values are better.

Table 3: Summary scores

Vendor Technology % Customer satisfaction % Market impact / 10

Atlassian 59.90 64.02 1.60

CollabNet 70.90 66.17 1.13

HP 87.10 69.37 7.92

IBM Rational 98.30 65.06 6.99

Micro Focus 76.70 56.88 1.64

Microsoft 91.30 72.35 10.00



Polarion 92.00 64.31 1.92

PTC 92.70 62.52 1.04

Serena Software 84.30 65.11 0.98

TechExcel 61.70 67.15 1.69

Source: Ovum

APPENDIX II

Features matrix

The features matrix is a spreadsheet listing product features expected in an ALM solution. The entries were given weights by row and by section (%) and formed the technology dimension assessment in the Ovum Decision Matrix on ALM 2013. Each key vendor in the ODM comparison and evaluation was required to fill out a features matrix with entries applicable to any product available in the vendor’s portfolio, and features had to be available outofthebox. This appendix reproduces the entries in the FM with the weightings.

ALM integration framework and architecture support – 12%

Repository support:

• Common repository – 1

• Federated repositories – 1

Workflow engine – 5

Support for collaboration between developers across lifecycle (share feedback, docs, code packages, link comments) – 5

Analytic reporting on ALM project health – 2

Native IDE integration with Eclipse and/or MS Visual Studio – 1

Cloud readiness/support:

• Hosted (SaaS/PaaS) tools – 2

• Development for the cloud – 2

Webbased client access – 3

Agile support:

• Continuous integration – 1 • Continuous testing – 1



• Continuous deployment – 1

• Process guide can be accessed within ALM system at point of application – 1

Product lifecycle management integration:

• PLM solutions supported – 3

• CAD, CAE systems supported – 1

Integration with document management systems (e.g. Microsoft SharePoint) – 1

Process integration with ITIL and integration into ITSM solutions – 1

Mobile development framework support – 1

LDAP integration – 1

User security: read/write access control – 1

Architecture:

• Graphical architecture modeling – 5

• Synchronization between models and code – 5

Tool interoperability – 9%

Internal (within the ALM suite and across vendor’s portfolio)

Does the application PPM tool read the project management tool? – 1

Does the project management tool read the requirements management tool? – 1

Does the project management tool read the change and configuration management tool? – 1

Does the project management tool read the quality management tool and repository? – 1

Does the requirements management tool fully integrate with project management tools? – 1

Twoway read and write between the quality management tool and defect tracking took – 1

Is there a tightly integrated platform?

• With entities and artifacts traced across whole lifecycle? 1

• Can new events and workflow automation be triggered by artifact relationship events? – 1



External tool interoperability (with thirdparty tools)

APIs published – 1

• Royaltyfree – 1

• SOAbased – 1

Is there an SDK? – 1

Change and configuration management – 7%

Collaboration

Can work artifacts be linked to documents? – 1

Can the following be annotated/commented:

• Source code? – 1 • Scripts? – 1 • Change requests? – 1

• Tests? – 1

Collaboration is wikibased with read and write capability – 3

Process/methodology authoring and publishing – 1

Author(s) of artifacts visible – 1

Version control

Provided natively (vendor has tool) – 1

Provided via integration to thirdparty tool – 1

If provided natively:

• Distributed version control system – 1 • Applies beyond just software CCM – 1

• Visual workflow with automation and enforcement – 1

Automated branch inheritance – 1

Explore tool following branch inheritance – 1

Track branches/files that have been renamed for merging purposes – 1



Move and rename branches – 1

Visual branch and merge – 1

Visual file history analysis and differencing – 1

Apply policies at checkin – 1

Audit tracking – 1

File and folder versioning – 1

Tool availability:

• On premise – 1

• Cloud (SaaS/PaaS) – 1

Project portfolio management – 9%

Multiple projects and teams management – 4

Financials management – 1

Planning and scheduling – 3

Automated timesheets (pulls data from project management tool) – 1

Whatif modeling – 2

Resource management – 2

Demand management – 1

Risk management:

• Analysis – 3

• Tracking using risk priority numbers – 1

Reporting:

• Ad hoc – 2

• Prebuilt – 1

Tool availability:



• On premises – 1


Project management – 10%

Agile user story/requirement and task estimation support (points, time, FPA) – 1

Visual and flexible tracking of work effort – 1

Resource allocation and scheduling – 2

Manage backlogs: product, release, project, sprint/iteration – 2

Management of hierarchical stories/tasks structure – 1

Prioritization of stories/tasks – 1

Customizable user stories allowing comments/annotations and flags – 1

Electronic workflow whiteboard:

• Standard Scrum process – 1

• Kanban – 2

Gather metrics from:

• Own pluggedin tools – 1

• Thirdparty pluggedin tools – 1

Agile charts:

• Velocity and burn down/up – 5

Project intelligence (BI for app development – realtime analytics and reporting) – 3

Project reporting:

• Metrics, stats, charts, dashboard – 1

• Report generation – 1

Tool availability:





Requirements definition – 7%

Semantic analysis of text for ambiguity, omissions, and compliance – 2

Visual prototyping – 1

Customizable user interface labeling – 1

Glossary – 1

Convert unstructured input data to structured formats – 1

Output formats:

• Structured – 1

• Semistructured – 1

Generate workflows:

• Logic/application process – 1

• UI renderings/screen – 1

Traceability (for audit trails) supported – 3

Multiple definitions versions supported – 2

Collaboration capabilities – 2

Tool availability:



Requirements management – 8%

Demand and request management – 1

Requirements interdependency mapping – 1

Requirements tracking through lifecycle – 1

Requirements reuse management and support for inheritance rules – 1

Baselining – 1



Change impact analysis – 1

Coverage analysis – 1

Requirementsbased test case generation – 2

Visual requirements representation:

• Modeling – 2

• Synchronized text and models – 2

Reporting – 1

Tool availability:



Software assurance – 6%

Governance and compliance

Outofbox process support (e.g. ITIL, COBIT, SarbanesOxley, HIPAA, Basel) – 1

Auditable processes and compliance reporting – 1

Software assurance integrations: electronic signature support – 1

Requirements/QA integration: Are requirements and test plans maintained in same or federated repository? – 1

• Test cases autogenerated by requirements – 1 • Automated validation of test plan coverage of requirements – 1

• Conduct gap analysis of test plans through requirements tool – 1

Application performance management integration: Use performance testing scripts to do synthetic tests of actual performance – 1

IT service desk integration: autopopulate defect/issue tracking with coderelated trouble ticket incidents/problems – 1

Release management integration with ALM solution – 1

PPM/QA integration: view test results and test coverage by project within the PPM tool – 1



Defect tracking/task management integration:

• Can defect tracking tool autopopulate task lists for specific team members? – 1 • Automatic generation of log entries from tests – 1 • Enduser access for log creation – 1 • Link back to related resources (build, VM, etc.) – 1

• Customizable visual workflow – 1

Quality management – 7%

Track tests by requirement – 1

Test case branching and reuse – 1

Automation in use case test authoring and refactoring – 1

Historicalbased analytics and reporting – 1

Can product features/tasks be represented on a risk chart – 1

Manage combinatorial testing – 1

Scheduling and automated execution of tests – 1

Test data management – 2

Unit test management – 2

Manage test lab environment:

• On premise – 1

• On cloud – 1

Thirdparty QA and requirements management tool integration – 1

Enterprise agile – 10%

APPM/agile planning integration: Are agile project KPIs visible inside APPM tool? – 2

Multiproject, multiteam agile planning:

• Do agile planning tools have visibility to multiple agile projects? – 2

Agile multiproject/multiteam resource balancing/allocation:



• Can agile planning tools rebalance resources across projects/teams? – 1

• Compare burndown rates by team – 1

Multiteam artifact dependency management:

• Does agile planning track relationships between projects? – 1

• Can projects and/or iterations be prioritized? – 1

Multiproject risk management:

• Assess risks for completing specific stories within project iterations/timelines – 2

Support for process impediments – 1

Support for hybrid agile and other processes – 1

Agile/QA management integration:

• Compare defect rates and other KPI across teams/projects – 1

Application security – 5%

Integrate with static testing (codescanning) tools – 1

Integrate with dynamic testing tools – 1

Security issues autopopulate defecttracking system – 2

Security tracking integrated with troubleticketing system – 2

Security issue impact analysis – 1

Risk prioritization of security issues – 1

Reporting – security assessment:

• Is reporting bundled as core functionality 1

Delivery management – 10%

Continuous integration of software in development:

• Private workspace – 1

• Public streams – 1

Continuous deployment of customdeveloped software into production – 2



Build rollback – 2

Dependency tracking – 1

Intelligent build (e.g. only needs to build files that have been changed) – 2

Build validation – 1

Integration with Ant or Maven – 1

Release management:

• Release planning – 1 • Release workflow and control – 1

• Release automation – 1

APPENDIX III

Authors

Chandranshu Singh, Senior Analyst, Software

[email protected]

Michael Azoff, Principal Analyst, Software

[email protected]

Ovum Consulting

We hope that this analysis will help you make informed and imaginative business decisions. If you have further requirements, Ovum’s consulting team may be able to help you. For more information about Ovum’s consulting capabilities, please contact us directly at [email protected].

Disclaimer

All Rights Reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of the publisher, Ovum (an Informa business).

The facts of this report are believed to be correct at the time of publication but cannot be guaranteed. Please note that the findings, conclusions, and recommendations that Ovum delivers will be based on



mailto:[email protected]



information gathered in good faith from both primary and secondary sources, whose accuracy we are not always in a position to guarantee. As such Ovum can accept no liability whatever for actions taken based on any information that may subsequently prove to be incorrect.



report - ovum decision matrix on selecting alm

Documents