Whitepaper on -

<Cost Effective Data Migration > HP QC 9.2 to HP ALM 11.52 Migration

Capgemini India Private Limited

June 2015

Prepared by:<Jayanta Nath>Email: <Jayanta.Nath@capgemini.com>

HP QC 9.2 to HP ALM 11.52 Migration

1

Table of Contents

1. Abstract..................................................................................

2. ALM Migration.........................................................................2.1 Introduction................................................................................................2.2 Generic Process Map..................................................................................

3. Challenges..............................................................................

4. Managed approach of ALM Migration........................................4.1 Scope and Timeline....................................................................................4.2 Primary Schedule.......................................................................................4.3 Setting up proper Communication Channel................................................4.4 Implementation of Automation...................................................................

4.4.1 Automation of Communication......................................................4.4.2 Automation ALM Migration.............................................................

5. Governance & Communication Process......................................5.1 8

6. Benefit Summary.....................................................................6.1 Procurement (Licence) Cost Benefit...........................................................6.2 Employer (Skill) Cost Benefit......................................................................

7. Conclusion...............................................................................

8. References..............................................................................8.1 Online Materials.........................................................................................

9. Annexure...............................................................................109.1 Abbreviations...........................................................................................9.2 Foot Notes................................................................................................

10. Author’s Biography................................................................11

HP QC 9.2 to HP ALM 11.52 Migration

2

1. Abstract

In a competitive scenario like today, while technology is getting updated, renewed and reinvented on every moment, it is utmost important to keep up with its pace in a timely and regular interval, in order to maintain sustainability of your business.Similar needs make upgrade and update, absolutely unavoidable, for the tools and machineries used by each and every industries/practices all across the globe.Here, while we talk about testing services, HP Quality Center is an extremely important tool, where testers create ‘Test Plan’, maintain ‘Test Log’, capture ‘Defects’ and ‘Releases’.QC has been deemed as out of life by HP, thus it was extremely high time to move on. In the mean time, HP has come up with an updated solution, named ‘Application Lifecycle Management (ALM). Although the most current version of ALM is 12.xx, however 11.52 is the most stable version around the park. Thus the client (The Leading Bank over Australia and New Zealand) has finally decided to update their testing services with the same.Version upgrade of tool consists the appended phases –

1. Update of Software Package2. Redesigning the Database3. Testing of the new Database4. Scheduling Movement of Data5. Migration of data to New Database, based on the best fitted test approach6. Release the tool to the End Used for further verification7. Finalizing

In this white paper, the basic focus will be on the last 4 points (3 to 6) from the above, of which point 4 will be paraphrased as ALM Migration, where all data movement has been done and point 3, 5 and 6 will be referred to as Effective Communication Management, where other administrative tasks have been performed to ensure BAU, from here onwards.This white paper will aid the testing fraternity to plan and complete E2E Migration with very limited resources (Like Ms Excel 2007 or Higher using VBA), minimum time, low cost and quality as high as using some high-end tool like Informatica etc and manage BAU even with an in flight DB upgrade activity with very minimal down time (Depending upon size of data may be an hour, max)In this document, it has been attempted to keep the content as simple and lively as possible in terms of all definition, examples, workflows and process maps.

2. ALM Migration

2.1 IntroductionMigration is a planned approach, to move data from one database to other may be manually or automatically, with minimum manual intervention.Here we have used automated approach using VBA with Ms Excel 2010 as base.Migration consists of 3 basic phases Extract, Transform and Load (ETL).Extract: In this phase the data gets extracted from the old database as is and stored in an intermediate storage. During ALM Migration Excel sheets were used as a base to store the data after extraction. To eliminate the possibility of data overflow Excel 2010 has been used, which can handle data of 220 Rows X 214 Columns (1048576 X 16384)

HP QC 9.2 to HP ALM 11.52 Migration

3

Transform: This is the phase, where the old data gets drafted in new form, which will be accepted by the architecture of new database. It may involve some or all of the appended tasks –

Applying business rules (Derivations like calculating new measures and dimensions e.g. – No non numeric data can be inserted in RTS)

Cleaning (e.g. - mapping NULL to “N/A” or “ST” to “03 – System Test” and "Regression" to "Standard Regression" etc.)

Filtering (e.g. – selecting only certain columns to load depending on the availability of the field in ALM)

splitting a column into multiple columns and vice versa joining together data from multiple sources (There could be a few fields which

cannot be migrated, may be due to field unavailability but in QC it contained vital data, they usually gets merged to the description field as addition e.g. - lookup, merge)

transposing rows and columns applying any kind of simple or complex data validation (e.g. – if that field has no

values in used data of QC, exclude from processing)Any or all of these tasks are executed in 2 stages –Map: This is a temporary phase where old data gets matched and a sample draft gets created as what should be what. Mapping also has two parts –

Field Mapping: Mapping QC Fields vs ALM Fields. Fields will be migrated if the fields are matching exactly or if the usage of the field matches with a field or description of a field is similar to a field

EG: Exact Field Map: <QC Field> BG_BUG_ID = <ALM Field> BG_BUG_IDDescription Match: <QC Field> BG_USER_11 = <ALM Field> TS_USER_TEMPLETE_15. Here the label for this field is like - <QC Label> Defect Creep From & <ALM Label> Earliest Phase Detectable but the description is similar Data Match: <QC Field> TS_USER_06 = <ALM Field> TS_USER_TEMPLETE_06. But the data captured in both is related to Test Phase

List/Values Mapping: Mapping of the QC List Item/Value vs ALM List Item/Value for a same field.

EG: Value:ST = 03 System TestSys Test = 03 System TestRegression = Standard Regression

Convert: According to the mapped items, conversion is done, to change the items as the ALM database architecture desirable entries.Load: This is the final stage to migration where the processed (after ET) data gets inserted to the new database (ALM).

HP QC 9.2 to HP ALM 11.52 Migration

4

2.2 Generic Process Map

HP QC 9.2 to HP ALM 11.52 Migration

5

3. Challenges

As a matter of fact, “smooth roads doesn’t make good driver”. There could hardly be anything which can be accomplished without challenges. Thus here are a few challenges that had been identified post completion of the 1 month long Pilot Testing of the entire activity –Sl No Challenges Impact

1. Mammoth manual effort Time consuming, erroneous thus impacts both quality & quantity

2. Automated tools are costlyAll the automated tools available in the market space are very high cost

3. Skilled resourcesResources required for using automated tools also costs equally high as the tools itself

4. Mapping of fields and lists

Different teams could be using customised tables/lists/values, and coming to new database there could be a few which cannot be mapped. Thus customisation makes mapping difficult

5. Validation of complex business logics Impacts transformation6. Network Speed Impacts TAT, may hamper BAU

4. Managed approach of ALM Migration

4.1 Scope and TimelineScope: This being an organization wide activity there were 750 + (+/- 10% Tolerance) units of instances that was planned to be migrated.Timeline: Appended are the agreed phases and timelines –Pilot: 1 MonthLive Migration: 5 MonthsAdditional Support (Change in Scope): 2 MonthsWarranty Support: 40 working days in TnM model spread over 3 monthsResources Employed: 5 Fulltime (Migration) + 1 Management (Planning, Scheduling, Communication & meetings) = 6 + 1 Senior Management (2 X 0.5 Working days)

4.2 Primary ScheduleBasis on the primary forecast and agreed SOW; primary target was set to end to end migration of 7 units per day.

4.3 Setting up proper Communication ChannelCompliance of the plans depends on the communication channel, information channel and follow ups. Appended steps are used for error proofing on this front –

HP QC 9.2 to HP ALM 11.52 Migration

6

1. Multiple, organization wide communication, mails had been circulated from different levels of the organization (Like Technology Head, Operations Head, TMO Head, TPM of ALM migration Team) inside out till the grass root level to ensure, the initiative and actions are properly spoken and heard

2. As per primary schedule intimation mails sent beforehand (at least 7 days prior to the schedule) to all the concerned points, from the different, to be migrated units, asking for their green signal and other required inputs, which has been planned for migration during next 30 days

3. Follow up for inputs in regular interval, if pending4. Communication of initialization, before commencement of the migration and

communication of completion, post completion of migration. Both with all the need to know information, suggestions, best practices and must know “How To” on the usage of new tool (ALM)

5. Designed standard content for all communication and information channels and content review process

4.4 Implementation of AutomationAutomation, being the only solution against waste of manual efforts, was evident to be implemented in this project to the maximum possible extent. Here are two fronts where automation was planned – (1) Mass, standard mailing as communication and information channel/s & (2) Automation1 of ETL based migration process.

4.4.1 Automation of CommunicationAutomation of communication consists 3 parts –

(A) Designing Standard Content (Manual Input): A standard content was designed for each and every steps like -

a. Communication of the initiativeb. Intimation of schedulec. Request for requirement input & nomination of SPoC as counter partd. Reschedule if requested by the SPoCe. Follow up for input if not shared by the SPoCf. Intimation of Initialization of activity with TATg. Intimation of delay if anyh. Intimation of completion with standard usage guidelines and must

know information(B) Defining the Target Audience (Automated): A macro was designed to run

a SQL Query to extract the entire user base (Select * From User Where UserGroup like Admin) with admin access from individual modules/units of QC. BCP Plan to be implemented in case of no response from the admins.

(C) BCP Target Audience (Automated): 2 Macro was designed to extract BCP targets –

a. Last 5 users, logged in to the module/unit (Access Based)b. Last 5 Users, who had executed test cases in QC (Usage Based)

(D) Compilation of Content and Audience & Send Mails during planned interval and follow ups (Automated): A excel template was designed where –

a. A form with sections to capture contents in and as per requirementb. Same form with section to capture the Target Audience list and cc listc. A Sheet to capture the Target Modules/Units with target list and cc list,

where mails to be sentd. A VB Script was created and set in the Job Scheduler with defined

intervals

4.4.2 Automation ALM MigrationAs per ETL process automation needs to be implemented for each extract, transform and load phase –

1 There had been a many popular tools in the market for the same purpose like Informatica, Ab InITIO, Oracle Data Integration & SQL Server etc. But all of these are high cost licence software compared to the solution technology was used.

HP QC 9.2 to HP ALM 11.52 Migration

7

(A) Extract: Extraction of data needs to be done for each and every table of the database which is in scope for Automation. In fact for mapping also, field and base data needs to be extracted as well.

a. Mapping Data: Macro was designed to Log into the module and extract field data from desired tables, values from desired fields and values from desired lists

b. Migration Data: Designed macro to log into individual module and extract data from various tables out of QC database. Appended are the tables2 from which data was extracted during ALM Migration

i. Userii. Bugiii. Test Planiv. Test Labv. Test Cyclevi. Releasevii. Requirementviii. Coverage (Linkage between Test Plan and Requirement)

c. Users, Details & Access Permission/s: Macro to use user details with permissions from the user table of QC

(B) Transform: Prepared macros for each transformations steps except mappinga. Mapping (Manual): Mapping (QC Vs ALM) is done in various stages as

per requirement and standard guidelines and finalized post verification by the SPoC –

i. Mapping of Fields: QC Field names of each and every table/s gets extracted trough a macro and matched with the corresponding one from ALM

ii. Mapping of Used Values: Unique values that has been used in QC, gets extracted through a macro and matched with the corresponding one from ALM

iii. Mapping Lists: List values also gets extracted through a macro and and matched with the corresponding list from ALM

iv. Mapping of Final Data: According to the previous 3 points of the mapping final mapping of the final data is prepared and vetted by the SPoC

v. Mapping of User Groups: Similar user groups were created in ALM, but with different identifier and name thus mapping of what vs what was required

b. Apply Transformation (Automated): Based on the Mapping, that has been finalized, final transformation is applied from another macro.

(C) Load (Automated): Now a new macro will interact with ALM and upload the data into similar table/s from which data had been extracted from QC. Again tables were -

i. From (QC) User to (ALM) Userii. From (QC) Bug to (ALM) Bugiii. From (QC) Test Plan to (ALM) Test Planiv. From (QC) Test Lab to (ALM) Test Labv. From (QC) Test Cycle to (ALM) Test Cyclevi. From (QC) Release to (ALM) Releasevii. From (QC) Requirement to (ALM) Requirementviii. Coverage (Linkage between Test Plan and Requirement)

5. Governance & Communication Process

Governance is the key of any successful project. In accordance to the complexity of the project, objective & result oriented governance model was designed. Appended are the primary attributes of the model –Sl No Activity Objective Output

2 Although he the utility that contents all the macros, classes and functions had procedures to extract data from all the table mentioned here but in this (QC to ALM Migration), scope was limited by the client as Test Set, Requirement Coverage & Defects only.

HP QC 9.2 to HP ALM 11.52 Migration

8

1. Daily Stand up Call

Planning, Troubleshooting & Review

Entire team remains focused on the same target/goals and all the updates gets disseminated in a before start of the day.In case there is any issues or road blocks faced or foreseen those also gets highlighted, addressed or POA planned, whichever is best applicable

2. Mid day review Review on progressA forum to check the update of first half, if it is on track or not. If not precautionary actions gets initiated, to avoid any issues that can be a showstopper

3. Daily Status Report

Day progress report

Helps maintain visibility of the work/efforts that has been accomplished/invested during the day.This also includes High level assistance request, if any issues as such occurred

4. Monthly Steering committee

Strategic Planning, Update Dissemination & Status Review

A monthly meeting/review with the Heads of the concerned departments, whichever applicable like TMO, Technology, Operations etc.

6. Benefit Summary

6.1 Procurement (Licence) Cost BenefitHere are the lowest indicative costs of most popular packages, which do not include AMC cost warranty support –

Where as if compared to the methodology documented here, tool used is Microsoft Excel. Eventually every system does have it and comes along with the Ms Office Package at a very low cost as $399.993 per single system licence. An enterprise licence comes at even lower per system cost

6.2 Employer (Skill) Cost BenefitAgain, apart from licence cost skill cost is another area where it will cut the direct cost. Once the tool (Prepared in VBA) is in place any person with knowledge of basic excel can carry the entire activity forward. And the evident part is, a person with basic excel experience can be hired at much lower cost compared to Informatica or Oracale Data Integration or SSIS.

3(3)http://www.microsoftstore.com/store/msstore/en_US/pd/ThemeID.27509700/SQL-Server-2008-R2-Enterprise-Edition-1-Processor-License/productID.221612500

(4)http://www.itbusinessedge.com/cm/blogs/lawson/dataflux-ups-its-game-with-new-integrated-solution/?cs=39620

(5)https://shop.oracle.com/pls/ostore/f?p=dstore:product:0::NO:RP,6:P6_LPI:4509382199341805719938

(6)http://www.microsoftstore.com/store/msstore/en_US/pd/ThemeID.27509700/SQL-Server-2008-R2-Enterprise-Edition-1-Processor-License/productID.221612500(7) Data Source: https://products.office.com/en-us/professional

HP QC 9.2 to HP ALM 11.52 Migration

9

Overall Cost Saving:

7. ConclusionMigration of data is necessary whenever there is an upgrade of application, Database or Data Warehouse, change in architecture etc and the timeline when you need to upgrade very much volatile on the basic of availability of new technology and advances. Thus the requirement of migration cannot be foreseen exactly can only be approximated. In such scenario, it might not be cost efficient enough to continue with the high cost tools like Informatica, ab InITIO, Oracle Data Integration for a long period of time although Ms SQL Server is comparatively low cost compared to the previous ones yet all of these has AMC associated with it, which will make continuity costly.Using this excel based methodology, in all circumstances will ensure lower direct cost in terms of logistics and employment, no maintenance cost at all over longer period of time and the output will remain as good as other premium tools when quality or quantity is concerned.This solution has been designed to automate and address the communication and governance issues as well. It can work seamlessly even with very minimal administration. What it needs is just a timely review of contents.

HP QC 9.2 to HP ALM 11.52 Migration

10

8. References

8.1 Online Materials1. http://www.wikihow.com/Write-White-Papers 2. https://en.wikipedia.org/wiki/HP_Quality_Center 3. https://en.wikipedia.org/wiki/HP_Application_Lifecycle_Management 4. http://www8.hp.com/in/en/software-solutions/quality-center-quality-

management/5. http://www8.hp.com/in/en/software-solutions/application-lifecycle-

management.html6. http://www8.hp.com/in/en/software-solutions/alm-software-development-

testing/7. http://datawarehouse4u.info/ETL-process.html 8. http://www.itcsoftware.com/services_dbmigration.htm 9. http://www.oracle.com/us/corporate/pricing/technology-price-list-070617.pdf 10. http://searchdatamanagement.techtarget.com/review/Informatica-

PowerCenter-Real-Time-Edition-PowerCenter-RTE11. https://products.office.com/en-us/professional 12. https://products.office.com/en-us/microsoft-office-for-home-and-school-faq 13. https://www.microsoft.com/en-us/licensing/product-licensing/

office.aspx#tab=114. https://en.wikipedia.org/wiki/SQL_Server_Integration_Services 15. http://www.microsoftstore.com/store/msstore/en_US/pd/ThemeID.27509700/

SQL-Server-2008-R2-Enterprise-Edition-1-Processor-License/productID.221612500

16. http://www.microsoftstore.com/store/msstore/en_US/pd/ThemeID.27509700/ SQL-Server-2008-R2-Enterprise-Edition-1-Processor-License/productID.221612500

17. https://shop.oracle.com/pls/ostore/f? p=dstore:product:0::NO:RP,6:P6_LPI:4509382199341805719938

18. http://www.itbusinessedge.com/cm/blogs/lawson/dataflux-ups-its-game-with- new-integrated-solution/?cs=39620

HP QC 9.2 to HP ALM 11.52 Migration

11

9. Annexure

9.1 AbbreviationsSl No Abbr Full Form Definition and Usage

1. HP Hewlett-Packard One of the world’s product based largest IT firms deals with Software’s, Hardware’s and complex IT solutions

2. QC Quality Center

Quality management software offered from the HP Software Division of Hewlett-Packard with many capabilities acquired from Mercury Interactive Corporation. HP Quality Center offers software quality assurance, including requirements management, test management and business process testing for IT and application environments. HP Quality Center is a component of the HP Application Lifecycle Management software solution set.Here it refers mostly to version 9.02 in some cases it can be version 10 as well; otherwise, it has been duly mentioned.

3. ALM Application Lifecycle Management

In addition to what QC has already offering, ALM comes with a set of software products designed for accelerating the delivery of secure, reliable modern applications. It is a combination of a common platform, several key applications and a dashboard targeted at managing the core lifecycle of applications. Application lifecycle management focuses on the core lifecycle from design through readiness for delivery to operations. It includes application development solutions and application testing solutions. The solution provides Information Technology departments with a centralized application management platform for managing and automating within and across application teams and throughout the complete process of developing an application, all within a single workflow.

4. ETL Extract, Transform & Load

The end to end process to migrate data from one database to another. It starts with Extraction of data from old data base, Transformation of data, where the old data gets adapted as per architecture of the new database and finally Load, where the final, drafted data gets inserted to a new database for further use.

5. RTS Raw Test Steps This is the Test Step count of any test cases

6. TMO Test Management Office

Concerned team that takes care of entire testing plans and activity of the fraternity

7. MS MicrosoftWorld’s largest both system and application software developer. For instance all the Windows operating systems are good example of system software and Office tools like Excel, Word, PowerPoint, Access, Visio, Developer tools like Visual Studio, SQL Server etc

8. AMC Annual Maintenance Contact

An agreement where service model is annual billing based, vendor will agree to a fixed cost for an year of service

9. SSIS SQL Server Integration Service

Platform for data integration and workflow applications. It features a fast and flexible data warehousing tool used for data extraction, transformation, and loading (ETL). The tool may also be used to automate maintenance of SQL Server databases and updates to multidimensional cube data.

HP QC 9.2 to HP ALM 11.52 Migration

12

9.2 Foot Notes1 – There had been a many popular tools in the market for the same purpose like Informatica, Ab InITIO, Oracle Data Integration & SQL Server etc. But all of these are high cost licence software compared to the solution technology was used.2 – Although he the utility that contents all the macros, classes and functions had procedures to extract data from all the table mentioned here but in this (QC to ALM Migration) scope was limited by the client as Test Set, Requirement Coverage & Defects only.3 – Data Source: http://www.microsoftstore.com/store/msstore/en_US/pd/ThemeID.27509700/SQL-Server-2008-R2-Enterprise-Edition-1-Processor-License/productID.2216125004 – Data Source: http://www.itbusinessedge.com/cm/blogs/lawson/dataflux-ups-its-game-with-new-integrated-solution/?cs=396205 – Data Source: https://shop.oracle.com/pls/ostore/f?p=dstore:product:0::NO:RP,6:P6_LPI:45093821993418057199386 – Data Source: http://www.microsoftstore.com/store/msstore/en_US/pd/ThemeID.27509700/SQL-Server-2008-R2-Enterprise-Edition-1-Processor-License/productID.2216125007 – Data Source: https://products.office.com/en-us/professional

10. Author’s Biography

Jayanta Nath is a Green belt certified consultant, working out of Prestige Shantiniketan for an FSGBU account, currently TSO team.Prior to TSO, he has been associated with ALM Migration & Automation of Financial reports and activities.His core competency lies with solution of Automation, Analytics and Strategic planning.Reach him – jayanta.nath@capgemini.com