www.etlsolutions.com

Migrating data: How to reduce risk

Data migration: A risk!

30% of data migration

projects fail.Source: Bloor Research, 2011

38% of data migration projects run over time or

budget.Source: Bloor Research, 2011

• The vendor hand-wrote the migration to target system

• Rewritten three times due to:• Structural limitations• Performance (twice)

• Bugs in code delayed the migration• Bugs in code caused incorrect data,

making it difficult to test actual migration logic

A real-life example of the risks involved

We were called in to rescue a project in which tools had not been used:

Business ownership: transparency, control, engagement

Use of toolsets Use of a project management

methodology A team with skills and experience in

data migration Use standards wherever applicable Use of a data migration methodology

Recommended best practice

Identify and mitigate risks Ensure confidence in the project

team Appropriate resourcing Keep the project on track:

Within budget On time

Reduce the overall cost and effort needed

Recommended team leader objectives

Overview of our recommended data migration process

ScopingLegacy

decommissioning

Business engagement

Assessments

Core migration

• Migration methodology: an in-depth process for each stage of the data migration, bringing in specialists at key times and working to a structured plan and documents

• Project management methodology: We use PRINCE2 to ensure that the migration is well managed to a common standard

• ISO standards: We use 15288,12207,and 8000 to ensure the methodologies have a core standard structure

• Migration software: The core migration steps should be carried out using a well established tool

• Migration specialists: Specialists should be available for the duration of the project

Embed best practice within the process

• Helps to identify gaps in the existing skill base

• Shows where training may be required

• Reduces reliance on external experts

• Grows the confidence of the project team

• Provides individuals with greater clarity about their roles

• Our proprietary Data Migration Competency Framework is designed specifically to support data migration teams

Data migration process: Assessments

Staff competency assessment

• A strategic review of the proposed project, examining:

• Plans• Processes• Workflows• Data targets• Systems

• Identifies and mitigates risks and issues before they occur

Project assessment

• A detailed, tactical examination of the proposed project, including:

• Stakeholders and deliverables expected

• Budgets• Deadlines• Communication plans• Available experts: business

domain, systems, migration experts

• Helps with project planning to mitigate risk

• Provides business leaders with a clear view of the project plan

Data migration process: Scoping

Project scoping

• A detailed examination of the project’s technical structure:

• Available models• Available software• Data volume and quality

• Identifies potential technical issues before they occur

Technical scoping

Data migration methodology

Core migration

Landscape analysis

Data discovery

Data modelling

Data assurance

Data review

Data cleansing

Requirements analysis

Project scoping

Configuration

Migration design

Testing design

Migration development

Testing development

Execution

Review

Legacy decommissioning

Iterative agile development is used throughout

• Landscape analysis encompasses the systematic process of identifying all source and target systems that may be involved in the data migration

• Gain an overview of the source and target systems

• Key tasks:• Understand how each system works• Understand how the data within each system

is structured• Model the systems• Model links and interactions between systems• Model data structures

Landscape analysis in more detail

• Data assurance puts measures in place to ensure that all information used within the data migration is handled accurately

• Data quality is a key element, along with data cleansing where required

• Planning is required for the retirement of data, for deletion or for storage due to industry regulations

• Key tasks:• Data review• Data cleansing

Data assurance questions:• What data migration and

profiling tools are available?• Are there key areas of

weakness in data?• Are rules for the data quality

(attribute and relational) within the source already documented?

• Which governing rules have to be applied to the data?

• Will all data be migrated?

Data assurance in more detail

• Profiling is carried out to identify areas of the data that may not be of sufficient quality to meet business requirements

• Data quality definitions define the quality that must be attained for elements, attributes and relationships within the source system; these definitions will be used during the profiling to verify if the data adheres to the rules defined

• The system retirement plan defines which data will be moved from the old system to the new and what is no longer required

The key data assurance tasks

Data review

• Define the cleansing rules which will be carried out manually and those which will be automated:

• The manual cleansing will be typically be carried out prior to migration

• The automated cleansing will be carried out as a first step of the migration or indeed may also be able to be carried out prior to migration

• Data verification is focused on the checking of data being available, accessible, in the correct format and complete

• Data impact analysis is carried out to evaluate the effect on other elements and systems

Data cleansing

What happens when data assurance is omitted?

• Multinational 3-way multi-company venture• US vendor migrating data• UK project management

• No migration management• Lack of resource• No verification or quality rules• No profiling carried out• Ad-hoc testing

• Security delays of 6 months• Testing time increased by 3 months• 1 month delay for invalid characters• Additional dry-run required due to issues encountered

• Use a structured approach with methodologies to guide the process

• Establish clear structures and boundaries: a methodology will help with this

• Ensure tool-based execution: we use our own commercially available software, Transformation Manager

• Carry out tool-based testing• Create an organisation-focused go live: scoping the project at

an early stage can assist in disseminating the information to stakeholders around timings and resource required to achieve the goals

Core migration in more detail

A core migration design

Our offerings for data management

Transformation Manager data

movement software

Support, training and mentoring services

Data migration packaged services

Everything under one roof Greater control and

transparency Identify and test against errors

iteratively Greater understanding of the

transformation requirement Automatically document Re-use and change

management Uses domain specific

terminology in the mapping

Why Transformation Manager?

For the user:

Why Transformation Manager?

Reduces cost and effort Reduces risk in the project Delivers higher quality and

reduces error Increases control and

transparency in the development

Single product Reduces time to market

For the business:

www.etlsolutions.com

Raising data management

standards

www.etlsolutions.com

Contact us for more information:

Karl Glenn, Business Development Directorkg@etlsolutions.com+44 (0) 1912 894040

Read more on our website:

www.etlsolutions.com

Images from Free Digital Photos freedigitalphotos.net