Data Migrationin an Agile Open Source World

Craig SmithSuncorp

Welcome…

Image http://www.neopoleon.com/blog/images/excel/7.jpg

According to Wikipedia:

Data migration is the process of transferring data between ... computer systems ... is usually performed programmatically to achieve an automated migration ... required when organizations ... change ... or upgrade to new systems, or when systems merge...

Image: http://i.ehow.com/images/GlobalPhoto/Articles/2065416/dictionary_Full.jpg

“In” End Point

Extract

Transform

Load

“Out” End Point

ExtractTransform

Load(ETL)

Pattern

(Enterprise)Application Integration

Patterns

File transfer

Shared database

Remote Procedure Invocation

Messaging

Image http://www.eaipatterns.com/eaipatterns.html

Data Migration

+ Application Integration

=Similar

ApproachImage http://kansolutions.net/yahoo_site_admin/assets/images/integration.302161703_std.jpg

Image http://blog.theworkinggroup.ca/wp-content/uploads/2009/04/agile-manifesto1.gif

Agile Manifesto

Image http://blogs.conchango.com/Admin/ImageGallery/blogs.conchango.com/Colin.Bird/Scrum%20Overview%20Diagram.png

Scrum Practices

Image Beck, Kent – Extreme Programming Explained: Embrace Change

XP Practices

Source http://www.assembla.com/wiki/show/burro/Development_Principles

Development PracticesFixing a

broken build is always the

highest priority

100% Code

Coverage

Commit Often

If In Doubt, Spike It

Out (On A Branch)

All Prod Code

Paired (Or Reviewed)

No Code Ownership

Test Code Is

Prod Code

Fast (<3 mins) Builds

Changing The Following Things

Is A Team Decision...

Code Is A Liability

Molecular Over

Atomic Tests

Use It Or Lose

It

Nulls Are Evil

Statics Are Evil

Use Delegation

Over Inheritance

Create Small

Classes And Methods

Use Outlandish

Class Names

Break It Down... IterativelyImage http://iqizone.com/images/paper%20pile.jpg

Image http://static.soxfirst.com/soxfirst.com/imgname--derivatives_time_bomb_for_big_banks---50226711--bomb.jpg

Agile Builds Confidence

Continuous (Inte)(Mi)gration

Image http://i.ehow.com/images/GlobalPhoto/Articles/2289906/P1120995RPM_Full.jpg

Throttle Data

Prioritisation & CategorisationImage http://www.mikkeman.nl/blog/wp-content/uploads/2009/10/priority-300x137.jpg http://ralphlosey.files.wordpress.com/2009/10/playing-cards.jpg

Image http://silverspaceship.com/static/shot_1.png

Volatility Categorisation

Analysis = BDUF NDUF SDUFImage http://www.domainlanguage.com/_library/images_people_working/designing_at_whiteboard.jpg

Image http://silverspaceship.com/static/shot_1.png

Implement “Big Bang”

Implement “Master + Deltas”Image http://sapplanning.files.wordpress.com/2008/07/integrate-technology.jpg

Image http://www.spycomponents.com/images/xml_at_work.gif

Data in non-binary format > diff

Master Database DumpsImage http://images.yogee.com.au/cat-dump-truck-609-2.jpg

Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg

Activity Migration

Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg

Reconciliation

Open Source Continuous Integration / Build Tools

Continuous Integration Servers Hudson: https://hudson.dev.java.net

Cruise Control:http://cruisecontrol.sourceforge.net

Build Tools Ant: http://ant.apache.org

Maven: http://maven.apache.org

Gradle: http://www.gradle.org

SCons: http://www.scons.org

Phing: http://phing.info

Rake: http://rake.rubyforge.org/

Open Source Developer ToolsIDE’s (multiple language support) Eclipse: http://www.eclipse.org

NetBeans: http://www.netbeans.org

RadRails: http://www.aptana.com/rails

Database Clients DBVisualizer: http://www.dbvis.com/

SQuirreL: http://squirrel-sql.sourceforge.net

Web Service / SOAP Tools soapUI: http://www.soapui.org

Version Control Systems Subversion: http://subversion.tigris.org

Git: http://git-scm.com

Database Versioning Liquibase: http://www.liquibase.org/

Open Source Testing ToolsTest Specification – xUnit Tools JUnit: http://www.junit.org

PHPUnit: http://www.phpunit.de/

Test Specification Tools – Expressive/BDD FitNesse: http://fitnesse.org

Concordion: http://www.concordion.org

Cucumber: http://cukes.info

easyb: http://www.easyb.org

Test Execution / Record / Log / UI Selenium: http://seleniumhq.org

Watir: http://wtr.rubyforge.org

Performance Test JMeter: http://jakarta.apache.org/jmeter/

Grinder: http://grinder.sourceforge.net/

Open Source ETL/ESB Tools

ETL Donkey:

http://www.assembla.com/wiki/show/burro

Pentaho (Kettle): http://kettle.pentaho.org/

Talend: http://www.talend.com

Clover: http://www.cloveretl.com/

Scriptella: http://scriptella.javaforge.com/

ESB Mule: http://www.mulesoft.org

Apache ServiceMix:http://servicemix.apache.org

Apache Camel: http://camel.apache.org/

Questions ??Craig Smithhttp://www.suncorp.com.auhttp://www.agileacademy.com.aucraig.smith@suncorp.com.au@smithcdau

Suncorp is one of Australia and New Zealand's largest diversified financial services providers, supplying banking,

insurance and wealth management products to around 7 million customers through well-established and recognised brands such as AAMI, Australian Pensioners Insurance Agency, Shannons, Vero, Asteron and Tyndall, as well as Suncorp and GIO. Today,

Suncorp is Australia's fifth largest bank and second largest domestic general insurance group, with over 16,000 staff. Suncorp has representation in 450 offices, branches and

agencies throughout Australia and New Zealand.