Migrating Legacy Data

Patrick CrowleyRob Kaufman

(We’re from SD Ruby)

Let’s talk about migrating

Suck => Less Suck

PHP => Rails

Perl => Rails

Java => Rails

Rails => Rails

Migrating = pain

• Develop new site

• Migrate data

• Deploy new site

Migrate data

• Dump database

• Drop some tables

• Rename attributes

• Tweak data by hand

• Import data into new app

• Pray things are okay

Sucks, right?

• Labor intensive

• Error prone

• One way trip

Trucker

Migrate legacy data(with less suck)

• Migrate data from day one

• Improve data over time

• Launch with confidence

rake +legacy classes +migration helper

Let’s get started

sudo gem install trucker

config.gem “trucker”

script/generate truck

$ script/generate truck create app/models/legacy create app/models/legacy/legacy_base.rb exists lib/tasks create lib/tasks/legacy.rake insert added legacy adapter to end of database.yml insert added new load path to environment.rb

Rails::Initializer.run do |config| config.load_paths += %W( #{RAILS_ROOT}/app/models/legacy )end

config/environment.rb

legacy: adapter: mysql encoding: utf8 database: trucker_legacy username: root password:

config/database.yml

class LegacyBase < ActiveRecord::Base self.abstract_class = true establish_connection "legacy" def migrate new_record = self.class.to_s.gsub(/Legacy/,'::').constantize.new(map) new_record[:id] = self.id new_record.save end

app/models/legacy/legacy_base.rb

class LegacyPost < LegacyBase set_table_name "blog_posts"

def map { :name => self.headline.squish, :body => self.body.squish } end

end

app/models/legacy/legacy_post.rb

class LegacyPost < LegacyBase set_table_name "blog_posts"

def map { :name => tweak(self.headline.squish), :body => self.body.squish } end def tweak(name) name.capitalize.gsub(/teh/, "the") end

app/models/legacy/legacy_post.rb

namespace :db do namespace :migrate do

desc 'Migrates posts' task :posts => :environment do Trucker.migrate :posts end

endend

lib/tasks/legacy.rake

Let’s do some migrating.

Don’t forget to import your legacy database!

$ rake db:migrate:posts

Migrating all posts (1/10)Migrating all posts (2/10)Migrating all posts (3/10)Migrating all posts (4/10)Migrating all posts (5/10)Migrating all posts (6/10)Migrating all posts (7/10)Migrating all posts (8/10)Migrating all posts (9/10)Migrating all posts (10/10)

$ rake db:migrate:posts limit=5

Migrating 5 posts (1/10)Migrating 5 posts (2/10)Migrating 5 posts (3/10)Migrating 5 posts (4/10)Migrating 5 posts (5/10)

$ rake db:migrate:posts limit=5 offset=5

Migrating 5 posts after 5 (6/10)Migrating 5 posts after 5 (7/10)Migrating 5 posts after 5 (8/10)Migrating 5 posts after 5 (9/10)Migrating 5 posts after 5 (10/10)

DEMO

Use helper method for custom migrations

namespace :db do namespace :migrate do

desc 'Migrate pain_in_the_ass model' task :pain_in_the_ass => :environment do Trucker.migrate :pain_in_the_ass, :helper => pain_in_the_ass_migration end

endend

def pain_in_the_ass_migration # Custom code goes hereend

What about?!?

Trucker helps you move.

But you still need to pack your stuff up.

Some other things to think about

Encoding issues

• MySQL issues:“ALTER TABLE mytable CONVERT TO CHARACTER SET utf8;

• iconv

Java to the rescue?!?

JDBC Adapters vs ActiveRecord

• There are about 12 adapters in the wild for pure ActiveRecord

• The ActiveRecord adapter for sqlserver runs on Win/Unix, but Unix setup is complicated

• Text adapter does CSV, Tab, and other plain text formats

JDBC Adapters

MySQLPostgreSQLOracleMicrosoft SQL Server

DB2FireBird

DerbyHSQLDBH2SQLite3Informix

ActiveRecord Tested:

MSSQL – An Example

• Download MSSQL driver (Google for it, they move it around a lot)

• Copy sqljdbc4.jar into RAILS_ROOT/lib

• Add require 'lib/sqljdbc4.jar' at the top of your environment.rb or application.rb

legacy: adapter: jdbc username: USERNAME password: PASSWORD driver: com.microsoft.sqlserver.jdbc.SQLServerDriverjdbc: sqlserver://63.134.199.59:1433

config/database.yml

Take it live

• Run in prod when you can

• Run locally when you can't (Heroku, VPNs, etc)

Resources

• http://github.com/mokolabs/trucker

• http://github.com/mokolabs/trucker_sample_app

• http://spkr8.com/t/3749

Special thanksto Dave Thomas

The End