Migrating Sites to DRUPAL

Piyuesh kumar QED42

AGENDA√ Old school migration strategies.√ Migrate Module√ Additional features of Migrate√ Walk through migration from rails 3 to Drupal7√ Q&A

Old School Migration Strategies

√ Writing Migration(php) scripts.√ Accuracy required.√ Time consuming.√ Not good when migrating large content.

√ x_export (node_export, user_export et al):√ Easy to setup but requires exact mapping√ No way to rollback

Migrate Module√ Rollback functionality provides for iterative

development.√ Simple and easy to understand migration

classes.√ Drush support.√ Extremely elaborate UI showing all the statistics.√ Support for importing custom fields data.

How To Migrate√ Identify you module to drupal using

“hook_migrate_api”.

√ Extend migration classes as per the requirement.

Example

Function mymodule_migrate_api {

return array(

‘api’ => 2

);

}

Extending Migration Class

√ Public function __construct()√ Constructor for the migrate class.√ Tells migrate about entity_type(node,taxonomy,users)√ Source of the content(Databases, csv, xml)

√ Public function prepareRow()√ Handles the post-processing of data imported.√ Can also be used to add extra data to entities being

imported.

Example

class DemoMigration extends Migration {

public function __construct() {

/**

* Override the basic Migration class here.

*/

}

}

Example

class DemoMigration extends Migration {

public function prepareRow($current_row) {

/**

* Alter the data for the row being migrated here.

*/

}

}

Metadata required in constructor√ Description : Text to tell what are we migrating.√ Source_fields: Primary keys and any field that is not found in

the initial query.√ Query: Query to fetch the data from legacy database.√ Source: MigrateSourceSql instance to define the source of

migration.√ Destination: MigrateDestination<entity_type> instance to

define the destination.√ Map: Mapping to track the relationship between the source

rows from the source database and their resulting Drupal objects.

√ Create one-on-one mapping between legacy db and drupal objects (Using addFieldMapping).

Query√ Query legacy database to fetch data to be

migrated.√ Legacy db on Local

Use db_select√ Legacy db on remote server

Database::getConnection();

MigrateSourceSql & Destination√ MigrateSourceSql: Takes the query object

and source_fields as an argument and defines the souce of data.

√ MigrateDestionation<entitytype>: Defines the Destination drupal object type.

Example√ $this->source = new

MigrateSourceSQL($query, $sourceFields);

√ $this->destination = new MigrateDestinationNode('posts');

MigrateSqlMap√ Responsible for creating mapping (source_id

+ destination_id).

√ Useful while creating relations between imported drupal objects whose primary key value has changed after migration.

Example##User_legacy.inc$this->map = new MigrateSQLMap($this->machineName, array( 'uid' => array( 'type' => 'int', 'unsigned' => TRUE, 'not null' => TRUE, 'description' => t('User id from old site'), 'alias' => 'u', ) ), MigrateDestinationUser::getKeySchema());

##Content.inc$this->addFieldMapping('uid', 'uid') ->sourceMigration(‘User_legacy')

Mapping fields√ AddFieldMapping : Takes the source field

name and dest field name as arguments.

√ Options:√ DefaultValue√ SourceMigration√ DNM√ Arguments and seperators.

Example√ $this->addFieldMapping('uid', 'user_id')

->sourceMigration('user')

->defaultValue(1);

√ $this->addFieldMapping('revision_uid')

->issueGroup(t('DNM'));

√ $this->addFieldMapping('body', 'body')

->arguments(MigrateTextFieldHandler::arguments(NULL, 'wysiwyg'));

Database Architecture√ Migrate_map_<classname> : Stores the

mapping data.

√ Migrate_message_<classname> : Stores the warnings or errors which come over while migration.

√ Migrate_status: Stores the status for all migration classes.

√ Migrate_log: Logs the migrations ran.

Processing data√ Pre-processing: function prepareRow().

√ Post-processing: function complete()

e.g., D6 uses md5 encryption while d7 uses sha512 for password. Logic for handling this while migrating should go in here.

Examplepublic function complete($entity, stdClass $row) {

$pass = 'U'.$entity->pass;

$uid = $entity->uid;

db_update('users')

->fields(array(

'pass' => $pass,

))

->condition('uid', $uid)

->execute();

}

Additional features of migrate v2√ Classes for varied import sources – Xml, JSON,

CSV, Databases√ Allows to define own field handlers.√ Allows migration of different entities: node,

taxonomy,users.√ Drush integration makes it just awesome.√ Rollback allows iterative migration easier i.e.,

you don't have to migrate all at once, migrate one item, test, fix and repeat.

√ Highwater Mark

Highwater Mark√ Allows to re-import newly added/modified

content in the legacy db.√ How??

√ Field marked as highwater inside migration class is saved by migrate when we import the data first time.

√ Overtime more content gets added/modified to the legacy db (for which this fields value is larger).

√ Next time we run import, migrate alters the source query to fetch only those content which have the value larger than highwater value saved by migrate.

How to choose highwater field?√ It should change everytime the content gets

modified.√ It should have a greater value for a new

content added(compared to all the previous nodes).

√ In case of drupal this can be the “changed” property of nodes.

√ Make sure to order the data from legacy db using this highwater field, so that last row processed has the highest highwater value.

Example√ Use of orderby clause in query.

$query->orderby(p.modified');

√ Define highwater field.$this->highwaterField = array(

'name' => ’modified','alias' => ‘p'

);

Drush Commands

√ Drush ms: lists all migration classes√ Drush mi <classname>: Import content√ Drush mr <classname>: rollback content√ Options:

√ Idlist: allows migration of content with specific ids.√ limit: allows to migrate n no of items.√ Feedback = n seconds/items :status of migration

after n items or seconds.

Modules based on Migrate

√ Ubercart_migrate√ Migrate_d2d√ Wordpress_migrate√ TYPO3_migrate√ phpbb2drupal

Putting it all together

Demo

Sources

√ https://github.com/qed42/rdm-drupal√ https://github.com/piyuesh23/joomla_drupal√ https://github.com/qed42/rdm-rails

Thank You!!

Questions?