Jasondb – a cloud based, scalable NoSQL database for social and mobile applications.

Rob LintonCTO & Founderwww.jasondb.com

Phone:      [AUS] + 61 0418 510 155Email:      robl@jasondb.com

Migrating AWS instances between regions

This paper describes the process for migrating AWS running instances between regions. There are various combinations that apply to this process such as:

1. Migrating EBS AMI’s between regions

2. Migrating EBS Snapshots between regions

3. Migrating EBS Volumes on running instances between regions

4. Migrating running AWS instances between regions

The processIn each of the above scenarios, the main starting point is to have an EBS volume to work with. Once this is achieved the process is mostly the same. This paper will focus on scenario 1 in the above list. Scenarios 2 – 4 are pretty straight forward once you understand the basic process for scenario 1.

For this paper we will be migrating an AMI in US-EAST to an AMI in US-WEST

Step 1 – Starting with an AMI in US-EASTThe AMI we will be starting with is a Fedora AMI based on Fedora 64bit, in the image below you can see it labeled 193564345407/migrate_me_src:

EBS AMI

Identify the AMI-id of the AMI and then select the list of EBS snapshots. The snapshot that is used as the basis for the AMI will be labeled in the Description colum as Created by CreateImage (IMAGE-id) for AMI-id from vol-id. The AMI-id is what you will be looking for to match to the AMI-id of the AMI you are migrating:

Finding the AMI-id

Step 2 – Create a Volume from the AMI Root snapshotOnce you have identified the root volume snapshot for the AMI, the next thing you will need to do is create a Volume from the snapshot. The easiest way to do this is to right click directly on the snapshot and select create volume.

2

Creating a volume from a snapshot

Once you have created the new Volume it is a good idea to set the Name Tag so that you can recognize it easily.

Step 3 – Mount the volume on a running EC2 instanceNow that you have a Volume ready to go, the next step is to mount it onto a running EC2 instance. The easiest way to do this is to start up a Micro instance specifically for the job, then destroy it afterwards.

So go ahead and start up a Micro instance, and attach the Volume from the previous step to /dev/sdf on the new instance.

Remember to start up the Micro instance in the same availability zone as the Volume created in the previous step or you won’t be able to attach it!

Step 4 – Configure a receiving instance in the destination regionNow that we have the source region configured and ready to go we need to set up the destination region. To do this we will need to set up another Micro EC2 instance with a blank Volume attached to /dev/sdf of the same size as the source volume.

So go ahead and create a new blank Volume in the destination region and attach it to a new Micro EC2 instance.

3

When you create the new Micro instance in the destination region, create a new Key Pair just for this instance, as we will be copying the private key to the source server in the next step.

New EC2 instance in the destination region

Step 5 – Copy the private key to the source server

Now that we have the desination server running, the next step is to copy the private key used to start the server across to the /tmp directory on the source server. The easiest way to do this is as follows:

1. Right click the source EC2 instance and select Connect to Server option and copy this command.

4

Select Connect on the source server

Copy the ssh command:

Copying the ssh command

5

On your own laptop or computer, change directory to the directory where you saved your new private key and paste the same command but make the following changes.

ssh -i ping-us-east.pem root@ec2-50-19-5-173.compute-1.amazonaws.com

to

scp -i ping-us-east.pem ping-us-west.pem root@ec2-50-19-5-173.compute-1.amazonaws.com:/tmp

This command will use the private key of the source server (ping-us-east.pem) to allow scp (a secure copy program) to connect and copy the private key (ping-us-west.pem) of the destination server to the source server’s /tmp directory.

This example assumes that you have a mac or unix laptop or PC, to execute the scp command under windows you will need to download a windows scp tool.

Copying up the private key to the source server

Step 6 – Copy the volume contents from the source server to the destination server.This step is the most time consuming step of the process and involves actually copying the data between the two volumes in the separate regions.

The visual process for copying the data is as follows:

6

Process for copying data between regions

Log onto the source server and run the following commands (making sure you change the key pair name and the name of the destination server)

$> cd /tmp

$> dd if=/dev/sdf |gzip -c -1 | ssh -i ping-us-west.pem root@ec2-50-18-32-121.us-west-1.compute.amazonaws.com "gunzip -c -1 | dd of=/dev/sdf"

To break this command down the steps are:

1. dd reads /dev/sdf on the source machine (dd if=/dev/sdf) and sends the result to std out.

2. gzip reads from std in and sends the compressed output to std out

3. ssh reads from std in and pipes the data over the wire to the following command on the destination machine: "gunzip -c -1 | dd of=/dev/sdf"

4. gunzip on the destination machine reads from std in (ssh) and sends the uncompress output to std out.

5. dd reads from std in and writes the output to /dev/sdf

7

The copy process takes about an hour between US-EAST and US-WEST, although I would expect it to take much longer between the other regions.

One of the great reasons for doing it this way is that you don’t mount the volume on either the source or the destination server from the operating system, and there is no file system creation required on the destination volume.

Step 7 – Create an AMI in the destination regionTo create an AMI in the destination region you will need to do two steps:

1. Create a snapshot of the volume we just populated with data

2. Create an AMI from the snapshot

To create a snapshot, once again it is just a matter of identifying the correct volume in the EBS volumes manage and right click Create Snapshot.

Creating a snapshot from a volume

Once the snapshot has been created, identify it in the EBS Snapshots.

Right click on the snapshot and select Create Image from Snapshot.

8

Creating an image from a snapshot

Once you have selected this option the following dialog will be presented:

Create snapshot dialog

Make sure you select the same architecture type as the original AMI, (in this case the original AMI was fedora 64 bit), and give it a name. you should be able to use the default options for the rest.

9

It may be that you were using an older AMI in the source region or that the Kernel ID or Ramdisk ID’s don’t work. In this case I would recommend starting up an existing AMI in the destination region that is as close as you can get to your existing AMI and copying the Ramdisk ID and Kernel ID’s from the running instance

That’s it!

Your new AMI should now be available:

New AMI

SummaryIn this paper we looked at the process for copying an EBS backed AMI between regions. The process used was a generic process and can be used for not only copying AMI’s between regions, but for copying generic EBS backed volumes.

10