Moodle is dead…

The Problem

• On the 30th of July 2013 the VLE, DBA and Network teams of Information services were invited to a meeting which was to test our Business continuity with our VLE environment (Moodle). The teams were given the following scenario.

• All University systems have been shut down due to a full power failure which has affected both Craiglockhart and Merchiston with no other current services at our Sighthill campus; this also means that there is no external internet access from inside the University.

• Moodle as being the most critical system at this time of the year is essential and has to be back on-line as quickly as possible to allow students access to their current course work.

Team brief

• The team are allowed access to any of the offsite backup systems (VMs, Data and Databases).

• No University hardware can be used for the process.

• Staff were allowed to use their on-call laptops.

• Staff have access to budget to gain resources if needed.

Incident Procedure

Incident Manager

Coordinates troubleshooting/Information

gathering

IT Incident Duty Manager

(Technical Services)

Takes ownership of Incident through to resolution

Technical Expert(s)

Update call notes with progress

Call Management system records the various stages of the Incident.

Who was involved?

Head Of Applications

DBA?

VLE/WEBData Centre

& Operations

Edinburgh Napier’sMoodle Infrastructure

Some stats…

What Solution? (Approx. 2hrs)

• The following decisions were made during the initial Emergency Incident meeting.

1. We would use Amazon Web Services and create a virtual machine in the cloud. AWS provided scalable solutions, import/export options for database

2. We would look to get database and files backup from tape

3. Switch user accounts to manual

4. We discussed how to communicate out to students.

Tasks involved Obtain a database backup (Approx. 4hrs)

a. Installed Zmanda Community Edition to recover MySQLb. No access to DBA and use of the Zmanda recovery software was

problematic(backup had to match the version of MySQL), meant a Hot backup was obtained.

c. Truncated logs and statistics to speed up the importd. Import Database into AWS MySQL database

Obtain Moodle user files from backup (Approx. 5hrs)a. Backup came from Symantec NetBackup as we found file storage was not

going to tape at the time.

Register for Amazon Web Services (AWS) accounta. Purchased AWS Business support.

Setup AWS (Approx. 30mins)a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.b. Create a Key Pair - AWS uses public-key cryptography to secure the

login information for your instance.c. Communicated the AWS account credentials to key team members

to allow other aspects of the service to be configured.

Transfer Moodle user files to Amazon (Approx. 7hrs)a. Initial size of backup was 475GB, this was reduced to 190GB after

removal of duplicate directories and redundant course backup files.b. Initially tried using WinSCP for transfer, this was going to take 14hrs,

switched to RSYNC transfer complete in 6hrs.

Tasks involved

Tasks involved

Obtain a database backup (Approx. 4hrs)a. Installed Zmanda Community Edition to recover MySQLb. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of mysql), meant a

Hot backup was obtained.c. Truncated logs and statistics to speed up the importd. Import Database into AWS MySQL database

Obtain Moodle user files from backup (Approx. 5hrs)a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.

Register for Amazon Web Services (AWS) accounta. Purchased AWS Business support.

Setup AWS (Approx. 30mins)a. Launch Amazon console in correct region.b. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.

Transfer Moodle user files to Amazon (Approx. 7hrs)a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup

files.b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.

Tasks involved Setup AWS (Approx. 30mins)

a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.b. Create a Key Pair - AWS uses public-key cryptography to secure the login

information for your instance.c. Communicated the AWS account credentials to key team members to

allow other aspects of the service to be configured.

Transfer Moodle user files to Amazon (Approx. 7hrs)a. Initial size of backup was 475GB, this was reduced to 190GB after removal

of duplicate directories and redundant course backup files.b. Initially tried using WinSCP for transfer, this was going to take 14hrs,

switched to RSYNC transfer complete in 6hrs.

Tasks involved

Obtain a database backup (Approx. 4hrs)a. Installed Zmanda Community Edition to recover MySQLb. No access to DBA and use of the Zmanda recovery software was problematic(backup had to match the version of MySQL), meant a

Hot backup was obtained.c. Truncated logs and statistics to speed up the importd. Import Database into AWS MySQL database

Obtain Moodle user files from backup (Approx. 5hrs)a. Backup came from Symantec NetBackup as we found file storage was not going to tape at the time.

Register for Amazon Web Services (AWS) accounta. Purchased AWS Business support.

Setup AWS (Approx. 30mins)a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.b. Create a Key Pair - AWS uses public-key cryptography to secure the login information for your instance.c. Communicated the AWS account credentials to key team members to allow other aspects of the service to be configured.

Transfer Moodle user files to Amazon (Approx. 7hrs)a. Initial size of backup was 475GB, this was reduced to 190GB after removal of duplicate directories and redundant course backup

files.b. Initially tried using WinSCP for transfer, this was going to take 14hrs, switched to RSYNC transfer complete in 6hrs.

Tasks involved Setup AWS (Approx. 30mins)

a. Create EC2 instance RHEL 6.4 64bit with 7.5GB RAM, 1TB disk space.b. Create a Key Pair - AWS uses public-key cryptography to secure the login

information for your instance.c. Communicated the AWS account credentials to key team members to

allow other aspects of the service to be configured.

Transfer Moodle user files to Amazon (Approx. 7hrs)a. Initial size of backup was 475GB, this was reduced to 190GB after removal

of duplicate directories and redundant course backup files.b. Initially tried using WinSCP for transfer, this was going to take 14hrs,

switched to RSYNC transfer complete in 6hrs.

Tasks involved continued... Installation & Configuration of Services (Approx. 2hrs)

a. Apache, PHP & MySQL Installed, services started.b. Installation of GIT c. Cloned our Moodle code from Git Hub where all our commits are backed

up automatically.d. Backed up ignored files separately, for example config.php e. Ensure Apache user has correct permissions on directory.f. Alter Moodle config.php with new database credentials.

Tasks involved continued...

Moodle Administration (Approx. 1hr)a. Recreated Moodle admin accountb. Switch Moodle user accounts to ‘Manual’ and regenerate passwordsc. Successfully access Moodle user accounts and data on AWS, various

admin/teacher/student accounts accessed and tested.

Moodle in the Cloud!

Lessons Learned

• Backups were only stored on high availability disks across two campuses• Recommend

• Backup to tape every 2 weeks.

• Passwords for a number of the services were only with the DBAs• Recommend

• Storing of passwords in the same way across the teams.• Access to all passwords for the teams.

• Further Backups • Recommend

• Additional Emergency MySQL dump• Filter out redundant files from backup to reduce time• Migrate only necessary tables• Search and replace text inside sql file before import (mainly for hardcoded urls)• Replicate data up to Cloud

Lessons Learned continued…

• SMS students about the downtime of the system • Recommend

• Storing of student SMS/phone details off campus or on on-call laptops

• Moodle • Recommend

• Create basic scripts for the processes against the database on user accounts

• Active directory is one of the main services to allow our system to work• Recommend

• Store an active directory server in the cloud or at another universities network

Going forward

• Have a cloud Active Directory solution available.

• Have a parked AWS instance available when required.

• Use Vagrant/PuPHPet GUI to have consistent setup and manage virtual machines

Useful links

• Amazon Web Services - http://aws.amazon.com/

• Windows Azure (Cloud AD) - http://www.windowsazure.com/en-us/

• Vagrant/PuPHPet - https://puphpet.com/