Dan Lawson, All Sites

VectorBase Releases

2 VectorBase 2012

A release cycle for VectorBase

• Regular release every 2 months• In place since June 2010

• Latest release is the 10th incarnation

• Review the aims of the release cycle• Evolution of the release procedure• Tracking/Organisation of a release• New hardware • Assessing releases• Current site responsibilities

2

3 VectorBase 2012

Aims of the release cycle for VectorBase

• Regular releases encourage• (More) rapid update of data content

• Pre-sites for new organism data• Updates for code base

• As VectorBase is semi-parasitic there is a need to be reasonable current with Ensembl web code base

• Benefit from external development (useability, functionality)

• De-mystify the procedure of making a release• Instill a service provision mentality for part of the project

3

4 VectorBase 2012

Timeline of release improvements

4

• VectorBase used to release on an ad-hoc schedule based on data cues• Staff turnover rapidly made this process troublesome because of php

hooks into the Ensembl web code• Effectively VectorBase could not update its browser code• ND developers visited EBI in May 2009 to overcome this bottleneck• at this point we were running e! 46 code with e!54 databases• Scott & Dan discussions in a Rennes cafe, Nov 2009• EBI to configure browser, ND to install • Apr-May 2010. EBI configures and tests new browser code • First VectorBase release in June 2010

5 VectorBase 2012

Timeline of release improvements

5

• Problems with browser installation at ND continued and from the VB_2010-10 release EBI both configured and installed the browser on ND servers

• Ongoing small issues with server configuration, dns addresses etc.• Mid 2011; proposal to use Virtual Machines (VMs) to encapsulate the

browser code• Late 2011; Refinement of what is to be included in each VM,

troubleshooting transfer and deployment of VMs from EBI to ND• End of 2011; New linux hardware comes online. Should provide

performance improvements in terms of raw speed but also resolve some compatability issues between browser code and legacy Mac servers

6 VectorBase 2012

Release coordination

• Release coordination is achieved via the Tuesday Developer calls and makes use of the project wiki and JIRA systems

• Uses semantic wiki extension of MediaWiki (SMW) to track deliverables and progress within a release

• Uses JIRA versions to document and track issues arising from a release

• Developer calls are the main forum to communicate issues and updates within the group

6

7 VectorBase 2012

Overview of release

7

8 VectorBase 2012

Release coordination on Project Wiki

8

9 VectorBase 2012

Summary overview

9

10

VectorBase 2012

Summaries and reports

10

11

VectorBase 2012

Overview of release

11

12

VectorBase 2012

Assessing releases

• The VectorBase site is complex• Many dependencies, not all of which are documented

• The project needs to have some concept of:• Monitoring serve uptime and potential issues without relying on

the user community to notify us• Expand knowledge and ability to rectify issues• Incremental improve ability to identify issues with release before

going live

12

13

VectorBase 2012

Monitoring infrastructure

• VectorBase has tested some systems for monitoring server uptime (e.g. nagios)

• Improved documentation, SOP for site maintenance

13

14

VectorBase 2012

Assessing a release

• Use of predefined tests to check site• Use of checklists to confirm pages/views are correct

• Always test new organism or gene set• Test random organism for a release (we can’t do all of them!)

• Use of testing suites such as Selenium• Under used, should be developed and ran more frequently

14

15

VectorBase 2012

Use of Virtual Machines (VMs)

• “completely isolated guest operating system installation within a normal host operating system”

• With reference to the Ensembl browser code• Configuration and testing of installation on linux server• Encapsulation inside a VM• Disk image sent to ND for deployment on their servers• Should be no compatability issues

• Advantages• Significantly easier deployment at ND• Should reduce system architecture issues

• Disadvantages• Potentially harder to make small corrections to deployment

15

16

VectorBase 2012

Should I stay or should I go

• What should be within the VM?• Browser web code and configuration/plugins• MySQL database server (data)• BioMart• Funcgen• PopBio

• Considerations• Testing an installation is time consuming• Testing is only truly possible when the whole site is deployed

16

17

VectorBase 2012

New hardware at Notre Dame, Linux servers

17

18

VectorBase 2012

Release Shangri-La

• VM built by EBI• Genome browser webcode and configuration• MySQL server with all relevant databases

• VM built by ND• Current php/postgres site (includes static content, CAP)• BioMart

• VM built by Imperial• Functional genomics (potentially)• PopBio (potentially)

18

19

VectorBase 2012

Release wish list for next 6 months

• The current release is the first one on the new linux servers. We need to monitor this and potentially review resource allocation (genome browsers are memory hungry)

• Renew efforts with regard to project wiki tracking/coordination• Extend and make more visible Selenium tests for the site

• When we encounter a bug/issue we should be thinking can I emulate this as a Selenium test. If yes then this should be shared and added to the regular suite of tests to be run. We are not saturated where running tests has become onerous.

• Better reporting of server monitoring. Do we have stats for server uptime etc. Should we be internally discussing these?

19

20

VectorBase 2012

Release wish list for next 6 months

• Agree (and adhere to) mechanism of building dev/pre/live.• Production/Live (Twilight) is essentially read-only• Rotate Galactica and Futurama between staging/pre and development• Development site needs to have components being tested• Staging/pre site needs to be a complete deployment• Release procedure then becomes synch/copy of VMs to the production

server, dev server becomes staging and staging becomes new dev.• EBI will maintain genome browser and MySQL server within a VM on

the staging server• ND will develop main site on dev server and support main site and

BioMart on the staging server

20