Business Continuity Planning for OPEN

OPEN Development ConferenceSeptember 18, 2008

Ravi RajaramIT Development Manager

9/18/08 Business Continuity 2

Two Perspectives

• IT – Deals with systems redundancy, system recovery strategies

under different system failure scenarios

• Operations– Deals with operational aspects of the system unavailability and

planning to perform the minimum required tasks.

9/18/08 Business Continuity 3

IT Perspective …• Oracle database with hot standby

• Database server on Data guard. No loss of committed transactions

• 15 min switch over from primary database server to the standby

• Primary and failover servers – Located on two different building, different power lines and different ISP

• Server monitoring 24/7. System groups gets notified within minutes of a server going down.

• Two firewalls for redundancy

• Company wide or area power failure, – Battery power - for the first 20 minutes - short term – Switched to back up diesel generator within 20 minutes – Long term

9/18/08 Business Continuity 4

IT Perspective …Contd.

• DB Physical backup– Daily incremental– Weekly full– Incrementals are recycled after a month– Fulls are kept for a year.

• Backup files – offsite storage

• Two application servers (for Jboss) – Primary and secondary– Automatic load balancing - Traffic routed based on the load

• T1 Link to CTEP database - Planning for web service based user credentialing

• Two report servers. One acts as a failover server.

9/18/08 Business Continuity 5

IT Perspective- Security• All the data entered and viewed over the internet is protected by 128

bit level encryption with SSL

• Three network zones Wesnet, Data and Web with varying levels of access restrictions configured in the firewalls

• Intrusion detection software running on the firewall

• Regular network penetration testing

• Access to the server room to authorized personnel

• Redundant air-conditioning system for server rooms

• Network login passwords need to be changed every 90 days

• More info on the security overview document

9/18/08 Business Continuity 6

IT Perspective …Contd.

9/18/08 Business Continuity 7

Operations Perspective …

• In the event of the OPEN System not being available due to a system failure and/or the Cooperative Group randonode not being available, the CTSU will provide for the manual processing of enrollments after 4 hours of down time.

• Manual processing of enrollments will depend on the ability of the Cooperative Group to process registrations/randomizations manually.

• Manual processing of enrollments may be limited to time of need enrollments.

9/18/08 Business Continuity 8

Operations Perspective … Contd.

GROUP - MANUAL BACK UP GROUP – NO MANUAL BACKUP

OPEN AVAILABLE/GROUP

RANDONODEUNAVAILABLE

Group views enrollment in OPEN

Group manually assigns treatment arm and/or patient ID

Information entered in OPEN by Group or CTSU OA

Site receives registration or randomization information via OPEN

OPEN collects enrollment data from site

Enrollment data remains in Group queue

Enrollment data is sent to Group randonode when available

Patient registration is delayed until the Group RandoNode is available

OPENUNAVAILABLE/

GROUPRANDONODEAVAILABLE

Sites fax enrollments directly to CTSU

CTSU will review data and fax to Group for treatment arm and/or patient ID

CTSU will fax patient enrollment information back to site.

CTSU/Site will hold enrollment forms until system is available.

9/18/08 Business Continuity 9

Contingency PlansContd.

GROUP - MANUAL BACK UP GROUP – NO MANUAL BACKUP

OPENUNAVAILABLE/

GROUPRANDONODEUNAVAILABLE

Sites fax enrollments directly to CTSU

CTSU will review data and fax to Group for treatment arm and/or patient ID

CTSU will fax patient enrollment information back to site.

Site will hold enrollments until both systems are available

Upon OPEN system unavailability, CTSU will

• Communicate to the Groups

• Post a notification on the CTSU Members’ Web Site within one hour. – Site instructions and the Help Desk phone number will be included in the

notification. – Once systems are restored, the notification will reflect the update.

9/18/08 Business Continuity 10

Version Control• OPEN portal software version

• Protocol form version

• Web Services Definition Language (WSDL) version for Randonode

• Other software versions changes (AXIS)

9/18/08 Business Continuity 11

OPEN Software Version• Affects site and Group users

– Does not require changes in Group Randonode

• Training needed based on the extent of the changes. – Communication to the Groups regarding the changes

• Software controlled by the SDLC process. – Change control board. – Group representation to decide the software changes.

• Major version development is through the RUP process– Inception, elaboration, construction, transition and production – Black box testing, regression testing and performance testing – Traceability matrix

9/18/08 Business Continuity 12

Checklist Form Changes• Affects site users

– Group Randonode should accommodate the new version of the Checklist metadata

• Handling change request – Procedure document available

• Form changes need to be done in caDSR first before propagated to OPEN.

• OPEN Data service - for downloading the new version of metadata xml

• Testing requirements– Form navigation– Transfer of data– Edit check (if required)

• New version of form moved to production upon Group approval

9/18/08 Business Continuity 13

Checklist Form ChangesContd.

• Changes that will result in new metadata– Addition or deletion of question– Addition or deletion of valid value

• Changes that will NOT result in new metadata– Edit checks– Screen layout

9/18/08 Business Continuity 14

WSDL Changes• WSDL – Web Services Definition Language

– Does not affect site users. Group randonode is affected.– Ability to handle different WSDL versions for different Groups

• WSDL changes are required only when input/output object definition for the Group randonode is changed.– Not expected to happen often

• Major WSDL changes– Groups can decide their timeline for moving to the new version

• More Info during the IT session

AXIS upgrades

• Expected to affect the randonode. Testing using the test environment before upgrading.