Disaster Recovery PlanningBlackBerry Enterprise Server v4.0 for IBM Lotus Domino

About Disaster Recovery• Disaster recovery is a broad term which can vary

amongst different organizations • Three main concepts of Disaster recovery are

– Hardware Failure– Smoking Hole– High Availability

• This presentation will provide guidelines for each of these three concepts

Agenda• Architecture Review

– BlackBerry Enterprise Server v2.2– BlackBerry Enterprise Server v4.0

• Disaster Recovery Points• Disaster Recovery Options

– Backup/Restore– Replication– Clustering– Move User command

• Other items to consider

Planning DR

• Items for planning– BlackBerry® should be planned in accordance

with your IBM® Lotus® Domino® DR plan– In planning off site or high availability full SQL

should be used– Include your SQL DBA to plan for BlackBerry

BlackBerry Enterprise Server v2.2 Architecture

• BlackBerry Enterprise Server runs as an Add-in Task

• BlackBerry Enterprise Server Databases:– BlackBerry User Profiles– BlackBerry Outgoing Queue– BlackBerry State Databases– BlackBerry Directory Database – BlackBerry Statistics Database

BlackBerry Enterprise Server v2.2 Architecture

What's New in BlackBerry Enterprise Server v4.0

• Requirement of relational database.– MSDE2000 or SQL2000

• BlackBerry Router – BlackBerry Handheld Manager for least cost routing

• BlackBerry Synchronization Service– Manages PIM Synchronization

• BlackBerry Policy Service– Manages Wireless IT Policy

Domino Database Changes in BlackBerry Enterprise Server v4.0• Domino Databases

– BlackBerry Profiles– BlackBerry User State Databases

• Domino Databases removed– Outbound Queue

• Information moved to the state databases– BlackBerry Directory

• Moved to SQL/MSDE– BlackBerry Statistics

• Moved to SQL

• Notes.ini configuration data now stored in SQL BlackBerry Enterprise Server Mgmt Database

BlackBerry Enterprise Server v4.0 for Domino Architecture

BlackBerry Enterprise Server

BlackBerry Profiles

DatabaseUser State

Databases

SQL

Application Servers

Lotus Domino Mail Servers

Admin PC

User 2

User 1

HTTP/S

NRPC

Corporate Firewall

Port 3101

Wireless Networks

BlackBerry Router

(Optional)

BlackBerry Profiles Database• BlackBerry Profiles Database

(BlackBerryProfiles.nsf)– Synchronized to the SQL Database– A link to the user’s BlackBerry State Database– Current 3DES or AES encryption key– Personal identification number (PIN)– Filter settings– Auto signature– Return email address– Statistics– In-cradle status and default behavior– Status: enabled or disabled– Network type (Mobitex, GPRS, DataTAC, iDen, CDMA )

BlackBerry State Databases• BlackBerry State Databases

– Critical to the operation of the BlackBerry Enterprise Server.

– The BlackBerry State Database has an instance created for each BlackBerry enabled user.

– Stores information about messages, calendar entries or PIM information that are sent or received by the BlackBerry handheld.

– Each entry links the message in the mail file and the message on the handheld by storing the message’s reference ID.

– Contains queue information, formally kept in BlackBerry outbound queue

SQL Database• Stores Configuration information

– Server Configuration• SRP• SRP Ports• Server Capabilities• Global Filters• AutoBCC• Debug Level• Licensing

– User Configuration• Username• Filters• Encryption Key (also retained in blackberryprofiled.nsf)

– User Stats

BlackBerry Router

• Optional component – Bypass – DMZ installation

• Runs as a service• Can be installed on separate server or on existing

BlackBerry Enterprise Server• Multiple BlackBerry Enterprise Servers can be

routing though one BlackBerry Router• Used with handheld manager to route data

internally

What happens when components fail?

• SQL database connection is lost– BlackBerry Enterprise Server continues to function for e-

mail and MDS– Administrator cannot administer BlackBerry Enterprise

Server or make changes– BlackBerry Enterprise Server will not be able to start if

SQL connection is unavailable• BlackBerry Router

– BlackBerry Enterprise Server attempts to connect directly to SRP Host

– Will stop routing if BlackBerry Router is in the DMZ– Internal least cost routing will stop

Disaster Recovery Points• SQL and Domino are both required to be restored in a fail

over• If you are planning disaster recovery full SQL should be

used• Data from BlackBerryProfiles.nsf is also stored in the SQL

Database• If BlackBerryprofiles.nsf is deleted a new database will be

created from SQL• BlackBerry State Databases are key component to user• BlackBerry Router may be on another machine

Disaster Recovery Options

• BlackBerry Enterprise Server for Domino – Backup and restore– Replication– Domino clustering– Move user

• SQL Database– Backup and restore– One way replication to DR site– SQL clustering

Backup/Restore• Method of choice for smaller deployments

– MSDE on same machine as BlackBerry Enterprise Server • Key data to backup

– BlackBerry Enterprise Server Mgmt database on SQL server• SQL Backup job (if SQL Server)• Open file backup

– BlackBerry Profiles Database• Located \lotus\domino\data\bes

– User BlackBerry State Databases• Located \lotus\domino\data\bes\state

– BlackBerry Enterprise Server Registry Settings• HKLM\SOFTWARE\Research In Motion

– BlackBerry Logs• Located c:\program files\Research In Motion\BlackBerry Enterprise

Server\Logs

Backup/Restore• To restore to a new server

– Install MSDE/SQL– Install BlackBerry Enterprise Server– Restore BlackBerry Enterprise Server Mgmt database

with restored copy– Restore Lotus/Domino/data/ BlackBerry Enterprise

Server folder with restored copy– Run Domino maintenance utilities

• Fixup• Updall• Compact

– Restart server

Backup/Restore• After Reboot

– Verify all Windows® Server Services started• BlackBerry Attachment Service (if on same as BlackBerry

Enterprise Server)• BlackBerry Controller (optional)• BlackBerry Enterprise Server• BlackBerry Mobile Data Service• BlackBerry Policy Service• BlackBerry Router• BlackBerry Synchronization Service

– Verify BlackBerry Enterprise Server add-in task has started

– User’s data will continue to route– User’s data is only as good as last restore, may have

some documents orphaned

Off Site DR option - Replication• Domino

– Key Data To Replicate• BlackBerryProfiles.nsf • User BlackBerry State DB

– Configure replication interval– Install BlackBerry Enterprise Server using same SRP id and key as

Primary• Use same version (Service Pack, Hotfix)

– Remove BlackBerry Enterprise Server task from Notes.ini on backup BlackBerry Enterprise Server

– Change the server document to deny user access– Copy of primary server.id to failover BlackBerry Enterprise Server

• SQL– Setup SQL replication and replicate BlackBerry Enterprise Server

Mgmt one way to remote site– This will be a role for DBA

Off Site DR Failover • If possible remove BlackBerry Enterprise Server task from

Primary Notes.ini• If you are using BlackBerry Desktop Manager

– Change the server.id to the primary server– Verify that the server is resolvable– Start Domino and BlackBerry Enterprise Server Task

• If you do not change the desktop clients will need to select the new BlackBerry Enterprise Server

• If you do not use BlackBerry Desktop Manager in your environment you can just start the BlackBerry Enterprise Server task

• Data should continue to flow for all users• MDS as a push service will not continue

– The application servers which push would need to point to the new host name

Off Site DR Architecture Replication

Production Site

SQL Server

BlackBerry Enterprise

Server

BlackBerryprofiles.nsfBlackBerry

State Databases

BlackBerry Enterprise Server Mgmt

DR Site

SQL Server

BlackBerry Enterprise

Server

BlackBerryprofiles.nsfBlackBerry

State Databases

BlackBerry Enterprise Server Mgmt

One Way SQL replication of

BlackBerry Enterprise Server Mgmt

Database

One way replication of BlackBerry Profiles

Database and User State Database

Users Denied Access

Extending Availability• Clustered SQL Server• Clustered BlackBerry Enterprise Server for Domino

– 2 Clustered Domino Servers– BlackBerry Server installed on each server

• Use same version (service pack, hotfix)– Use clustered BlackBerry Enterprise Server Mgmt

database on clustered SQL server– Add-in task not running on backup

• Remove BlackBerry Enterprise Server task from notes.ini– BlackBerry Windows Services not running

• Set to manual– Deny users access to backup server to prevent

replication/save conflicts

During a failure• If possible remove BlackBerry Enterprise Server task from

Notes.ini on primary server• Start all BlackBerry Windows Services• Start BlackBerry Enterprise Server task on backup

BlackBerry Enterprise Server – Add BlackBerry Enterprise Server to server tasks in notes.ini

• User data will continue to flow• BlackBerry Desktop Manager will connect to the backup

BlackBerry Enterprise Server after it is closed and reopened

• MDS Push applications will not function as they will be pointing to the FQDN of the primary server

High Availability Architecture

Clustered SQL servers

SQL Clustered BlackBerry Enterprise

Server Mgmt

Domino Clustered BlackBerry Enterprise Servers during normal operation

Primary Server BlackBerry

Enterprise Server Task Running

Backup Server BlackBerry Enterprise Server Task off

Clustered Databases

Domino Mail Servers

High Availability Architecture

Clustered SQL servers

SQL Clustered BlackBerry Enterprise

Server Mgmt

Domino Clustered BlackBerry Enterprise Servers during failover

Primary Server BlackBerry Enterprise Server Task disabled

Backup Server BlackBerry Enterprise Server Task Running

Clustered Databases

Domino Mail Servers

Backup BlackBerry Enterprise Server takes over routing data to the users handhelds after it has been manually started.

If possible disable the primary to prevent the primary BlackBerry Enterprise Server from attempting to come back online

Move User• About Move User

– Move user can move a user from one BlackBerry Enterprise Server to another with no user interaction

– User configuration data is stored in SQL shared by all BlackBerry Enterprise Server

– A replica of the users BlackBerry State Database will automatically be created on the destination BlackBerry Enterprise Server

Move User –cont’

– If current BlackBerry Enterprise Server is not available the move will fail because access to the user’s state database is unavailable

• DoUserMove: Cannot open profiles database on server CN=Primary Domino/O=Home. Attempting sourceless move... (The server is not responding. The server may be down or you may be experiencing network problems. Contact your system administrator if this problem persists. 0x0807)

• DoUserMove: state database C:\Lotus\Domino\Data\BES\state\1696641664.nsf does not exist for user CN=Gregg Testing/O=Home; will not perform sourceless move

Move User – cont’• Move User Process

– Move change is entered in SQL– User State Database is replicated to destination

BlackBerry Enterprise Server – Destination BlackBerry Enterprise Server looks for user

BlackBerry SD• Replica copy now on destination BlackBerry Enterprise Server

– Profiles Database is updated with new user information on the new BlackBerry Enterprise Server and removed from the old

– Updated service books are sent to the handheld with the new SRP information

– User move is complete and they are able to send and receive data

How does Move User fit into DR?• Moving Users if their BlackBerry Enterprise Server is unavailable

– You can move a user to secondary BlackBerry Enterprise Server with a unique SRP ID

– The key is to maintain replica copies of user BlackBerry State databases on the alternative BlackBerry Enterprise Server Replicating the profiles database is NOT necessary

– In the event of a failure use the Move User tool in the BlackBerry Management console

– The alternate BlackBerry Enterprise Server will locate the local replica copy of the state database and create an entry in the Profiles database from the information stored in SQL• Encryption Key• Filter• ETC

– Updated service books are send to the handhelds– The users will now be active on the alternate BlackBerry Enterprise Server

Move User DR Architecture

BESA (Active BlackBerry Server)

BlackBerry Enterprise Server B (Active BlackBerry Enterprise Server)

BlackBerry Enterprise Server Mgmt database contains master copy of user information for

all BlackBerry Enterprise Servers

Admin PCLotus Domino Mail Servers

BlackBerry Enterprise Server A profiles

database

BlackBerry Enterprise Server B profiles database

User state databases

Replica copy of user state databases

One way replication of user BlackBerry State Databases

User 1 active on BlackBerry

Enterprise Server A with record in

profiles database

During normal operation

Move User DR ArchitectureBlackBerry Enterprise Server A

(Active BlackBerry Enterprise Server)

BlackBerry Enterprise Server B Active BlackBerry Enterprise Server))

2. User 1 now changed to be running on BlackBerry Enterprise

Server B in the BlackBerry Enterprise Server Mgmt database

Admin PCLotus Domino Mail Servers

BlackBerry Enterprise server A profiles

database

BlackBerry Enterprise ServerB profiles database

User state databases

Replica copy of user state databases

1. Admin uses Move User to change

BlackBerry Enterprise Server for User 1

During failure of BlackBerry Enterprise Server A

3. Update is sent to BlackBerry

Enterprise Server B 4. User 1’s state

database is detected

5. An entry for User 1 is created in BESB’s BlackBerry

profiles domino database

The user is now active again

Other Considerations

• BlackBerry Router– If the BlackBerry Router is installed on another machine

• A backup machine can be installed with BlackBerry Router• It also can be running simultaneously as Primary

– If the primary BlackBerry Route fails• Manually point BlackBerry Enterprise Server to alternate

BlackBerry Router– Run BlackBerry Enterprise Server configuration and change the

router host– Registry Change HKLM\Software\Research In Motion\BlackBerry

Enterprise Server\Dispatcher\NetworkAccessNode

Domino Directory

• Domino Directory– BlackBerry Enterprise Server uses the Domino Universal

Note ID (UNID) of the Person document to uniquely identify each user.

– The UserUNID field is stored in the profiles database and links the BlackBerry user to the Lotus Notes user.

– Restore a Replica of the Domino Directory• Copy the Domino Directory at the file system level• If you fail to use a file system-level backup, users will be unable

to send or receive messages with their handheld because their Lotus Domino UNID has changed.

User Mail Databases

• User Mail Databases– Restore a replica of the users mail database

• This maintains the UNID link between mail docs in the users mail database and the docs in their BlackBerry State Database

– Problems caused by restoring database copies• If you fail to use a replica, users will be unable to reply-with-text,

forward, or request more content on the handheld for messages that are in the restored mail database, and all calendar items on the handheld will no longer be linked to the calendar items in Lotus Notes

Summary• Best options for Disaster Recovery

– Hardware Failure• Backup/Restore• Replication• Clustering• Move User

– High Availability• Clustering• Move User

– Smoking Hole• Replication• Move User

Thank You

Questions?