webinar presentation: stories of accidental data loss

Confidential and Proprietary1

Did I Do That? Stories of Accidental Data Loss


Agenda Real-world Data Loss Incidents

Lessons Learned

Business Impacts of Inadequate Data Protection

Case Studies on Data Protection

Architecture Considerations for Backup & Recovery


Incident #1 – Multiple Backup Solutions Do Not Ensure Protection

Production – db1Issues, Merge Requests, Code Snippets

Standby– db2

• 6 Hours of Data Lost

• ~40 Hours of Downtime

Data Protection Production Issue Restore Issue• Snapshots• NFS Backups• S3 Backups• Database Dump• Replication

• Spammers destabilized DB• Replication stopped

working• DBA deleted files on

production DB instead of standby

• Replication not working• Snapshot 24 hours old• NFS backup could not be

located• Database dump failing• Backup to S3 not working


Incident #2 – Multiple Replicas Do Not Ensure Protection

• 2 weeks of downtime• Resources working

24x7• Total Cost - $1.1M Hadoop Cluster

Multi-PetabyteHIVE

Data

Inge

st

Data Protection Production Issue Restore Issue

• Relying on 3 replicas in Hadoop

• Developer cleaning old tables• Accidentally dropped active

table• 500 TB data lost

• All replicas deleted• No backups• Rebuilt table by re-ingesting

all data


Incident #3 – Immutable Storage Systems Do Not Ensure Protection

• Regulatory certification denied• Large fines

CRM Application

Secure Storage for Contracts

Retention Management

Data Protection Production Issue Restore Issue• Secure storage

environment with retention management

• Sales rep changed retention for the wrong customer from 24 mos. to 6 mos.

• 18 months of contracts deleted from storage

• No backups resulting in permanent data loss


Lessons Learned

Production Environment

Use descriptive database names

No testing in production databases (limited access)

Use multiple replicas to protect from hardware failure – NOT BACKUP

Use remote replication for Disaster Recovery – NOT BACKUP

Backup Architecture Design

Not all data is equalDesign based on Recovery-Time-Objectives & Recovery-Point-Objectives (RPO)Store backups off-hostKeep multiple restore pointsKnow the cost of recreating lost data

Operations

Create and maintain a playbook

Monitor backup processes

Regularly practice recovery scenarios


Business Impact of Inadequate Data Protection

Extended Application Downtime• Revenue loss• Productivity loss• Lose customers to competitors• Damage to reputation/brandData Loss• Valuable resources spent on rebuilding data• Regulatory & compliance violations resulting in

fines• Customer frustration• Lawsuits


Data Protection Case Study #1

Amazon EC2Cassandra Production Cluster

52 nodes, 58 Terabytes

BACKUP

Backup/Recovery Challenges• Backing up and restoring a 58 TB

table• Updating QA/DR cluster with

production data

Solution Characteristic• Combined backup & DR• Incremental-forever backups & restores• Storage de-duplication & compression• Backup to Amazon S3

Amazon EC2Cassandra DR Cluster

36 nodes, 58 Terabytes

DR


Talena GUI

Five node Talena

cluster usingdirect-attached

storage

Vertica Cluster10-node

Vertica Cluster10-node

HadoopCluster


Backup/Recovery Challenges

• Native backup tool had limitations

• Multiple scripts hard to manage

Solution Characteristics• Hadoop Namenode backup• Topology-agnostic Vertica

backup/recovery• Single pane of glass



Hadoop Cluster37 nodes, 425

TB

Hadoop Cluster8-node, 50 TB

Hadoop Cluster16 nodes, 180

TB

PROTECT

ARCHIVE

BACKUP

ARCHIVEBACKUP

BACKUP

Backup/Recovery Challenges• Complexity of managing

600+TB of backups• Unable to meet RPO

requirements

Solution Characteristics• Single pane of glass for all Hadoop

clusters• Hourly backup to meet stringent RPO

needs• Scale-out solution to accommodate

scale

Talena GUI


Architectural Considerations for Backup & Recovery

Dealing with Distributed Deployments• Scale-out solution• Parallel data transfers between primary & backup

Storage• An agentless modelDealing with Volume & Velocity• De-duplicated, compressed backups• Scalable, searchable catalog that can store millions of

objects• Incremental-forever backups with single-step restore• Fast & granular recoveryDealing with Data Variety• Application-aware backups and restores• Content-aware storage optimization


Q&AWe’ll send you a link to our eBook “Compendium of Data Loss Horror Stories”

Additional resources: talena-inc.com/resources and talena-inc.com/blog

Ping us with any additional questions: [email protected]


THANK YOU

webinar presentation: stories of accidental data loss

Technology