cassandra london march 2016 - lightening talk - introduction to incremental repair

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

CASSANDRA LONDON FEBRUARY 2016

LIGHTENING TALK: INTRODUCTION TO

INCREMENTAL REPAIR

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

Aaron Morton@aaronmorton

Co-Founder & Team Member

About The Last Pickle.

Work with clients to deliver and improve Apache Cassandra based

solutions.Based in New Zealand, USA,

Australia, France.

Repair Incremental Repair

Repair

:(

Why Repair?

It is the way to ensure consistent data on disk.

Why Repair?

Hinted Handoff and (non-blocking) Read Repair are

optimisations.

Why Repair?

Consistency ensures Tombstones are distributed,

preventing deleted data returning.

A Repair Of Two Halves

Step 1:Calculate a hash tree to

describe the data.(aka Merkle Tree)

A Repair Of Two Halves

Step 2:Stream differences between

nodes.

gc_grace_seconds

Repair will ignore Tombstones older than gc_grace_seconds.

So run repair every gc_grace_seconds.*

RepairIncremental Repair

Why Incremental Repairs?

SSTables are immutable, so repair was processing data

that had not changed.

Why Incremental Repairs?

Repairing less data, more frequently, reduces the impact

on the cluster.

repairedAt

Track repaired timestamp per SSTable.

Ignore SSTables that have been repaired.

repairedAt

Set after when all data in the SSTable was repaired.

repairedAt$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 0

$ bin/nodetool repair --incremental --parallel

$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 1454500502306

Anti Compaction

Splits partially repaired SSTables into Repaired and

Un-repaired SSTables.

Anti Compaction$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-9-Data.db keyspace1-standard1-ka-10-Data.db

$ bin/nodetool repair --incremental --parallel -st 3074457345618258602 -et -9223372036854775808

$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-11-Data.db keyspace1-standard1-ka-12-Data.db keyspace1-standard1-ka-13-Data.db keyspace1-standard1-ka-14-Data.db

Anti Compaction$ tools/bin/sstablemetadata keyspace1-standard1-ka-11-Data.db SSTable: ... Repaired at: 1454504607623

$ tools/bin/sstablemetadata keyspace1-standard1-ka-12-Data.db SSTable: ... Repaired at: 0

$ tools/bin/sstablemetadata keyspace1-standard1-ka-13-Data.db SSTable: ... Repaired at: 1454504607623

$ tools/bin/sstablemetadata keyspace1-standard1-ka-14-Data.db SSTable: ... Repaired at: 0

Compaction

Two sets of SSTables are maintained: Repaired and Un-

repaired.

Compaction

Compaction Strategy runs in each SSTable set.

Migration Option 1

1. Disable compaction.2. Run a classic full repair3. Stop the node4. stablerepairedset to mark all the SSTables from before step 1

Migration Option 2

bin/nodetool repair --incremental --parallel

More Information

http://www.datastax.com/dev/blog/more-efficient-repairs

Thanks.

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

Aaron Morton@aaronmorton

Co-Founder & Team Memberwww.thelastpickle.com

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License