DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen

Download DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen

Post on 12-Apr-2017

681 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

  • Testing Cassandra Guarantees under Diverse Failure Modes with JepsenJoel Knighton

    @joelknighton

    DataStax

    #CassandraSummit

  • Who I am

    Mathematician

    Software hobbyist

    Logic enthusiast

    Former DataStax Intern

    DataStax Cassandra Developer

  • What I Do

    Deconstruct

    Formalize

    Communicate

    Prove

    Automate

  • How We Test #1

    Unit Testsant test

    in-tree

  • How We Test #2

    Distributed Testsnosetests

    On GitHub available at riptano/cassandra-dtest

  • Why Youre Here

    JepsenKyle Kingsbury (aphyr)https://aphyr.com/tags/jepsen

  • What Jepsen Is

    A blog series about distributed systems behavior

    A talk series about distributed systems behavior

    A Clojure library to test the behavior of distributed systems

    A collection of tests written using those libraries

  • What We Hope

    Jepsen

    Cassandra

  • What I Did

    Jepsen Testslein test

    On GitHub available at riptano/jepsen

  • A Test Incarnate

    {:name

    :os

    :db

    :client

    :generator

    :conductors {:nemesis }

    :checker }

    names the results

    prepares the os

    configures/starts/stops the db

    interacts with the db

    instructions on how to interact

    interacts with the environment

    looks at and assesses test run

  • What You Need

    One machine to run the tests

    +

    n machines to run Cassandra

  • How A Test Runs

    lein testos

    n1

    n2

    n3

    n4

    n5

  • How A Test Runs

    lein testdb

    n1

    n2

    n3

    n4

    n5

  • How A Test Runs

    lein testclient 1client 2client 3client 4client 5nemesis

    n1

    n2

    n3

    n4

    n5

    readwrite 3

    start nemesiswrite 4

    readstop nemesis

    write 1cas 2 -> 3

  • How A Test Runs

    lein testchecker

    1 read2 write 3 1 read 0n start nemesis2 write timed-out3 write 4n started nemesis3 wrote 44 read4 read 4n stop nemesis0 write 11 cas 2 -> 3n stopped nemesis

    valid?

    Latency

  • Single Test Deep-Dive

    lein test :only

    cassandra.collections.set-test/

    cql-set-isolate-node-decommission

  • Single Test Name

    Test name used to label folder where test results, logs, and history will be stored with timestamp

    cassandra cql set isolate node decommission

  • Single Test Nodes

    [:n1 :n2 :n3 :n4 :n5]

  • Single Test Net

    net/iptables

    (drop! ;use iptables to drop packets)

    (heal! ;flush iptables)

  • Single Test OS

    debian/os(setup! ;adjust hostfile

    ;update package manager;install base packages like curl, iptables, etc.

    ;make sure network is healed)(teardown!)

  • Single Test DB

    cassandra.core/db(setup! ;shutdown and wipe Cassandra if running

    ;install, configure, and start Cassandra)(teardown! ;shutdown and wipe Cassandra)

    (log-files ;return path to log files)

  • Single Test Client

    cql-set-client(setup! ;driver connect to all nodes

    ;create schema)(invoke! ;add? Run CQL to add to set, handle errors

    ;read? Read value of CQL set, handle errors)(teardown! ;disconnect driver)

  • Single Test Generator

    (gen/phases

    (->> (adds)

    (gen/stagger 1/10)

    (gen/delay 1/2)

    std-gen)

    (read-once))

  • Single Test Conductors

    {:nemesis (nemesis/partition-random-node)

    :decommissioner (c/decommissioner)}

  • What a Conductor Is

    Its just a client

  • Single Test Checker

    checker/set(check ;look at history of run

    ;find ok or uncertain adds

    ;compare these to final read

    ;return map with validity and

    ;ok, lost, unexpected, recovered)

  • Invariants We Test

    Do CQL collections (maps, sets) merge cleanly when add-only?

    Do counters merge to accurately reflect increments/decrements?

    Does LWT in a single datacenter allow us linearizability?

    Do materialized views converge to matching the base table?

    Do batch writes eventually get applied atomically?

  • Failures We Consider

    How does this work under a variety of network partitions?

    What about with node crashes?

    Even if nodes are flushing and compacting?

    And when nodes are being bootstrapped?

    Or decommissioned?

    While clocks drift?

  • How We Run

    Start the Docker container

    Install Java driver, Cassaforte, clj-ssh, and Jepsen

    Use environment variables to point to build under test

    Run lein test with any desired selectors and profiles

  • Tunable Options

    Should we make a best-effort attempt to scale test length?

    Should we enable commitlog compression, the coordinator batchlog on materialized views, or hinted handoff?

    Is a different compaction strategy or phi value in the failure detector appropriate for this test?

    Should we install from a tagged release, a URL pointing to a tarball, or a local tarball?

    Should we leave Cassandra running after the test?

  • What Weve FoundIssues with counter undercounting/overcounting (#10143)

    Decommission race conditions causing gossip problems (#10231)

    Write durability violations when recovering commitlog (#9851)

    Problems with merging of collections (#10001)

    Batchlog replay failures after decommission/crash (#10068)

    Incorrect asserts in counter write-path when timestamps collide

    A variety of materialized view issues during development

  • Work We Shared

    Minor Jepsen fixes/features (Jepsen PRs #58, 59, 62)

    Docker images to run Jepsen tests (Docker Hub: tjake/jepsen)

    Multibox Vagrant configurations to run Jepsen tests (on GitHub)

    Upstream library fixes (clj-ssh PR #36)

    Cassandra Jepsen tests (on GitHub)

    Available on CassCI (on cassci.datastax.com)

  • Jepsen on CassCI

  • Lessons I Learned

    Tests verifying invariants under failures are valuable and practical

    These tests can and should be a part of regular development

    Testing complex systems is hard, but there are low-hanging fruit

    Jepsen provides one readily available way to accomplish this goal

    Considering invariants against a recorded test run is effective

    Invariants should be explicit and carefully considered in design

  • Thanks

    Jake Luciani

    DataStax

    The Cassandra community

    Kyle Kingsbury

  • QUESTIONS?TLA+ TLC TLAPS Clojure

    Formal Methods Jepsen CRDTs Cassandra GossipConsistency Models Alloy

    Model Checking Testing

    @joelknighton#CassandraSummit