cassandra day atlanta 2015: troubleshooting with apache cassandra

Troubleshooting Cassandranodetool & system.log deep dive

J.B. Langston, Senior Support Engineer

Agenda

1 Things to keep in mind

2 Useful Linux & Java tools

3 Useful nodetool commands

4 Things to look for in system.log

Troubleshooting Process

1 Ask what changed

2 Determine which nodes have problems

3 Find and understand errors

4 Examine bottlenecks

5 Determine root cause

6 Take corrective action

Troubleshooting Process

1 Ask what changed

5 Share relevant information

6 Ask for help

What changed?

• Did it work before?

What changed?

• What did you change?

What changed?

• What did you change?• Settings

What changed?

• What did you change?• Settings• Application Code

What changed?

• What did you change?• Settings• Application Code• Read/Write Load

What changed?

• What did you change?• Settings• Application Code• Read/Write Load• Data Volume

What changed?

• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware

What changed?

• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network

• Does it work in another environment?

What changed?

• Did you upgrade?

What changed?

• Did you upgrade?• Cassandra

What changed?

• Did you upgrade?• Cassandra• Kernel

What changed?

• Did you upgrade?• Cassandra• Kernel• JVM

What changed?

• Did you upgrade?• Cassandra• Kernel• JVM• Driver

What changed?

• Did you upgrade?• Cassandra• Kernel• JVM• Driver

• Change one thing at a time!

What changed?

Cassandra Processes

• Startup

Cassandra Processes

• Startup

• Foreground

Cassandra Processes

• Startup

• Foreground• Coordinating requests

Cassandra Processes

• Startup

• Foreground• Coordinating requests• Local reads

Cassandra Processes

• Startup

• Foreground• Coordinating requests• Local reads• Local writes

• Background

Cassandra Processes

• Startup

• Background• Flushes

Cassandra Processes

• Startup

• Background• Flushes• Compactions

Cassandra Processes

• Startup

• Background• Flushes• Compactions• Garbage collections

Cassandra Processes

• Startup

• Background• Flushes• Compactions• Garbage collections• Gossip

Cassandra Processes

• Startup

• Background• Flushes• Compactions• Garbage collections• Gossip• Hinted Handoff

Cassandra Processes

• Startup

• Background• Flushes• Compactions• Garbage collections• Gossip• Hinted Handoff• Read Repair

Cassandra Processes

• Startup

System Resources

• CPU

System Resources

• CPU• Single Core utilization

System Resources

• CPU• Single Core utilization• Multi Core utilization

System Resources

• Memory

System Resources

• Memory• Heap space

System Resources

• Memory• Heap space• Off-heap space

System Resources

• Memory• Heap space• Off-heap space• OS page cache

System Resources

• Garbage Collection

• Disk

System Resources

• Disk• Disk space

System Resources

• Disk• Disk space• I/O bandwidth

System Resources

• Network

System Resources

• Network

• OS Resources/Limits

System Resources

• Network

• OS Resources/Limits• File Handles

System Resources

• Network

• OS Resources/Limits• File Handles• Processes

System Resources

• Network

• OS Resources/Limits• File Handles• Processes• Mapped Memory

System Resources

Linux monitoring commands

Command What it tells you…

Command What it tells you…top CPU utilization and memory use per process

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per process

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk space

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilization

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usage

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections established

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections establishediftop Network bandwidth utilization

Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections establishediftop Network bandwidth utilizationsar All (most) of the above, with history!

Java monitoring commands

Command What it tells you…jstack -l Status and stack trace of each thread

Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)

Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generation

Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activity

Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activityjmap -dump Take a heap dump for further analysis

Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activityjmap -dump Take a heap dump for further analysisMemoryAnalyzer Post-mortem heap-dump analysis

nodetool commands

Command What it tells you…status / ring Overall cluster status

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single node

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single node

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single node

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local node

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local node

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local nodecompactionstats Compactions pending and in progress

nodetool commands

Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local nodecompactionstats Compactions pending and in progressnetstats Network activity: streams, read repair, and in-flight commands

nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace

Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b

Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a

nodetool ringNote: Ownership information does not include topology; for complete information, specify a keyspace

Datacenter: us-east ========== Address Rack Status State Load Owns Token -3074457345618258603 54.174.19.98 1b Down Normal 22.82 MB 16.67% -9223372036854775808 54.174.245.247 1b Up Normal 22.68 MB 16.67% -6148914691236517206 54.173.171.164 1b Up Normal 22.78 MB 16.67% -3074457345618258603

Datacenter: us-west ========== Address Rack Status State Load Owns Token 6148914691236517205 54.153.39.203 1a Up Normal 22.67 MB 16.67% 0 54.153.107.100 1a Up Normal 22.72 MB 16.67% 3074457345618258602 54.153.108.157 1a Up Normal 22.79 MB 16.67% 6148914691236517205

nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds

nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0

Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0

nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0

Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0

nodetool cfhistogramsKeyspace1/Standard1 histograms

SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1

Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2

Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340

nodetool cfhistogramsWrite Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839

Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494

nodetool proxyhistogramsRead Latency (microseconds) 124 us: 18 149 us: 445 179 us: 5866 215 us: 14541 258 us: 19123 310 us: 20941 372 us: 32847 446 us: 37464 535 us: 26451 642 us: 25291 770 us: 37870 924 us: 35865 1109 us: 30432 1331 us: 22929 1597 us: 17592 1916 us: 12895 2299 us: 9760 2759 us: 7887 3311 us: 6024 3973 us: 4751 4768 us: 3691 5722 us: 3038 6866 us: 2453 8239 us: 1991

Write Latency (microseconds) 29 us: 4 35 us: 60 42 us: 331 50 us: 471 60 us: 1334 72 us: 2856 86 us: 4661 103 us: 65741 124 us: 37999 149 us: 42697 179 us: 45598 215 us: 30515 258 us: 23481 310 us: 21831 372 us: 22741 446 us: 26045 535 us: 33045 642 us: 43533 770 us: 51021 924 us: 57496 1109 us: 61548 1331 us: 60741 1597 us: 57538 1916 us: 52255

nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s

pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a

nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db

851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db

3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db

34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db

9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db

1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db

46561942/46561942 bytes(100%) sent to /54.153.39.203

Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746

Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081

46561942/46561942 bytes(100%) sent to /54.153.39.203

system.log

Default Location /var/log/cassandra/system.log

system.log

Configuration File /etc/dse/cassandra/log4j-server.properties

system.log

Log SettingsLocationLog Settings

log4j.appender.R.File=/opt/dse-4.6.0/logs/system.log

system.log

log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings

Logging Level

Log Settings

log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG

system.log

Logging Level

Log Settings

Class Override

Log Settings

log4j.logger.org.apache.cassandra.package.Class=DEBUG

system.log

Logging Level

Log Settings

Class Override

Log Settings

Basic FormatLevelBasic Format

Basic Format

system.log

Logging Level

Log Settings

Class Override

Log Settings

Basic Format

Basic FormatThread Type & ID

Basic Format

[CompactionExecutor:155]

Basic Format

[GossipTasks:1]

Basic Format

[AntiEntropySessions:1]

Basic Format

[OptionalTasks:1]

system.log

Logging Level

Log Settings

Class Override

Log Settings

Basic Format

[GossipTasks:1]

Basic Format

[OptionalTasks:1]

Basic FormatDate & Time

Basic Format

2015-02-13 02:18:40,986

Basic Format

2015-02-17 19:47:37,331

Basic Format

2015-02-17 20:32:11,959

Basic Format

2015-02-20 11:29:14,056

system.log

Logging Level

Log Settings

Class Override

Log Settings

Basic Format

[GossipTasks:1]

Basic Format

[OptionalTasks:1]

Basic Format

2015-02-13 02:18:40,986

Basic Format

2015-02-17 19:47:37,331

Basic Format

2015-02-17 20:32:11,959

Basic Format

2015-02-20 11:29:14,056

Basic FormatSource File

Basic Format

CompactionTask.java

Basic Format

Gossiper.java

Basic Format

CassandraDaemon.java

Basic Format

ColumnFamilyStore.java

system.log

Logging Level

Log Settings

Class Override

Log Settings

Basic Format

[GossipTasks:1]

Basic Format

[OptionalTasks:1]

Basic Format

2015-02-13 02:18:40,986

Basic Format

2015-02-17 19:47:37,331

Basic Format

2015-02-17 20:32:11,959

Basic Format

2015-02-20 11:29:14,056

Basic FormatSource File

Basic Format

CompactionTask.java

Basic Format

Gossiper.java

Basic Format

CassandraDaemon.java

Basic Format

ColumnFamilyStore.java

Basic FormatLine No.

Basic Format

(line 287)

Basic Format

(line 648)

Basic Format

(line 199)

Basic Format

(line 298)

Exceptionsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Stack Tracejava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Organizationjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Subsystemjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Classjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Nested Classesjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Methodsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Source Filejava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Line Numberjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Nested Exceptionsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more

Exceptions – Error Messageorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more

Exceptions - Organizationsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more

Exceptions - Subsystemsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more

Google Tips

Google TipsDO:

• Use exception and several package+class+method names

Google TipsDO:

• Use exception and several package+class+method names• Use quotation marks around individual elements

Google TipsDO:

• Use “site:” to limit search to relevant web sites

Google TipsDO:

• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/user@cassandra.apache.org

Google TipsDO:

• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/user@cassandra.apache.org• JIRA: site:issues.apache.org/jira/browse/CASSANDRA

Google TipsDO:

• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/user@cassandra.apache.org• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra

Google TipsDO:

• Narrow or broaden as necessary

Google TipsDO:

DON’T:

Google TipsDO:

DON’T:

• Include source file + line number

Google TipsDO:

DON’T:

• Include source file + line number• Include specific numbers and string

Exceptions – Search Termsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)

Exceptions – Search Termsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more

Startup

What to look for… What it tells you…

Startup

What to look for… What it tells you…Logging initialized Node just restarted

Startup

What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …

Versions of major components

Startup

Loading settings from file: … Settings file locations

Startup

Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …

Data file and commit log location

Startup

JVM vendor/version: … JVM vendor and version

Startup

JVM vendor/version: … JVM vendor and version• Heap size: … • Par Eden Space Heap memory: … • Par Survivor Space Heap memory: … • CMS Old Gen Heap memory: … • CMS Perm Gen Non-heap memory:…

Heap settings for each generation

Startup

Classpath: … Classpath (jar files and directories)

Startup

Classpath: … Classpath (jar files and directories)JNA mlockall successful JNA is installed (make sure it is!)

Startup

Classpath: … Classpath (jar files and directories)JNA mlockall successful JNA is installed (make sure it is!)• Starting listening for CQL clients on ... • Listening for thrift clients…

Node is serving requests

Flushes

INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)

INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)

INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)

Flushes

Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)

Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]

INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally

INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }

Garbage Collections

INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248

INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248

WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.

You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically

Garbage Collections

Flapping INFO [GossipTasks:1] 2015-02-17 14:38:05,129 Gossiper.java (line 901)

InetAddress /10.98.12.216 is now DOWN

INFO [GossipTasks:1] 2015-02-17 14:38:39,768 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN

INFO [RequestResponseStage:18] 2015-02-17 14:39:14,169 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP

Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)

Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203

INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203

RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)

[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]

INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])

INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98

INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1

INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1

INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157

INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98

INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced

INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully

Bringing it full circle…

1 Ask what changed

5 Share relevant information

6 Ask for help

Thanks!Q&A

cassandra day atlanta 2015: troubleshooting with apache cassandra

helpcompany confidential

logcompany confidential

settingscompany confidential

agenda21 things

cassandra kernelwhat

cassandra kernel jvmwhat

root cause6

deep dive

Technology

pycon 2012 apache cassandra

support apache cassandra in production · anuj wadehra ....

apache cassandra, part 3 – machinery, work with cassandra

apache cassandra en smartpolitech

cassandra day atlanta 2015: building your first application...

cassandra day denver 2014: introduction to apache cassandra

apache cassandra

introduction to cassandra • why spark - apache cassandra |...

apache cassandra cosnola

using apache spark, apache kafka and apache...

apache cassandra at wayin

cassandra summit 2014: apache cassandra on pivotal...

about "apache cassandra"

cassandra day atlanta 2015: software development with apache...

76613613 apache cassandra

devcenter apache cassandra

cassandra day atlanta 2015: feeding solr at large scale with...

conhecendo apache cassandra meetup - conhecendo … · eiti...

introduction to apache cassandra

apache cassandra in bangalore - cassandra internals and...