Cassandra Runbook

FOR OPERATORS

What to check when the cluster is already in trouble.

Short notes from production incidents. Each one is a decision you can make on a live ring: what the error actually means, which node to test, and what not to run.

NODE DOWN

Repair, or replace-address?

Three different operations. Start the same node if the disks survived. Replace only if the host is gone and you still want those tokens. Remove only if nothing is coming back.

Read the decision tree

SAI INDEX

ReadFailure: INDEX_NOT_AVAILABLE

An indexed query failed because a replica could not serve its storage-attached index. Test each node at LOCAL_ONE, then rebuild the index only where it failed.

Read the check order

READ PATH

Tombstones overwhelming a read

The table is up. The query scanned deletion markers until Cassandra warned or failed. Bound the read, then compact only the table the log named.

Read the check order

HINTS

Hints piling up

A coordinator is storing writes a replica did not acknowledge. Fix the down node first. Pause replay if the recovered node is falling over. Truncate only after those writes are safe.

Read the check order

CAPACITY

Adding nodes to a live cluster

A new node pulls data from the nodes that already own it. On a hot ring, add one at a time, rotate across availability zones, and watch the stream before you start the next.

Read the check order

STARTUP

Startup aborted by a compaction transaction log

Cassandra will not start if a compaction log and an SSTable disagree on timestamps. Backup the mismatched files, remove them, start the node, then repair the table.

Read the check order

HINTS / NETWORK

Hints accumulating between data centers

Hints pile up on some coordinators while others drain, and describecluster flaps. The cluster is up. The difference is cross-DC latency on specific nodes.

Read the check order

SAI INDEX

ReadFailure from a large IN on an SAI column

A large IN against an SAI-indexed clustering column, with no partition key, makes every replica do expensive index work. The index is up. The query is the problem. Do not rebuild it.

Read the check order

CPU / STREAMING

High CPU from Stream-Serializ threads

A node stays hot after a repair or topology change. Compaction and netstats are quiet. The hot threads are Stream-Serializ. Wait if it is still progressing. Restart if it is stuck.

Read the check order

When a new note is up

One email when a runbook page is added. No course, no digest.