Repair, or replace-address?
Three different operations. Start the same node if the disks survived. Replace only if the host is gone and you still want those tokens. Remove only if nothing is coming back.
Short notes from production incidents. Each one is a decision you can make on a live ring: what the error actually means, which node to test, and what not to run.
Three different operations. Start the same node if the disks survived. Replace only if the host is gone and you still want those tokens. Remove only if nothing is coming back.
An indexed query failed because a replica could not serve its storage-attached index. Test each node at LOCAL_ONE, then rebuild the index only where it failed.
The table is up. The query scanned deletion markers until Cassandra warned or failed. Bound the read, then compact only the table the log named.
A coordinator is storing writes a replica did not acknowledge. Fix the down node first. Pause replay if the recovered node is falling over. Truncate only after those writes are safe.
A new node pulls data from the nodes that already own it. On a hot ring, add one at a time, rotate across availability zones, and watch the stream before you start the next.
Cassandra will not start if a compaction log and an SSTable disagree on timestamps. Backup the mismatched files, remove them, start the node, then repair the table.
Hints pile up on some coordinators while others drain, and describecluster flaps. The cluster is up. The difference is cross-DC latency on specific nodes.
A large IN against an SAI-indexed clustering column, with no partition key, makes every replica do expensive index work. The index is up. The query is the problem. Do not rebuild it.
A node stays hot after a repair or topology change. Compaction and netstats are quiet. The hot threads are Stream-Serializ. Wait if it is still progressing. Restart if it is stuck.
One email when a runbook page is added. No course, no digest.