Cassandra Runbook

NOTE 01 · NODE DOWN

Repair, or replace-address?

A down node is one of three operations. Pick from what is left of the host, not from the word "down." Starting the old node, replacing it, and removing it are different commands. Mixing them leaves the range streaming twice, or not at all.

The three operations

OperationWhenWhat it does
Start the same node, then repairSame host, same IP, disks intact, downtime shortKeeps the data already on disk. Hints replay. Repair fills what hints missed.
ReplaceHost or disk is gone, and you want those tokens back on a new nodeNew node streams a fresh copy from the other replicas. It does not read the dead disks.
RemoveYou are not bringing that node backLive nodes take over the ranges. No new node is joining.

Decide in this order

  1. Look at the host, not the gossip state. DN only means Cassandra is not up. It does not say whether the disks survived.
  2. Can this same machine boot with the same IP and the same data directories? If yes, and it has been down for less than the hint window (default three hours, max_hint_window_in_ms), start it. Do not replace it.
  3. Is the machine or the disk gone, and do you still want those tokens in the ring? Replace. Do not removenode first. Replace only works while the dead node is still listed as down.
  4. Are you retiring the node and not putting one back? Remove it from a live node. Do not also set a replace address.
  5. Before either replace or remove: other replicas for those tokens must be UN. One replacement or removal at a time. No second stream in the same token neighborhood.

Start it and repair

Use this when the process died and the data directories are fine. Find out why it stopped before you call the node healthy. A full disk or a bad disk will kill it again.

  1. Start Cassandra on that host. Same IP. Do not set a replace property.
  2. Confirm it reaches UN. Hint replay starts on its own if the downtime was inside the hint window.
  3. Repair the ranges this node owns, not the whole cluster, unless your normal schedule is already a full repair. On Cassandra 4 and later a targeted repair is enough. Watch nodetool netstats and pending compactions.
  4. Done when the node stays UN and that repair finishes. A restart is not the proof.

Replace it

Use this when a new machine must take the dead node's tokens. The new node must run the same major version. Take the dead IP out of the seed list on every node before you depend on the new one. Leave the dead node in gossip as DN. removenode first makes replace impossible.

  1. From a live node, record the dead IP and confirm the other replicas are UN.
  2. On the new node only, set the replace property for the first boot. Cassandra 4 and later uses -Dcassandra.replace_address_first_boot=. Older versions use -Dcassandra.replace_address=. Do not set both. Do not set auto_bootstrap: false.
  3. Start the new node once. It streams from the remaining replicas. Watch nodetool netstats on the new node.
  4. After it is UN, remove the replace property. A second start with that property still set is a mistake.
  5. Repair on your normal schedule after streaming finishes. Do not run a ring repair while this stream is the only copy being built.

Remove it, and stop

Use this only if nothing is going to take that node's place. From a live node, nodetool removenode . The host id comes from nodetool status, not from the IP. If gossip cannot see the node and removenode sits forever, that is a different problem. Do not assassinate a node you still plan to replace.

Do not

The property name changed in Cassandra 4.0, from replace_address to replace_address_first_boot. Confirm the flag against the version you run. This is a decision aid, not a procedure for a specific cluster.

Back to notes