Repair, or replace-address?
A down node is one of three operations. Pick from what is left of the host, not from the word "down." Starting the old node, replacing it, and removing it are different commands. Mixing them leaves the range streaming twice, or not at all.
The three operations
| Operation | When | What it does |
|---|---|---|
| Start the same node, then repair | Same host, same IP, disks intact, downtime short | Keeps the data already on disk. Hints replay. Repair fills what hints missed. |
| Replace | Host or disk is gone, and you want those tokens back on a new node | New node streams a fresh copy from the other replicas. It does not read the dead disks. |
| Remove | You are not bringing that node back | Live nodes take over the ranges. No new node is joining. |
Decide in this order
- Look at the host, not the gossip state.
DNonly means Cassandra is not up. It does not say whether the disks survived. - Can this same machine boot with the same IP and the same data directories? If yes, and it has been down for less than the hint window (default three hours,
max_hint_window_in_ms), start it. Do not replace it. - Is the machine or the disk gone, and do you still want those tokens in the ring? Replace. Do not
removenodefirst. Replace only works while the dead node is still listed as down. - Are you retiring the node and not putting one back? Remove it from a live node. Do not also set a replace address.
- Before either replace or remove: other replicas for those tokens must be
UN. One replacement or removal at a time. No second stream in the same token neighborhood.
Start it and repair
Use this when the process died and the data directories are fine. Find out why it stopped before you call the node healthy. A full disk or a bad disk will kill it again.
- Start Cassandra on that host. Same IP. Do not set a replace property.
- Confirm it reaches
UN. Hint replay starts on its own if the downtime was inside the hint window. - Repair the ranges this node owns, not the whole cluster, unless your normal schedule is already a full repair. On Cassandra 4 and later a targeted repair is enough. Watch
nodetool netstatsand pending compactions. - Done when the node stays
UNand that repair finishes. A restart is not the proof.
Replace it
Use this when a new machine must take the dead node's tokens. The new node must run the same major version. Take the dead IP out of the seed list on every node before you depend on the new one. Leave the dead node in gossip as DN. removenode first makes replace impossible.
- From a live node, record the dead IP and confirm the other replicas are
UN. - On the new node only, set the replace property for the first boot. Cassandra 4 and later uses
-Dcassandra.replace_address_first_boot=. Older versions use-Dcassandra.replace_address=. Do not set both. Do not setauto_bootstrap: false. - Start the new node once. It streams from the remaining replicas. Watch
nodetool netstatson the new node. - After it is
UN, remove the replace property. A second start with that property still set is a mistake. - Repair on your normal schedule after streaming finishes. Do not run a ring repair while this stream is the only copy being built.
Remove it, and stop
Use this only if nothing is going to take that node's place. From a live node, nodetool removenode . The host id comes from nodetool status, not from the IP. If gossip cannot see the node and removenode sits forever, that is a different problem. Do not assassinate a node you still plan to replace.
Do not
- Run
removenodeand then tryreplace_address. Remove takes the node out of the ring. Replace needs it still listed as down. - Replace a node whose disks are fine and whose downtime is inside the hint window. You would throw away data you could have started.
- Replace two nodes that share token ranges at the same time. The second stream may have nowhere correct to read from.
- Leave the replace property in place after the first successful boot.
- Treat
UNas proof the range is consistent. Streaming finished means the new node has a copy. Repair is what checks it.
The property name changed in Cassandra 4.0, from replace_address to replace_address_first_boot. Confirm the flag against the version you run. This is a decision aid, not a procedure for a specific cluster.