Repair, or replace-address?
A node that is down is a choice, not a command. Repair assumes the data already on that node is worth keeping. Replace assumes it is cheaper to stream a fresh copy and move on.
Decide in this order
- Is the process down, or is the instance gone? A stopped service on a healthy disk is not the same problem as a dead VM.
- How long has it been out? If it missed a long window of writes, hint replay and incremental repair may cost more than a replace.
- Is the token range already covered? Check that the other replicas are up before you remove anything.
- Are other nodes already streaming or repairing? One replacement at a time. Do not stack them.
| Situation | Lean toward |
|---|---|
| Process crashed, disk is intact, downtime is short | Start it and repair |
| Disk is suspect, or the host will not come back | Replace |
| Node has been down longer than your hint window, and the data on it is stale | Replace |
| Several nodes in the same rack are unhealthy | Stop. Restore coverage before any replace |
If you repair
Bring the node up only after you know why it stopped. Then run a targeted repair for the ranges it owns, not a full-cluster repair, unless you already have a repair schedule that expects one. Watch streaming and pending compactions before you call it done.
If you replace
Bootstrap a new node with the same token ownership intent. On DataStax Enterprise and Cassandra this is the replace_address / replace_address_first_boot path, set only on the first boot of the replacement. The new node streams data from the remaining replicas. It does not trust the dead node's disks.
Remove the old host from monitoring and from any seed list before you depend on the new one. Do not run repair across the ring while that stream is the only thing catching the range up.
Do not
- Run
removenodeandreplacenodeas if they were the same operation. - Replace two nodes in the same token neighborhood at once.
- Treat a successful bootstrap as proof the range is consistent. Check streaming completion, then repair on your normal schedule.
This is a decision aid, not a procedure for a specific cluster. Versions differ on the exact property name and on whether a node can still be replaced after it has been removed. Confirm against the docs for the version you run.