Hints piling up
Hints are the coordinator's copy of a write that a replica did not acknowledge. A pile of them means some node was down or too slow. The hints are the symptom. The down node is the problem. Deleting the symptom first can drop the only copy of those mutations.
What you are looking at
On a write, the coordinator stores a hint when a replica misses the ack and the miss is inside max_hint_window_in_ms. The default window is three hours. Hints live on the coordinator, under the hints directory in the data path, not on the node that was down. When that node comes back, coordinators replay the hints at it.
Two failures follow. Coordinators fill a disk with hint files. Or the node comes back and hint replay, plus the normal write load, stalls it again.
Confirm it
- On a coordinator that is growing disk, check the hints directory. The data path is often
/var/lib/cassandra/hints. A directory that keeps growing is the pile. Snapshots and commitlog are different directories. Do not clear those because hints are large. nodetool tpstatsand look at theHintedHandoffrow. It is in the pool table, not a separate command:
Pool Name Active Pending Completed Blocked All time blocked
HintedHandoff 1 1842 3904411 0 0
Pending is the count to watch. Falling means replay is catching up. Climbing, or Blocked and Dropped rising, means the coordinator is behind or refusing work. On some versions the row is named HintDispatcher or sits under a Hints section. The pending number is the same idea.
nodetool status. The usual cause is aDNnode, or a node that just returned toUNand is now the target of every coordinator's replay.- Note how long that node was down against the hint window. Past the window, coordinators stop storing new hints and expired hints will not replay. Inside the window, the files may be the only extra copy of writes the down replica missed.
Fix the target first
Bring the down replica back if the host and disks are fine. That is the start-and-repair path, not a replace. Hints replay on their own once it is reachable. If the host is gone, replace or remove it before you spend time on the hint files. Replaying at a node that will never return does nothing.
If the node is back and hints are falling, leave it
The common case needs no command. The node that was down is UN again, it is staying up, and the hint counts on the coordinators are going down. Slow is still working. Hint replay is throttled on purpose so it does not bury the node that just returned. A drain that takes hours after a multi-hour outage is normal, especially when many coordinators are replaying at once.
Watch HintedHandoff pending and the size of the hints directory. If both are falling and the recovered node stays UN with ordinary read and write latency, stop. Do not pause handoff, do not lower the throttle, do not truncate, and do not bounce the coordinators to make a graph look better. Those are for the cases below, where replay is hurting the node or the files will never be delivered. Intervening on a healthy drain only delays the catch-up.
When replay is the problem
Use these only if the recovered node is not staying healthy. It is flapping, latency is climbing, or pending hints are not falling. Pause handoff on the coordinators that are sending, not on the recovered node. The hint files are on the nodes that accepted the writes. Replay is those nodes pushing mutations at the replica that missed them. pausehandoff only stops the node you run it on.
nodetool pausehandoff
Run that on each coordinator whose hints directory is large or whose HintedHandoff pending is climbing. Let the recovered node become stable. Compaction and reads should settle. Then resume on those same coordinators:
nodetool resumehandoff
If resume floods it again, lower the throttle on the coordinators with nodetool sethintedhandoffthrottlekb, then resume. Pausing is a delay. It is not a delete. disablehandoff stops new hints from being stored. Use that only while you are in the incident, and turn it back on with enablehandoff. A cluster with handoff left disabled drops the safety net for the next outage.
Truncate only after the writes are safe
Truncating throws away stored hints on that coordinator. It does not repair anything.
nodetool truncatehints 10.1.2.21
That form drops hints for one endpoint. Omit the address and you drop every hint on this coordinator. Do it only in one of these cases:
- The target has been down longer than the hint window, the hints are already expired, and disk is the emergency. You are deleting files that were not going to replay.
- The target is back, a repair of its ranges has finished, and you have decided the repair is the copy you trust. Truncate is cleanup, not the repair.
- You accept losing writes that no other replica acknowledged. At RF 3 and
LOCAL_QUORUM, a hint is often the third copy, not the only copy. At RF 1, or if the other replicas also missed the write, it may be the only copy.
Run it on each coordinator that holds a pile, not once for the cluster. Hints are local files.
Done when
The node that was down stays UN. Hint directories stop growing. HintedHandoff pending falls and does not climb again. A truncated directory with the target still DN is not done. You removed the warning and left the outage.
Do not
- Delete the hints directory out from under a running node. Use
truncatehints. A removed directory can come back as a replay error or a restart problem. - Truncate hints for a node that has been down for twenty minutes and is your only missing replica. Those files may be the writes the other copies do not have.
- Raise
max_hint_window_in_msduring the incident to "stop the pile." A longer window stores more hints. It does not drain them. - Leave handoff disabled after the incident.
- Pause, throttle, or truncate a drain that is already falling on a node that is staying
UN. Slow catch-up is the normal path. - Bounce the coordinator nodes unnecessarily while hints are being handed off. A coordinator node restart interrupts hint handoff that is already running. There is no operational gain.
- Pause handoff on the node that just came back. That node is the target. Pausing it does not stop the other nodes from replaying hints at it.
Hint file layout and the exact tpstats names differ by version. The commands above are the long-standing nodetool set. Confirm them against the version you run. This is a decision aid, not a procedure for a specific cluster.