Cassandra Runbook

NOTE 04 · HINTS

Hints piling up

Hints are the coordinator's copy of a write that a replica did not acknowledge. A pile of them means some node was down or too slow. The hints are the symptom. The down node is the problem. Deleting the symptom first can drop the only copy of those mutations.

What you are looking at

On a write, the coordinator stores a hint when a replica misses the ack and the miss is inside max_hint_window_in_ms. The default window is three hours. Hints live on the coordinator, under the hints directory in the data path, not on the node that was down. When that node comes back, coordinators replay the hints at it.

Two failures follow. Coordinators fill a disk with hint files. Or the node comes back and hint replay, plus the normal write load, stalls it again.

Confirm it

  1. On a coordinator that is growing disk, check the hints directory. The data path is often /var/lib/cassandra/hints. A directory that keeps growing is the pile. Snapshots and commitlog are different directories. Do not clear those because hints are large.
  2. nodetool tpstats and look at the HintedHandoff row. It is in the pool table, not a separate command:
Pool Name                    Active   Pending   Completed   Blocked  All time blocked
HintedHandoff                     1      1842     3904411         0                 0

Pending is the count to watch. Falling means replay is catching up. Climbing, or Blocked and Dropped rising, means the coordinator is behind or refusing work. On some versions the row is named HintDispatcher or sits under a Hints section. The pending number is the same idea.

  1. nodetool status. The usual cause is a DN node, or a node that just returned to UN and is now the target of every coordinator's replay.
  2. Note how long that node was down against the hint window. Past the window, coordinators stop storing new hints and expired hints will not replay. Inside the window, the files may be the only extra copy of writes the down replica missed.

Fix the target first

Bring the down replica back if the host and disks are fine. That is the start-and-repair path, not a replace. Hints replay on their own once it is reachable. If the host is gone, replace or remove it before you spend time on the hint files. Replaying at a node that will never return does nothing.

If the node is back and hints are falling, leave it

The common case needs no command. The node that was down is UN again, it is staying up, and the hint counts on the coordinators are going down. Slow is still working. Hint replay is throttled on purpose so it does not bury the node that just returned. A drain that takes hours after a multi-hour outage is normal, especially when many coordinators are replaying at once.

Watch HintedHandoff pending and the size of the hints directory. If both are falling and the recovered node stays UN with ordinary read and write latency, stop. Do not pause handoff, do not lower the throttle, do not truncate, and do not bounce the coordinators to make a graph look better. Those are for the cases below, where replay is hurting the node or the files will never be delivered. Intervening on a healthy drain only delays the catch-up.

When replay is the problem

Use these only if the recovered node is not staying healthy. It is flapping, latency is climbing, or pending hints are not falling. Pause handoff on the coordinators that are sending, not on the recovered node. The hint files are on the nodes that accepted the writes. Replay is those nodes pushing mutations at the replica that missed them. pausehandoff only stops the node you run it on.

nodetool pausehandoff

Run that on each coordinator whose hints directory is large or whose HintedHandoff pending is climbing. Let the recovered node become stable. Compaction and reads should settle. Then resume on those same coordinators:

nodetool resumehandoff

If resume floods it again, lower the throttle on the coordinators with nodetool sethintedhandoffthrottlekb, then resume. Pausing is a delay. It is not a delete. disablehandoff stops new hints from being stored. Use that only while you are in the incident, and turn it back on with enablehandoff. A cluster with handoff left disabled drops the safety net for the next outage.

Truncate only after the writes are safe

Truncating throws away stored hints on that coordinator. It does not repair anything.

nodetool truncatehints 10.1.2.21

That form drops hints for one endpoint. Omit the address and you drop every hint on this coordinator. Do it only in one of these cases:

Run it on each coordinator that holds a pile, not once for the cluster. Hints are local files.

Done when

The node that was down stays UN. Hint directories stop growing. HintedHandoff pending falls and does not climb again. A truncated directory with the target still DN is not done. You removed the warning and left the outage.

Do not

Hint file layout and the exact tpstats names differ by version. The commands above are the long-standing nodetool set. Confirm them against the version you run. This is a decision aid, not a procedure for a specific cluster.

Back to notes