Hints accumulating between data centers
Hints piling up on some nodes, while others in the same data center drain, usually means those coordinators are timing out writes to another data center. The cluster can be fully up. The difference is latency, not a partition.
What you are looking at
Hint files are growing on disk in one or two data centers. Dropped messages are elevated, often on the far side. nodetool status shows every node UN. nodetool describecluster sometimes reports a node unreachable, and the view is not the same from every node. A minute later the same command is clean. Direct TCP to ports 7000 and 7001 succeeds. Newer data centers are not accumulating hints. The older ones are.
This is not the single-node hint pile from a node that was down. The target nodes are up. The coordinators that keep storing hints are the ones with higher latency to the other data center.
Confirm it
nodetool statuson several nodes. AllUNis expected. ADNnode is a different problem.nodetool describeclusterfrom more than one node, more than once. Intermittent "unreachable" that clears on its own, and that not every node agrees on, is gossip lag under load. It is not a stable partition.- From a coordinator that is still accumulating hints, measure latency to nodes in the other data center. Ping is enough to see a split. A slower response time on the cross-DC ping, compared with the other nodes in the same data center, is a symptom. The nodes that keep accumulating hints are often the ones with the higher latency. The nodes that drain are the lower-latency ones.
- Check
max_hint_window_in_ms. A long window (hours or days) means the files will keep growing until the timeouts stop. It does not cause the timeouts. - TCP connectivity on 7000/7001 working rules out a simple firewall block. It does not rule out loss, retransmits, or a path that is slower for some hosts.
Immediate relief, then the network
A restart of the coordinators that are accumulating hints clears stuck messaging state. After the restart, describecluster usually stays clean, dropped messages fall, and most hint backlogs begin to drain. A gossip bounce alone often does not. The restart is relief. It is not the fix if the latency split is still there.
After the restart, watch which nodes drain and which do not. If two nodes in the same data center keep accumulating hints while the others drain, and those two have higher latency to the far data center, the pattern is a feedback loop. Application writes plus hint replay increase load on the far side, latency rises, the slower coordinators time out more writes, and they store more hints. The cluster is not partitioned. It is working through a backlog under uneven cross-DC latency.
Hand the latency split to the networking team. Ping from each node in the affected data center to the other, and note which hosts are slower. Interface counters, TCP retransmits, and a path trace are the next measurements. iperf3 is useful if it is allowed. Do not change phi_convict_threshold, write timeouts, or the hint throttle to hide the symptom unless you have a measured reason and a change window. Those settings mask the latency. They do not remove it.
Done when
Hint directories on the affected coordinators are shrinking, not growing. Dropped messages on the far data center have returned to the normal rate. describecluster is stable from the nodes that were flapping. The latency split has been identified and is with the networking team, or has been corrected. A restart that made the graphs look better for an hour, while the slow nodes are still slow, is not done.
Do not
- Treat intermittent describecluster "unreachable" as a partition when status is
UNeverywhere and TCP succeeds. Partition means nodes that stay down or that cannot reach each other. - Bounce every node in the data center because two of them are slow. Restart the coordinators that are accumulating hints. Leave the ones that are draining alone.
- Truncate hints while the latency split remains. The files will return as soon as the next writes time out.
- Raise timeouts or the hint window so the errors stop. A longer window stores more hints. A longer timeout hides the latency the application is already feeling.
- Assume the newer data centers are the cause because they were added recently. In this pattern they were not accumulating hints. The path between the older data centers was.
Cross-DC latency and hint accumulation interact. The commands above are the long-standing nodetool set. What counts as slow will differ by network. Confirm the split on your hosts before you call it a network problem. This is a decision aid, not a procedure for a specific cluster.