High CPU from Stream-Serializ threads
A node stays at high CPU after a repair or a topology change. Compaction and netstats are quiet. The hot threads are Stream-Serializ. This is residual work from streaming, not a new failure.
What you are looking at
The Cassandra process is the top consumer, often 80โ100% on a small core count. nodetool compactionstats and nodetool netstats show nothing active. top -H on the process shows Stream-Serializ or Stream-Deserial as the hottest threads. Logs from the same period may still mention Merkle trees or repair. Cluster status is otherwise healthy. No error is required for this to happen.
The main streaming sessions can finish and disappear from netstats while the serializer threads continue finishing Merkle trees or cleanup. On eight cores or fewer the residual work is obvious as high overall CPU and elevated load average. On a larger machine it can be one hot thread among many and easier to miss.
Confirm it
- On the hot node, identify the threads:
Sort by CPU.top -H -p $(pgrep -f CassandraDaemon | head -1)Stream-SerializorStream-Deserialas the top entries is the signature. - Re-check the usual suspects. Empty or near-empty is expected here:
nodetool compactionstats nodetool netstats -H nodetool tpstats - Look at recent log activity for the same period:
Ongoing Merkle-tree sends or late repair messages confirm the source.grep -E "merkle|Merkle|STREAM|repair" /var/log/cassandra/system.log | tail -30 - Note the core count with
lscpuornproc. Fewer cores make the residual work visible as high overall CPU.
What to do
- If Merkle or repair log lines are still advancing and client latency is acceptable, wait 10โ30 minutes and re-check
top -H. Many cases quiet down once the remaining trees are sent. - If the threads stay pegged, the logs show no progress, or applications are affected, restart the Cassandra process on that node. The residual threads die with the JVM. CPU drops immediately.
- If you need to reduce impact without a restart, temporarily lower outbound stream throughput:
Use a lower number if 50 is still too much. Restore the previous value after the node settles. Do not leave it lowered permanently unless you have measured that the default is too aggressive for the hardware.nodetool setstreamthroughput 50
After a restart, confirm the node is UN, netstats is empty, and top -H no longer shows hot Stream-* threads. Overall CPU should return to the normal range for the workload.
Done when
The Stream-Serializ threads are gone or idle and node CPU is back to its baseline. Cluster status remains healthy. If the high CPU followed a data-center addition or a large repair, verify the new nodes are UN and that no large streaming or compaction backlog remains on them.
Do not
- Treat an empty
netstatsas proof nothing streaming-related is still running. The serializer threads can outlive the session visibility. - Run additional repairs or cleanups on the hot node while the threads are pegged. You add load to an already saturated process.
- Assume a permanent bug or a stuck thread that requires more than a restart. In the common case after a repair or a topology change, a restart clears it.
- Leave a lowered
streamthroughputin place permanently unless you have measured that the default is too aggressive for the hardware.
Streaming and Merkle-tree behavior differ slightly by version. The thread names and the pattern (high CPU, empty netstats, recent topology or repair work) are consistent. This is a decision aid, not a procedure for a specific cluster.