Cassandra Runbook

NOTE 09 ยท CPU / STREAMING

High CPU from Stream-Serializ threads

A node stays at high CPU after a repair or a topology change. Compaction and netstats are quiet. The hot threads are Stream-Serializ. This is residual work from streaming, not a new failure.

What you are looking at

The Cassandra process is the top consumer, often 80โ€“100% on a small core count. nodetool compactionstats and nodetool netstats show nothing active. top -H on the process shows Stream-Serializ or Stream-Deserial as the hottest threads. Logs from the same period may still mention Merkle trees or repair. Cluster status is otherwise healthy. No error is required for this to happen.

The main streaming sessions can finish and disappear from netstats while the serializer threads continue finishing Merkle trees or cleanup. On eight cores or fewer the residual work is obvious as high overall CPU and elevated load average. On a larger machine it can be one hot thread among many and easier to miss.

Confirm it

  1. On the hot node, identify the threads:
    top -H -p $(pgrep -f CassandraDaemon | head -1)
    Sort by CPU. Stream-Serializ or Stream-Deserial as the top entries is the signature.
  2. Re-check the usual suspects. Empty or near-empty is expected here:
    nodetool compactionstats
    nodetool netstats -H
    nodetool tpstats
  3. Look at recent log activity for the same period:
    grep -E "merkle|Merkle|STREAM|repair" /var/log/cassandra/system.log | tail -30
    Ongoing Merkle-tree sends or late repair messages confirm the source.
  4. Note the core count with lscpu or nproc. Fewer cores make the residual work visible as high overall CPU.

What to do

After a restart, confirm the node is UN, netstats is empty, and top -H no longer shows hot Stream-* threads. Overall CPU should return to the normal range for the workload.

Done when

The Stream-Serializ threads are gone or idle and node CPU is back to its baseline. Cluster status remains healthy. If the high CPU followed a data-center addition or a large repair, verify the new nodes are UN and that no large streaming or compaction backlog remains on them.

Do not

Streaming and Merkle-tree behavior differ slightly by version. The thread names and the pattern (high CPU, empty netstats, recent topology or repair work) are consistent. This is a decision aid, not a procedure for a specific cluster.

Back to notes