Cassandra Runbook

NOTE 05 · CAPACITY

Adding nodes to a live cluster

A new node is a stream of data from the nodes that already own it. On a cluster that is already hot, that stream is extra load. The decision is not "can Cassandra add a node." It is whether the ring can absorb one bootstrap without the application falling behind, and in what order.

Before you start the first node

These checks are on the new host, before Cassandra is started. A node that already has data directories, or the wrong version, will not bootstrap cleanly.

  1. Confirm you are on the host you intend to add. Hostname, IP, and rack or availability zone. Adding the wrong machine is a longer outage than a slow stream.
  2. Cassandra version matches the rest of the cluster. Java version matches. A newer minor version on the new node is a mixed-version ring. Do not start it.
  3. Data directory is empty. Hints, commitlog, and saved_caches directories do not already exist, or are empty. This is a new node. Leftover files from a previous install make it look like a restart of an old node.
  4. Every filesystem Cassandra needs is mounted, with space. A node that starts and immediately fills a volume is worse than a node that has not started.
  5. Seeds list does not include this node yet. It can be added to seeds after it is UN, if you use seeds that way. Do not list a node as a seed before it has joined.
  6. auto_bootstrap is true. For vnodes, do not set initial_token. Cassandra will allocate tokens. Setting tokens by hand on a vnode cluster fights the allocation.

One node, then the next rack

Add one node at a time. Wait until it is UN and streaming has finished before you start the next. On a cluster with racks or availability zones, rotate: one node in zone 1, then zone 2, then zone 3, then repeat. Do not fill one zone first. A zone that suddenly holds more tokens than the others changes which nodes serve reads while the rest of the addition is still running.

Two data centers do not change the rule. Finish the rotation in one data center, or keep the same zone order in both, but do not start a dozen nodes at once. Each new node pulls data from the existing owners. On a cluster that is already near capacity, several concurrent bootstraps are how the application falls behind.

While it streams

On the new node, tail system.log. Bootstrap progress, stream errors, and the transition out of joining show up there. From an existing node, the practical check is:

nodetool status --resolve-ip | grep UJ

UJ is Up and Joining. Watch the Load column on that row. It should climb as data arrives. A Load that stays flat for a long time, with no movement in the log, is a stuck stream. A Load that climbs slowly while the source nodes stay healthy is a slow stream. Those are different. nodetool netstats on the new node shows the same streams in more detail if the Load number is ambiguous.

When the UJ grep returns nothing, the node has left the joining state. Stop the log tail. Confirm it from an existing node:

nodetool status --resolve-ip | grep `hostname`

You want UN. That is the signal the node has joined. It is not yet the signal that cleanup has run, or that a repair has checked the ranges.

If the application falls behind during a single bootstrap, the cluster was already short on capacity. That is the reason you are adding nodes. It is also the reason to add them one at a time, and to watch the next one more closely. Cassandra can run maintenance during business hours when the ring has spare capacity. When it does not, a bootstrap is visible to the application. Adding the nodes is the fix. Starting several at once is not.

After the new nodes are UN

Existing nodes still hold copies of data for tokens they no longer own. nodetool cleanup on those nodes drops the extra data. It is disk and CPU work. Run it after the new nodes are stable, not during the next bootstrap, and not on every node at once. One node, or one rack, then the next. Cleanup is how you get the disk space back. It is not required for correctness on the next read. Reads will use the new owners. Cleanup reclaims the old copies.

Repair on your normal schedule after the addition settles. Do not start a full repair while nodes are still bootstrapping.

Done when

Every new node is UN. Streaming has finished. The application latency during the next business day is no worse than before the addition, or better. Cleanup can still be in progress. The addition is not done when the first node reaches UN. It is done when the capacity you added is actually serving traffic and the extra copies are being removed on a schedule you control.

Do not

Token allocation and the exact streaming output differ by version. Vnodes are the common case. Single-token clusters need an explicit token plan before the first start. Confirm against the version you run. This is a decision aid, not a procedure for a specific cluster.

Back to notes