Cassandra Runbook

NOTE 06 · STARTUP

Startup aborted by a compaction transaction log

Cassandra will not start if a compaction transaction log says a set of SSTables should be in one state and the files on disk disagree. The usual cause is a compaction that was interrupted, often during an upgrade or a crash. The fix is local to the node. Do not wipe the table directory.

What you are looking at

On startup, Cassandra reads the transaction logs under the table directory. A compaction log records which SSTables were inputs and which were outputs. If the log has a COMMIT but one of the old SSTables has a different modification time than the log expects, Cassandra treats the disk as inconsistent and aborts. The log includes the transaction contents. The line that matters is the one marked unexpected:

Unexpected disk state: failed to read transaction log [bb_txn_compaction_<uuid>.log in /var/lib/cassandra/data/<keyspace>/<table>-<id>]
Files and contents follow:
...
REMOVE:[.../bb-<generation>-bti-,<mtime>,20][<size>]
***Unexpected files detected for sstable [bb-<generation>-bti-]: last update time [21:59:22] should have been [22:03:20]
...
COMMIT:[,0,0][<timestamp>]

The node is down. The other replicas are still serving. This is not a cluster-wide schema or streaming problem. The ADD lines are the new SSTables the compaction produced. The REMOVE lines are the old ones it intended to delete. One of those old files has an mtime that does not match the log, so Cassandra will not finish the cleanup.

Confirm it

  1. The node is stopped. Do not start it again until the files are handled. A second start will hit the same check.
  2. In the table directory named in the log, find the transaction log file and the SSTable it names. The SSTable is a set of files with the same generation prefix (bb-4240389-bti-Data.db, -Statistics.db, and the rest).
  3. Check that the other SSTables for the table are present and that the data directory is otherwise intact. You are removing one generation and one log, not the table.

Backup, then remove, then start

With the node still stopped, copy the mismatched SSTable files and the transaction log to a directory outside the data path. A temporary folder on the same host is enough. You are keeping a copy in case the removal was wrong.

Then remove, from the table directory only:

Leave every other SSTable and every other transaction log. Start Cassandra normally. If startup succeeds, the transaction log is no longer blocking. The node will rejoin and stream or repair as needed to cover the data that was in the removed SSTable.

After the node is UN, run a repair of the affected table, or let your normal repair schedule cover it. A primary-range repair of that table is enough if you want to force it:

nodetool repair -pr <keyspace> <table>

The removed SSTable was an input to a compaction that did not finish cleanly. The other replicas still have the data. Repair fills this node. You do not need to restore the files you set aside unless startup fails for a different reason.

Done when

The node starts and reaches UN. The error does not reappear in the log. A repair of the table has been run, or is on the normal schedule. The backup copy of the removed files can be deleted after the node has been up and repaired.

Do not

Transaction log names and the exact timestamp check differ by version. The pattern is the same: a compaction log and an SSTable that disagree, on a node that will not start. Confirm the file names against the log on the node you are fixing. This is a decision aid, not a procedure for a specific cluster.

Back to notes