Data-parallel training is embarrassingly parallel right up until the coordinator dies.
The shape is familiar. Shard the data across workers, each computes gradients on its shard, and a coordinator — the parameter server — averages them and applies the update. Workers are replaceable: if one dies, reschedule its shard. The coordinator is not. It holds the only copy of the model everyone else is agreeing about. Kill it deep into a run and you start again from step zero.
Many-As-One, the storage and compute stack I've been building, ends with exactly that problem: a data-parallel logistic-regression trainer sitting on top of a Raft-backed key-value store. The interesting question was never "can I run Raft". It was what actually deserves to go through consensus.
Gradients don't. Checkpoints do.
The tempting answer is "replicate everything". It's also the wrong one, and the reason is throughput.
A Raft commit costs a round trip to a majority of the cluster before the leader can apply it. That's fine for something that happens occasionally. It is ruinous for something that happens on every step from every worker. Gradients arrive constantly, they're only useful for one update, and they're cheap to recompute — the worker still has its shard, so a lost gradient costs one forward and backward pass.
Model checkpoints are the opposite. They're periodic, and they represent all the compute spent so far. Losing one is expensive precisely because you cannot cheaply recompute it.
So the rule I settled on: put the state you can't cheaply recompute through consensus, and let everything else be best-effort. Gradients flow over plain gRPC to whoever is currently coordinating. Checkpoints go into the Raft key-value store, which means a majority of nodes has them before the write is acknowledged.
A commit is not an acknowledgement
Raft's guarantee is narrower than people assume, and the gap is where the bugs live.
When a leader appends an entry, it isn't committed yet. It's committed once it's replicated to a majority and the leader has advanced its commit index. Only then may it be applied to the state machine and reported to the client. The two facts that matter for a trainer:
- If the leader crashes after appending but before that majority exists, the entry can be silently dropped by whoever wins the next election. Any "yes, saved" you returned before that point was a lie.
- If the leader crashes after the entry is committed but before it replies, the write happened and your client has no idea.
The first is why you never acknowledge a checkpoint before commit. The second is why acknowledgement alone can't be your protocol.
Idempotent retries are the real failover story
The second case above is the one that actually shows up in practice. The coordinator writes a checkpoint, the leader dies mid-flight, the client sees a broken connection, and now it has to decide: did that land or not?
Retrying blindly can double-apply. Not retrying can lose the run. The way out is to make the write carry its own identity — a request id, and a compare-and-swap against the version the coordinator believes is current:
put(key="model/checkpoint", value=blob,
expected_version=41, # CAS: only if nobody else moved it
request_id="coordinator-7:step-40000")
Now the retry is safe. If the first attempt committed, the store recognises the request id and returns the same result instead of applying it twice. If it didn't, the CAS still matches and the write goes through. If a different coordinator has since taken over and written its own checkpoint, the CAS fails — which is the correct outcome, because this coordinator is no longer the one in charge.
That combination — idempotent retries plus compare-and-swap — is what makes "resume after a coordinator crash" a property of the system rather than a hope.
Reads can lie
A subtler failure: the new coordinator starts up, reads the last checkpoint, and gets a stale one.
This happens because a leader that has been partitioned away doesn't know it yet. It still believes it's the leader, and it will happily serve a read from its own state — state that a newer leader has since moved past. Nothing in the log is corrupted; the reader simply asked a node whose worldview expired.
The fix is a ReadIndex barrier: before serving a linearizable read, the leader confirms with a heartbeat round that a majority still considers it the leader, then serves the read at that commit index. It costs a round trip.
Which means it's a choice, not a default. Resuming training reads the checkpoint exactly once, so paying a round trip for certainty is free in practice. A dashboard polling training progress every second can take the stale read and be happier for it. My store makes the barrier optional per read for that reason.
The log grows, so snapshot it
Every checkpoint is an entry, and model blobs are not small. A log that keeps every checkpoint forever means a restarting node replays the entire training history to catch up, and a lagging follower may need entries the leader has already discarded.
Snapshotting is the answer: capture the state machine as it stands, keep the log after that point, and give a sufficiently-behind follower the snapshot instead of the entries. It's the piece that turns a working Raft implementation into one that survives a long run.
The bug worth keeping
I hit a lost write in my own commit path: a write the system had acknowledged that wasn't visible after failover. Exactly the class of bug this whole post is about, in the code that was supposed to prevent it.
What made it findable was writing a test that controls the interleaving rather than the clock. Sleep-and-hope tests pass on a fast laptop and fail in CI, and they don't tell you what happened. A hermetic test that drives the nodes through one specific sequence — append here, crash there, elect this node — either reproduces the bug every run or doesn't. Mine now does, and it's the test I'd point at first if someone asked what I learned building this.
Consensus code is easy to write and hard to prove. The failure modes hide in orderings you didn't imagine, which is precisely why the test has to name the ordering.
What this isn't
This is logistic regression across a handful of processes, not a foundation model across a cluster. At real scale the shape changes: checkpoints go to object storage and only the metadata — the pointer, the version, the step number — goes through consensus, because pushing multi-gigabyte blobs through a replicated log is a poor use of a replicated log.
But the reasoning transfers, and it's the part worth keeping. Ask what you cannot recompute, pay for consensus only there, make every write idempotent, and treat "acknowledged" and "committed" as two different words.