M37

What a database keeps when the machine dies

This is the material behind the video. Four database systems were run in thirteen arms — ten database configurations and three controls — and killed forty times, in two ways that are not the same thing: ten kills of the database process alone, and thirty kills of the whole machine. After every kill, every write the database had already told its client was committed was checked against what actually came back. That produced 500 trial-arm outcomes, and all of them are below.

The protocol came first, and it is on this page in full rather than summarised. It was committed on 2026-09-01 at 22:55 Berlin time, commit e4f5d4f3; the first trial data was committed 00:01 the following morning, commit 341ed1e3. The repository is private, so that is a statement rather than a link, and it is the one thing on this page you cannot check for yourself. Everything downstream of it you can.

The protocol, as registered

Nothing below was written or amended after a result was seen. One thing was added part-way through the run, and it is declared at the end of this section with its reason; the original text was left standing.

The question

When a database tells a client that the write is committed, and the machine then dies, is the write still there when it comes back?

That is the plain-language form of what every one of these systems promises somewhere in its own documentation. The test is whether the promise holds as configured out of the box, and where the documented durable setting differs from the default.

Two kinds of kill, and they are not the same thing

This is the distinction the whole run turns on.

Process crash

SIGKILL to the database server process. The operating system survives, so anything the database had handed to the kernel with write() is still in the kernel's page cache and reaches the disk normally afterwards. A database can pass this while having fsynced nothing at all.

Power loss

The entire virtual machine process is SIGKILLed. The guest kernel's page cache dies with it, so only bytes the database explicitly fsynced, and which therefore reached the virtual disk, survive.

A process-crash result is not a power-loss result and none of them is described as one here. The process crash is in the run precisely because it is the cheap test, and the gap between the two tiers is the finding.

Stable storage here is the disk image file on the host. Killing the guest virtual machine does not touch it, so a guest fsync that reached the host file is durable by construction. The controls below are what establish that this is actually true of the setup rather than assumed.

The workload, and what acknowledged means

One client per arm, in a strict loop: insert one row whose primary key is a rising integer, in its own committed transaction, and wait for the database's commit acknowledgement; send that number to a collector process running on the host, outside the virtual machine; wait for the collector to confirm it has written and fsynced the record; only then move on to the next number.

The collector's log is the record of what was acknowledged, and it lives outside the machine being killed, so a kill cannot destroy it. Waiting for the collector is what makes the boundary exact: at the moment of the kill at most one number can be in flight, so the highest number the database definitely acknowledged is either the last one the collector confirmed or the one after it.

That one-record ambiguity is resolved in the database's favour. A missing final record is never counted against a system. This biases the whole run toward finding fewer failures than really occurred, and every number here should be read with that in mind.

The kill, and what counts as a pass

The kill fires at a uniformly random time in a five to fifteen second window after the client starts, so it lands at an arbitrary point in each system's own flush cycle rather than at a moment chosen to suit anyone. Afterwards the system is restarted, given up to 120 seconds to become available, and the table is read in full.

A trial passes when every acknowledged record is present. Nothing else is required: not ordering, not timing. Each trial ends in exactly one of four verdicts, and those are the words in the run log below.

clean

Every acknowledged write survived.

lost-tail

The highest surviving record is below the last one acknowledged, and everything below the survivor is present. Acknowledged writes were lost from the end, and the count lost is recorded.

hole

Some acknowledged record is missing while a higher one survived. Worse than a lost tail: what came back is not a prefix of what was promised. No trial in this run ended here.

corrupt

The system did not restart, or the table could not be read at all.

Records present that were never acknowledged are counted as extra and are not a failure: a commit can land after the kill has interrupted the acknowledgement path.

The controls, which run first and without which nothing counts

The instrument has to be shown capable of producing both answers before any of its answers count for anything.

The positive control

A small program that appends a record and fsyncs it before acknowledging, over the same collector protocol. It must come through power loss clean. If it loses records, the virtual disk is not honouring fsync, every database would look guilty for a reason that has nothing to do with the database, and the run is void.

The negative control

The same program with the fsync removed. It must lose records under power loss. If it does not, the kill is not discarding the page cache, that tier has no dynamic range, and a clean result from it would mean nothing.

Both also run under the process crash, where both are expected to come through clean. That is the clearest available statement of what a process crash does and does not test.

What would have falsified what

Written before the run, so that no result could be reinterpreted afterwards as agreeing with whatever it turned out to be.

The strong claim

A loss under power loss for PostgreSQL at stock, MySQL at its default flush setting, SQLite at its library defaults, or MongoDB with j:true would contradict those systems' own documented guarantees. That one is held to the highest bar: any such result is re-run and reported only if it reproduces.

The expected losses

Losses under power loss for the other settings confirm documented behaviour and are not defects. They are interesting only because several of them are defaults, or near-universal habits, and the loss they permit is larger or likelier than a reader of the headline promise would guess.

The cheap tier

Any arm passing the process crash proves nothing about power loss and is not reported as if it did.

The null result

If every arm had come through every kill clean, the finding would have been that these systems keep their promises, and that would have been reported as the result.

What this run does not cover

Stated in advance rather than discovered afterwards.

One node

No replication, no failover, no network partitions. MongoDB's w:1 on a single node is a weaker setting than a replica set would use, and nothing here says anything about a real cluster.

Not a benchmark

Throughput is deliberately crippled by the collector round trip, and no performance number from this run means anything.

A virtual disk, not physical hardware

Real hardware adds a drive write cache that can lie about fsync. This test cannot see that layer, so a real machine can do worse than these numbers, never better.

Distribution defaults

The configurations as the Ubuntu packages ship them, which are not always identical to what a cloud provider or a container image gives you.

No fault injection

No data-corruption fuzzing, no torn-page injection, nothing at the filesystem level.

The one amendment, declared

A third control was added after the first twenty power-loss trials, when the SQLite-at-defaults result needed a channel tested that the original two controls did not cover. A rollback-journal commit ends by deleting the journal and fsyncing the directory, and the positive control had only proved that fsync of file data was durable here. The added control does the directory operation and nothing else.

It is a control, not an arm: it changes no arm's configuration and no verdict rule. Power-loss trials 21 to 30 carry it, which is why its power-loss count is ten where the other arms have thirty. The original protocol text was left standing; this paragraph is the appended amendment.

What it ran on

Ubuntu 24.04.4 LTS, kernel 6.8.0-138 aarch64, in a disposable Lima virtual machine with 4 vCPU and 3 GiB of memory on an Apple M1 Max. Data on ext4 on the virtual machine's own virtual disk.

• postgresql. psql (PostgreSQL) 16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)
• mysql. /usr/sbin/mysqld Ver 8.0.46-0ubuntu0.24.04.4 for Linux on aarch64 ((Ubuntu))
• sqlite. 3.45.1 2024-01-30 16:01:20 e876e51a0ed5c5b3126f52e532044363a014bc594cfefa87ffb5b82257ccalt1 (64-bit)
• mongod. db version v8.0.31
• kernel. Linux 6.8.0-138-generic aarch64
• distro. Ubuntu 24.04.4 LTS

The numbers in the video

Every arm, under both kinds of kill

One row per arm per kill. Clean counts the trials in which every acknowledged write came back. Writes lost is the range of acknowledged writes that went missing across that arm’s failing trials, so an arm that never failed has no range rather than a zero. Opening a row shows the full verdict breakdown, the failure rate with its interval, and the manual the documented claim was read from.

The run log, every trial-arm outcome

All 500 of them: thirteen arms across ten process-crash kills, plus thirteen arms across thirty power cuts, less the third control, which only exists from power-loss trial 21. Acknowledged writes is how far that arm got before the kill landed, under a workload deliberately handicapped by the collector round trip, so it is not a throughput figure. Lost is the count of acknowledged writes missing afterwards, and it is absent rather than zero on the trials where the table itself was gone.

The manuals these claims were read from

Each arm’s documented promise above is a compression of what its own vendor publishes. The four pages it was read from are PostgreSQL on write-ahead logging, MySQL on the InnoDB flush parameters, SQLite on its pragmas, and MongoDB on write concern. Where a compression and a manual disagree, the manual is right and we would like to know.

Ask us to break something

Your system makes a claim somewhere. It is in a README, a status page, a support answer, or a sentence a vendor said on a call. Send us the claim and the thing that makes it, and we will write down in advance what would count as breaking it, then spend a serious effort trying to.

The protocol goes up before the run starts, the same way the one above did, and the result is published whichever way it goes — including the boring way, where the claim holds and the page says so. We read everything and we answer, including when the answer is no.