feat(storage): replace the LMDB chunk store with one file per chunk, and migrate onto it - #216
Open
grumbach wants to merge 62 commits into
Open
Conversation
…up chunk The chunk store can only grow. LMDB returns a deleted page to its own free list and never to the filesystem, so a node that deletes chunks frees no disk. Moving the fleet onto a store that does return space means a node short of disk will have to give up some chunks while it moves the rest across. It cannot avoid being seen doing that, and it cannot stop the consequence, because the penalty is the auditor's decision, not the audited node's. So the auditors stop one release ahead of the migration, and this is that release. What is withheld is deliberately narrow: only the accusation "you did not have a chunk you were supposed to be holding". That covers the responsible-chunk audit, the fresh-replication possession check, the prune audit, a sole-source replica hint whose sender then denies possession, and the fetch paths where a peer that answered Present could not serve the bytes. A node giving up chunks produces every one of those, so withholding some and not others would stop only some of its accusers. The commitment-bound subtree audit is untouched and still penalises. That is not a compromise, it is what makes the rest work: a migrating node reduces its signed commitment precisely so its peers hold it to the smaller claim, and suspending that enforcement would make the reduction meaningless. A sole-source hint the close group rejects outright is also still punished, because that is a claim about a key that does not exist rather than about the sender's own storage. Audits of both kinds keep running and keep recording. Only the trust event is withheld, and the record they leave is how we will know when it is safe to switch the penalty back on, which is a later release rather than a compiled-in expiry so the date can move on evidence. The switch is a build constant, not a configuration field: a node writes its effective configuration back to disk, so shipping it as a setting would bake this release's value into every operator's file and the next release would change nothing. It is initialised from that constant rather than defaulting to "penalise", so a construction path that never applies the policy behaves like this release instead of the previous one. Known cost, accepted: between this release and the one that restores the penalty, a peer that publishes no commitment at all can answer Present, fail to serve, and pay nothing for it. It is bounded by the restore and visible in the audit record. See ADR-0012.
…ging feature `AuditType::as_str` was gated on the `logging` feature because every caller was inside a log macro, which compiles to nothing when that feature is off. The penalty helper takes the label as an ordinary argument, and arguments are evaluated whether or not the macro that consumes them survives, so a `--no-default-features` build stopped compiling. Ungated rather than worked around at the call sites: it is a `const fn` over a three-variant enum returning a string literal, so it costs nothing in a build that never logs, and passing hand-written literals instead would let the structured-log labels drift from the enum they are meant to name.
…iled Review found that `FetchResponse::Error` was routed through the suspended lane, and it should not be. Its only producer is the responder's storage read returning an error: an I/O fault, an exhausted descriptor table, or a chunk whose bytes no longer hash to their address. A peer that simply does not hold the chunk answers `NotFound`, which is a separate variant and stays suspended. Nothing about a node giving chunks up produces an error answer, so withholding the penalty for one hid real faults for no benefit. The response mapping and the charging decision are now two small functions used by the real paths, so the meaning a responder puts on the wire and the charge a fetcher applies cannot drift apart. Tests pin both: a key the node does not hold reads as a plain miss and is answered `NotFound`, a failed read is answered `Error`, and the two answers are classified as different faults. This brings the count back to the six call sites the ADR describes, and the ADR now says explicitly that a failed responder read is not one of them.
…and migrate onto it
LMDB returns a deleted page to its own free list and never to the filesystem, so
a node that deletes chunks frees no disk. Last week the fleet deleted 2.29
million chunks and got back zero bytes. Operators read that as a bug and are
tempted to wipe node directories to reclaim space, which costs the network real
replicas. There is no partial way out: compaction needs free space equal to the
live data, which is exactly the condition a full node does not meet, and it does
not get us off LMDB anyway. Disk comes back exactly once, when chunks.mdb is
removed whole.
THE STORE
One immutable file per chunk, at chunks/<last two lowercase hex characters of
the address>/<full 64-character lowercase hex address>. 256 shard directories,
one level, recorded in a layout marker at creation.
Suffix, never prefix. A node holds keys it is among the closest to, so its
holdings share roughly log2(N / close_group_size) leading bits with its own node
ID, and that shared prefix grows as the network grows. At today's fleet size a
two-hex prefix already resolves to about two directories, and past a million
nodes even four hex resolves to one. The address is a BLAKE3 output and
close-group membership constrains only its leading bits, so the trailing byte is
uniform by construction at every network size.
Lowercase hex because NTFS and default APFS fold case: under an encoding with
both cases two distinct keys can share one case-folded filename, which is a
silent overwrite. No hex string can spell a reserved Windows device name.
The filesystem is the only authority. The key set is a BTreeSet rebuilt at every
open from directory entries, names only, no stat and no content read. There is
no sidecar index, because a persistent one cannot remove reconciliation: commit
the index first and a crash leaves a phantom key, rename the file first and a
crash leaves an unindexed file, and repairing either means reading the
filesystem anyway. Every in-memory mutation mirrors a filesystem operation that
has already completed, never one that is about to.
Writes are a temp in the destination directory, flushed, renamed, and only then
admitted to the index, so a name can never appear on partial content: the name
is the hash. Reads are bounded, refuse anything that is not a regular file, and
repair a corrupt or missing chunk from the network by dropping it from the key
set. Deletes unlink and return the blocks immediately, which is the entire point.
THE MIGRATION
A node that fits its payload copies everything and then removes chunks.mdb. It
is never unable to serve, so it needs no coordination.
A node that does not fit copies closest-first, then commits to what it can hold
while continuing to serve everything it ever committed to, and only then gives
the rest up. Serving reads the union of both stores; the commitment reads the
file-backed set. A node is at worst over-honest.
Nothing is given up without three things being true, in this order:
- the node is not near the front of the group for that chunk (measured against
the admission width the pruner already refuses to delete inside)
- its close group has demonstrably RECEIVED its reduced commitment, proven by
those peers answering a neighbour sync that carried it; until they have the
smaller key set they audit it against the one it used to hold
- all but one of the chunk's current close group has answered a cryptographic
possession challenge over a nonce it has never seen, and is itself currently
publishing a commitment
That last point is the pruner's own evidence, reused deliberately. The cheap
VerificationRequest was not enough: it carries a self-reported present flag, and
a node that has silently lost a chunk still answers yes.
Close groups migrate in waves derived from a hash of each node's own ID, so about
two of seven give chunks up at a time. If every holder went at once none could
prove to the others that a copy survived and the group would deadlock waiting on
each other. A host-wide advisory lock separately serialises nodes sharing a
volume, which is a different question: one machine's disk rather than one chunk's
replicas.
Before chunks.mdb is removed, every chunk both stores hold is re-hashed and
rewritten from the legacy copy if it disagrees. A filename is not proof the bytes
behind it are good, and the startup scan reads names only. Removal itself renames
the environment aside and flushes the parent before recording the migration as
finished, because remove_dir_all is not atomic and a partial failure would
otherwise leave a node claiming completion over a half-deleted store.
The destructive step is off in this release. It ships enabled in the next one,
once the fleet has been seen bridging without incident.
WHAT THIS COSTS
There is no rollback once a node has removed its legacy store; the staged rollout
is the only control. A node whose close group is also short of disk will not get
possession proofs, will not free its disk, and will tell its operator to add
storage, which is the correct answer under "no data loss" but means the migration
does not complete unattended everywhere.
The paid list is still LMDB. It is a fixed 256 MiB map that contributes nothing
to the disk problem, but it is why heed cannot be dropped yet.
See ADR-0012.
…testing Four end-to-end tests assert that a peer loses trust. With the close-group storage penalty now defaulting to whatever the release ships, those assertions silently became a statement about the release rather than about the mechanism they were written for, and they passed or failed on test ordering: one test happened to leave the switch on for the next. Each now sets the switch explicitly, so it tests possession, pruning or repeated failures rather than the release it was compiled into, and the result no longer depends on which test ran first.
`FileStore::all_keys` is async and awaits nothing on purpose: the key set is already in memory, and its callers across the replication engine cannot all be made synchronous in one change. It carried an allow for `clippy::unused_async`. That lint was renamed, so a newer toolchain than the one this was written on fires `clippy::unused_async_trait_impl` instead and the build fails. Allowing both, under `unknown_lints` so whichever name the compiler in use has never heard of stays quiet, rather than pinning a toolchain or restructuring an interface to satisfy a lint.
Retirement is refused on Windows because NTFS documents no ordering between the rename that publishes a copied chunk and the deletion of the store it came from, and there is no way to flush a directory through the standard library. That guard was reading an environment variable directly, which meant the three tests that exercise retirement could not clear it without mutating process-global state, and they failed on Windows CI while passing everywhere else. It is now `storage.migration.allow_windows_retire`, still defaulting to off and still honouring the environment variable for its default. Unlike the release switches this one does persist in an operator's configuration, deliberately: someone who has tested power loss on their own hardware should not have to re-assert that on every start, and the decision is theirs rather than the release's. The three retirement tests now clear it the way such an operator would, and a Windows-only test covers the refusal itself, so the guard has coverage on the platform it exists for rather than only being asserted away.
…found Five findings from an adversarial review of this branch. Four were real. THE MIGRATION WAS NEVER STARTED `migration::run` had no caller. Porting this work onto a newer base involved resetting a bad copy of `node.rs`, and the hand-written wiring that went with it was never redone: the store opened in its bridging phase, dual-wrote, served the union of both stores, and then sat there. No chunk was ever copied out, no commitment was ever narrowed, no legacy environment was ever removed. The one thing this work exists to do did not happen on any node. Nothing caught it. Every test constructed the store directly, and a node with no legacy environment starts no migration, so the end-to-end suite could not see the difference either. There is now a test that the driver is reachable from its entry point and returns on its own when there is nothing to do. A DELIVERY COULD BE CREDITED TO A ROOT ITS PEER NEVER SAW A neighbour sync snapshots the commitment root it will carry, sends it, and on the reply records that peer as having received it. If a rotation happened in between, the recorded root was the new one. The doc comment claimed a rotation empties the recipient set; nothing did that, and the set was instead cleared lazily by the first late reply, which then credited itself to a root it had never been sent. Rotation now clears the set, and the caller names the root it actually put on the wire so a reply that arrives after a rotation is dropped rather than miscounted. This gates whether a node may give chunks up, so a wrong count here means shedding while the close group still audits against the larger key set. A SUCCESSFUL REPLACEMENT SYNC DID NOT COUNT When the primary peer does not answer, the round retries with a replacement, carrying the same commitment. The reply proves delivery exactly as the primary's does, but only the primary path recorded it. A node whose close group is slow enough to fall through to replacements could never accumulate enough recipients before the next rotation reset the count, and would wait forever. That is the node most likely to be short of disk in the first place. A FAILED FREE-SPACE QUERY READ AS A FULL DISK The capacity verdict collapsed both error cases into "full". The verification cycle treats full as a standing condition worth minutes of backoff, and documents that a failed query must not be read that way, because it says nothing about available space and may succeed on the next pass. A transient fault on a network mount would have stalled probes and promotes across every pending key on a node that was not full at all. The three-way answer is restored. ONE RELEASE SWITCH, NOT TWO Two constants of the same name existed, with two environment overrides, on either side of the same decision: whether peers withhold the penalty for not holding a close-group chunk. Nothing coupled them. Setting one without the other gave a node willing to give chunks up while every peer applied the full penalty, which is the outcome the release ordering exists to prevent. The migration now reads the switch the auditors read. The fifth finding, that two same-named constants could drift, is the one above.
The wiring that starts the migration went missing during a rebase and every test still passed. That was possible because each test built the store directly and drove its pieces, and a node with no legacy store starts no migration, so nothing in the suite could tell a working migration from an absent one. Two tests close that, both driven through `run`, the same entry point node startup calls: A node with room to hold its chunks starts with 24 chunks in a real LMDB store, and finishes with `chunks.mdb` gone from the filesystem, all 24 readable out of files, and each one under the suffix shard its address names. That is the whole purpose of this work, asserted rather than assumed. A node that cannot fit its chunks and cannot prove anyone else holds them keeps both stores, stays in the bridging phase, deletes nothing, and serves every chunk throughout. That is the case that must fail safe: refusing costs the node disk, proceeding would cost the network data.
The wiring that starts the migration went missing once and nothing noticed: the store opened, dual-wrote, served the union of both backings, and never freed a byte. A node without a legacy store starts no migration, so the absence looked exactly like the normal case. Two guards, because one was clearly not enough. At runtime, a node that still has a legacy chunk store and no task migrating it now logs an error naming the condition, rather than running indefinitely in a state where its disk can never be reclaimed. It is a wiring fault, so it says so. In tests, `should_migrate` is the single predicate both the spawn site and its test use, so "does this node need migrating" cannot be answered one way by the wiring and another way by whatever checks the wiring. The test drives both cases: a fresh node must not get a task, and a node with an LMDB store must.
An adversarial review of this branch found three. All are the same family: a gate that looked sufficient but was measured against the wrong thing, or at the wrong moment. THE POSSESSION BAR WAS SET BY WHOEVER HAPPENED TO ANSWER The number of proofs required was computed from the peers that qualified, not from the close group. With one neighbour publishing a commitment, one proof was enough. Two last holders of a chunk could each see only the other as qualifying, each demand a single proof, each receive it from the other, and both delete. The bar now comes from the whole group and only qualifying peers count toward it, and a routing view too thin to see a full group is not evidence about that group at all. THE GATES WERE CHECKED HOURS BEFORE THE DELETION Rank, commitment delivery and possession were established, and then verification ran, which re-reads the entire store and can take hours on a large node. Nothing was rechecked afterwards. In that window peers leave, replicas are pruned elsewhere, and this node can become the last holder while its own reduced commitment no longer names the chunk. Verification now runs first and every network gate is re-asked immediately before the removal. A CHUNK COULD ARRIVE AFTER THE GATES AND BE DELETED BY THEM A write that reaches the legacy store and then fails to write its file adds a legacy-only key. Such a key is in no commitment, so the answerability check could not see it, and it would have been destroyed having passed nothing. The removal now takes the exact set the gates cleared and refuses if anything else has joined it, which the critical section proving sole ownership makes authoritative. Also: a retirement whose rename failed left the store closed, so the node could no longer serve chunks that lived only there, and the next tick saw no handle and reported success. It now reopens the legacy store and says whether that worked.
…to run there
Retirement was refused on Windows, on the grounds that there is no way to flush a
directory through the standard library and Microsoft does not document
`MoveFileEx` as durable at return, so the copied chunk could not be shown to have
reached the disk before the old store was deleted.
That was not a solution. It left Windows operators with exactly the problem this
work exists to remove: a store that only grows. Refusing to solve a problem for a
platform is not the same as solving it.
There is a documented way, and it is to stop using a rename there. Windows now
creates the chunk under its final name with `create_new` and flushes it.
Microsoft documents that creation metadata is cached and that `FlushFileBuffers`,
which `sync_all` calls on Windows, is how it is flushed. A successful create,
write and flush is therefore a durable publication under a documented contract,
with no rename and no directory flush involved. The rename path stays everywhere
else, where the directory flush does the same job.
The cost of publishing in place is that a crash mid-write leaves a partial file
wearing a real chunk name, so three paths now refuse to trust a name:
- a write that finds the name taken re-reads and verifies it, and replaces it
when it is wrong, instead of reporting a duplicate and discarding the good
copy that had just arrived to repair it
- the read path already verified and repaired
- the pre-retirement pass already re-hashed everything both stores hold
Separately, the migration waves did not work at all as configured. They opened at
0, 24, 48 and 72 hours from first start while nothing could shed until hour 72,
so every wave was open the moment the first one could act and a close group would
have migrated together, which is the pile-up the waves exist to prevent. They now
open from the end of that hold, and a test asserts the stagger under the shipped
defaults rather than under either setting alone.
Four fixes from the production review, each one a path where a node could delete its only copy of a chunk or never reclaim its disk at all. Commitment recipients are now intersected with the close group as routing sees it at the moment of the check. Counting a peer that received the reduced commitment and has since left the group is no evidence about the peers that will actually audit this node, and it let a node shed chunks while its real neighbours still held it to the larger key set. A directory flush that fails on the publish path is now reported instead of swallowed. That flush is what makes the rename durable, and a copy reported as successful is what authorises deleting the legacy store, so discarding the failure let a power loss take the directory entry after the only other copy was already gone. The migration is no longer skipped when the replication engine fails to start. A node with a legacy store depends on the engine for the commitment state, the routing view and the possession challenges, so it now refuses to start rather than running on forever serving from both stores. The engine build and the migration spawn moved into one function so the two cannot come apart again. Shutdown stops protocol routing before waiting on the migration, and the wait is bounded at 30s. Inbound traffic kept starting new legacy reads, which could stop the drain from ever completing and hang the process. Tests: a departed peer no longer opens the commitment gate; an unflushed publication is not reported as stored; and a fully built node holding a legacy store is asserted to be migrating it. That last one goes through build() rather than the spawn helper, because the failure that already happened here was the call site going missing, and it was verified by deleting the spawn and watching it turn red.
…ws build The Windows build failed on a dead non-Unix helper, which was the visible half of a real gap underneath it. Splitting the publish path on `windows` rather than `unix` was the wrong boundary. The rename-plus-directory-flush route is the Unix route, and every other platform should take the create-in-place route, which needs no directory flush at all. Gating on `unix` removes the dead definition and stops the two halves drifting apart. The repair path had the same gap the publish path had. `write_and_replace` rewrites a chunk whose bytes do not match its address, from the legacy store, during the pass that decides whether the legacy store can be deleted. It finished with a best-effort directory flush, so a repair could be reported as done while a power loss could still undo it, leaving that chunk with the wrong bytes and no other copy. On Unix the flush failure now propagates. Off Unix the replacement overwrites the existing file and flushes it, changing no directory entry, which is durable under a documented contract. That overwrite is not atomic, which is safe only because a crash means no report was produced and nothing was deleted, so the next start repairs it again from a store that is still there. Small-file writes now go through the rename retry as well. The layout marker and the migration state are rewritten while the node runs, and off Unix a scanner holding a handle for a few milliseconds turned an ordinary rewrite into a hard failure. Both platform families were compiled and linted with warnings denied.
Retirement now ships on. Deleting the legacy environment is the only step that returns disk, and a build with it off migrates every node and reclaims nothing, which is the condition this work exists to end. Every gate in front of it is unchanged, and a single node can still be told to keep both stores. A test asserts the shipped configuration retires with nothing set by hand. Durability. Publishing a chunk whose name was already on disk returned success without flushing the directory, so a previous attempt whose rename landed and whose flush failed could be laundered into a copy that authorises deleting the last other one. The flush now covers every successful return. Creating a shard directory flushed its parent best-effort and marked the shard usable regardless, so the first chunk written into a directory that was never made durable counted as stored. That flush is load-bearing too now. Reads. A verifying read that finds rotted bytes throws the file away, and until the key is put back in the union view it appears to live in neither store. Retirement decides it may delete the environment by proving it is the only holder of the handle, so a read that took its handle afterwards could find its fallback already gone. Both read paths now take the handle before they touch the file. Duplicate writes. A name on disk is not proof of the bytes under it: off Unix a chunk is created under its final name before it is written. The protocol handler acknowledged `AlreadyExists` from names alone and the file store had an index fast path in front of the verification, so a good copy offered to repair a damaged chunk was thanked and discarded. Both now compare the length first, one metadata call, and a mismatch falls through to the real write. Availability. The rollback copy into the legacy environment could veto a PUT the file store had ample room for: the capacity verdict is optimistic and LMDB can still refuse a write. It is best-effort now, as its own comment already said it should be. The file write is what decides the PUT. The volume lock is keyed by the filesystem rather than by the path beside the root, so two nodes on one disk no longer take two different locks and copy at once, and only genuine contention counts as contention: a filesystem without locking used to leave a node waiting forever for a holder that did not exist. A node whose file-backed set is empty commits to nothing, so waiting for its close group to receive a commitment that does not exist stranded its disk permanently. That gate is skipped when there is nothing to commit to; the possession check that protects the data still runs. Shutdown stops the protocol children as well as the loop that spawns them, and a migration that overruns its grace is aborted rather than left detached over the teardown it depends on. Repair takes a real reservation instead of an unreserved check. A directory entry the scan cannot identify now fails the scan rather than being counted as "not a file" and dropped from the index.
grumbach
force-pushed
the
storage/file-chunk-store-and-lmdb-retirement
branch
from
August 25, 2026 12:00
63308b2 to
3cf2e92
Compare
Holding a legacy handle for the whole of every read closed the window where a read could lose its fallback, but it replaced one problem with another: the check that authorises retirement is sole ownership of that handle, and on a node serving any traffic there would always be another holder, so retirement would never run and the disk would never come back. Neither ownership alone nor holding a handle states the actual requirement, which is that no read is in progress. A read that has decided the file store cannot answer, and has not yet taken a handle, holds nothing and is invisible to an ownership check while being exactly the reader that must not lose its fallback. So reads now take a shared guard for their whole duration and retirement takes it exclusively before it takes the environment. Because the lock is fair, a waiting retirement stops new readers rather than starving behind them. Test: a read in flight blocks retirement, and retirement completes as soon as that read finishes. Verified by removing the guard and watching it fail.
Retirement moves the legacy environment aside and then deletes it under the new name. The flush in between was best-effort, so if the rename had not reached the disk when the delete landed, a power loss would bring the environment back under its old name with its contents already removed, and the next start would find an environment it cannot open. The flush now reports, and a failure stops before the delete. The migration is still recorded as finished, because the node is serving from files and needs nothing from the environment; the tombstone is simply left for the next start to sweep. Off Unix there is still no way to flush a directory through the standard library, and the helper says so rather than implying otherwise.
The test that copies a whole store and watches the legacy environment go was setting the retirement switch itself, so it proved the machinery worked without proving the release turns it on. It now sets nothing: if the shipped default ever goes back to off, this fails along with the two tests that check the default directly, rather than passing on a value no node would have.
Nine fixes, all on the path that deletes the legacy environment. The set of keys the removal is allowed to destroy was captured after the gates rather than before, so a key that joined between the last gate and the capture counted as approved having passed nothing. It is snapshotted first now, every gate is asked about exactly that set, and a set that moved while the gates ran stops the tick. Retirement is now recorded durably before anything moves. The rename that puts the environment aside cannot be shown to be durable off Unix, so a power loss could bring it back with its contents already deleted and the node would fail to start on it. A marker created with the same create-and-flush that publishes a chunk says the file store was proven to hold everything; a start that finds it finishes the removal instead of opening the remains. It is cleared when the removal completes and when a recoverable failure sends the node back to bridging, so it only survives a crash. The startup sweep of a leftover tombstone now flushes before deleting, for the same reason. The pre-retirement proof establishes that names are durable, not only that bytes are. A publish whose rename landed and whose directory flush failed leaves a name nothing goes back to flush, and re-reading the right bytes from it does not make it survive a power loss. The pass flushes the chunks directory and every populated shard first, and a failure is a proof it did not produce. Writes and deletes take the retirement guard as reads do. Retirement waits for the legacy environment to go idle, and work that could keep starting in it made that wait unbounded. A client offering a chunk this node already holds is now answered from the bytes rather than the name, and a damaged copy is repaired from the offer. Comparing lengths caught an interrupted create but not rot, and either way acknowledging the offer discarded the copy that would have fixed it. The close group was derived one member too wide: the self-excluding routing call returns close_group_size remote peers while the threshold is computed from a group that includes this node, so four real neighbours plus one peer outside the group cleared a bar meant to need five real ones. Both the commitment-delivery check and the possession check now use the self-inclusive call, as the pruner does. A node waiting on its close group no longer holds the volume against every other node on the machine: that wait is a network condition that may never resolve. A six-hour cap backstops any branch that turns out not to give the lock back on its own. Queued request handlers give up when shutdown starts rather than acquiring a permit and beginning fresh storage work under a store being torn down. Off Unix the volume lock is keyed by the volume root rather than by each node's own parent directory. A repair invalidates the capacity measurement, because replacing a short file with a full one adds real bytes the cache does not know about. Tests: a damaged chunk, short or rotted, is repaired from the copy being offered; an interrupted retirement is finished by the next start rather than reopened. Both verified by breaking the fix and watching them fail.
… deleted Holding the exclusive retirement guard through the whole removal meant every chunk request on the node waited behind `remove_dir_all` on a store that can be hundreds of gigabytes. That turns the one moment the migration pays off into an outage. The guard is released once the handle is out and the directory has been renamed aside, which is the point after which nothing can reach the environment: no handle exists and no code looks for the new name. The deletion that follows is slow but reaches nothing anyone is waiting on. It is also released on the path where nothing was taken, rather than being held to the end of the function for a tick that is deferring anyway.
… round Seven fixes. Most are consequences of the previous round's fixes rather than of the original design, which is what a fourth pass is for. The retirement marker was trusted on sight, and it can be stale: a rename that fails recoverably clears it, and that clearing could itself be lost. The environment may have taken a key since that has been through none of the gates. Opening it is now the test. One that opens cleanly is intact, so it is kept and the migration starts again from the copying stage with every gate re-run; only one that cannot be opened is treated as the remains of an interrupted removal, which is the case the marker exists for and the only case where deleting is both safe and the only way the node starts. Clearing the marker is durable now too. The duplicate write path compared lengths, which catches an interrupted create but not rot, and answered "already have it" without reading. It reads now, and repairs from the offered copy on a mismatch. A chunk held only in the legacy environment was assumed good for the same question; those bytes can be wrong too, and when the copier finds out it drops the key from the union view, so refusing the good copy would have left the node holding nothing. It is read and compared, and the offer is taken if it does not match. The six-hour cap on holding the volume lock was armed only on one of the two paths that take it, so a node that took it while copying and then sat waiting could still hold it forever. Acquisition is one function now, which stamps every time, and giving the lock up at the cap starts a cooldown so another node actually gets it rather than losing the race to the one that just had it for six hours. The possession threshold was derived from however many peers routing happened to return. A view that has lost a peer lowered the bar exactly when it should not be trusted; it comes from the configured group size now, and a group that is short, or that still contains this node, is not evidence. Off Unix the volume lock is resolved to an absolute path before the volume is read from it, so two nodes started from different working directories on one drive do not each take their own lock. The deletion of the retired directory runs on its own thread, so shutdown can walk away from a recursive delete of hundreds of gigabytes rather than sitting through it; the marker means the next start finishes it. Tests: an intact environment is never deleted on a stale marker, an unopenable one left by an interrupted removal is finished off, and a legacy-only chunk whose bytes are wrong is replaced by the copy being offered. Each verified by breaking the fix and watching it fail.
Deciding whether a leftover retirement marker is stale means opening the environment, and opening it is a full key scan to derive which keys the file store does not have. The environment was then dropped and opened again by the ordinary path, so a node that came up after an interrupted removal paid for that scan twice. The handle it proved was worth keeping is now the one it keeps.
The fifth review round found that using "failed to open" as evidence of a half-deleted environment was wrong, and it was the load-bearing step of the previous round's fix. Opening an environment queries free space, maps the file, takes a write transaction and scans every key, so a full disk, a permission change, a mapping limit or a transient fault all look exactly like corruption. Deleting on any of those destroys a perfectly good store. The mark now goes inside the directory rather than beside it, and is written only after the rename has already succeeded. A directory that reverts to its old name reverts carrying its own evidence, so what it is no longer has to be inferred from anything. There is nothing to cancel, so nothing can go stale: the previous design needed the mark cleared when a retirement was abandoned, and a clearing that failed or was lost would authorise deleting an environment that had since taken a chunk. A missing handle is no longer read as a finished migration. A rename that failed and could not be reopened leaves the directory on disk with no way to read it, and the driver would have logged the migration complete over a store still holding chunks nothing else could serve. Reading a chunk to check it now has four answers rather than two. "Could not read it this time" was being treated as "wrong", and off Unix replacing a chunk truncates it in place, so a transient fault could turn a healthy sole copy into an empty one. The duplicate check holds the key's critical section for the whole of it, so the pruner cannot delete both backings between the read and the answer and leave the offered copy refused for a chunk the node no longer has. Deleting a retired directory runs on a detached thread that nothing waits for. On the blocking pool a normal runtime shutdown waits for it anyway, and in the migration task an abort is not observed until the call returns, so the advertised shutdown bound did not apply to a recursive delete of hundreds of gigabytes. The startup sweep is detached for the same reason: a node should serve immediately rather than wait out a leftover deletion. The volume-lock cap now distinguishes using the lock from sitting on it. Copying and verifying are the exclusive disk work the lock exists for, and a store large enough that verification runs past the cap would have had the cap interrupt and restart it, which is the cap causing the problem it prevents. Tests: an environment carrying no mark is kept however badly it reads, one carrying its own mark is removed whatever it is named, the mark survives the rename it exists to outlive, and a lost handle beside a live environment blocks retirement.
A start that found a reverted environment cleared any existing tombstone before renaming, which put a synchronous recursive delete back on the path that had just been cleared of one. The node would have waited it out before opening its store. Retired directories are now named so they cannot collide: the environment is moved under whichever retired name is free, and the sweep detaches a deletion for each one it finds rather than assuming there is at most one. Nothing on the startup path deletes anything itself.
…e it durable The sixth review round found that the previous round's mark was neither required nor durable, which meant its guarantee did not hold. The sweep deleted anything wearing the retired name without looking inside it. That name comes from a rename, and the rename happens after every gate, with the mark written straight afterwards; a crash in between leaves a whole environment wearing a name that says otherwise. Such a directory is now restored to its own name and the migration runs again, and when both names are taken neither is touched and the operator is told which the node is using. The mark itself is flushed along with the directory that now contains it: flushing the file makes its contents durable, and the entry naming it lives in the directory. Three places still folded the four-valued read back into two. A metadata call that failed counted as a length mismatch and triggered a destructive replace; that pre-check is gone, since the read that follows answers the question properly. An unreadable indexed chunk returned success from a write, which a client reads as an acknowledgement and acts on by dropping its own copy; it returns an error now. And quarantine treated a failed re-read as an empty file and deleted the chunk, which could throw away a copy a concurrent repair had just published. A node that lost its handle to an environment still on disk now tries to reopen it every tick. Saying so once and waiting for a restart left an otherwise healthy node unable to serve part of what it holds, for a reason that is usually transient. A verification pass that failed no longer counts as work, so a node whose store cannot be read cannot hold the volume against every other node on the machine for good. The completion line now says the space is being returned, and a separate line says when it actually is. Deleting a large environment takes minutes, and an operator could not otherwise tell a slow deletion from a failed one. The background reaper retries with backoff rather than giving up on the first sharing violation. Tests: an unmarked retired directory is restored rather than deleted, and one beside a live environment is left alone. Verified by reverting to name-based deletion and watching both fail.
Three fixes from the seventh review round. It found no blockers. The four-valued read was still collapsed on the path that runs after a chunk is published. Only "wrong" was handled; "not there" and "could not read it" both fell through to success, and a caller that hears success acts on it. A client drops its own copy, replication marks the key held, and the copier takes it out of the legacy-only set. All four answers are handled now, and three of them are failures. Deleting a retired directory takes its mark away last. A recursive delete walks in whatever order the filesystem gives, so it could unlink the mark and then fail on the data file, which is exactly what a sharing violation produces. What was left was a genuinely retired, partly deleted directory carrying no evidence of it, and the next start would have read that as an intact environment and restored it. The reaper also keeps trying for about a day with capped backoff rather than giving up after five attempts and stranding the disk until the next restart. A directory under the live name that says it has been retired is never opened, even when it cannot be moved aside. It may be partly deleted, and opening it would put keys back into a commitment they have already left. The node serves from files alone, which is what the mark records as safe, and says so. Recovering a lost handle no longer scans the whole environment under the exclusive guard, and no longer does it every tick: the open happens outside the guard, which is then taken only to install the result, and a failure backs off. A node in that state also gives the volume lock back, since no amount of exclusive disk access will fix a store it cannot read. The migration tests wait longer for a phase change. The deadline is measured on the wall clock while the driver it waits on runs on the runtime, so on a saturated machine both stretch and a deadline sized for the work rather than for the contention turns a slow build into a failing test. Seen once here while a full lint and a review agent were running alongside it. Tests: a deletion that fails leaves the mark in place, and a marked directory under the live name is not served from. The second forces the rename to fail, because with it succeeding the test passed either way.
…eanup The eighth review round found that the manual deletion walker added last round follows a symlink at the top level. An operator who points the chunk environment at another volume leaves a link there; retirement renames the link, writes the retirement mark through it into the target, and the walker then deletes the target's contents, which are not this node's to delete. A linked environment is now copied out of but never retired, the operator is told to remove it by hand once the migration has settled, a link is never treated as retired whatever is written through it, and the walker refuses to descend one. Recovering a lost handle worked out which keys only the environment holds before taking the guard, then installed that answer after. Reading a large environment takes minutes, and a verifying read in that window can find a file rotted and throw it away; with no handle installed there was nothing to put the key back into, so it would have been missing from every gate and from verification, and retirement would have destroyed the intact copy. The environment is still read outside the guard, but the comparison against the file store happens under it, where nothing can move. A commitment that could not be recorded is no longer reported as progress. It was resetting the volume hold cap every tick, which let one node whose filesystem had gone read-only keep every other node on the machine from migrating for as long as it lasted. A chunk that cannot be read stops being advertised. The error alone was not enough: the index entry stayed, so the copier dropped the key from the legacy-only set on the strength of the name and replication answered "already held" and never repaired it. The file is left alone and a later successful read puts it back. Two removal paths that gave up for the rest of the process now keep trying: the driver retries a cleanup that could not finish, whatever phase it is in, and only exits when there is nothing left on disk. A deletion that removes the contents and the mark and then cannot remove the directory puts the mark back, since an unmarked directory that still exists is the one state the scheme says cannot happen. Tests: a linked environment blocks retirement, is never treated as retired, and deleting through it is refused with nothing touched behind it; a directory that outlives its own deletion still says what it is.
The ninth review round found that last round's fix for an unreadable chunk created a way to lose one. Dropping the index entry stopped the copier and replication treating the name as possession, but a key already copied is not in the legacy-only set either, so it ended up in neither view. Verification skipped it because the file store did not claim it, nothing else looks at anything but those two views, and retirement then deleted the environment holding its only copy. The index entry stays now. Removing one is the quarantine path's job, which removes the file with it after a read that succeeded and proved the bytes wrong, so the index and the disk stay in step. The real gap it exposed is closed at the same time: verification used to skip any key the file store did not have. That is correct for a key this node is giving up, which is in the legacy-only set and has gates of its own, and wrong for anything else. A key in neither view has been through nothing and is protected by nothing, so it now goes back into the legacy-only set where the gates can see it, and refuses the proof for that pass. The driver no longer exits while a directory is still being deleted. Both the finished-retirement path and the file-only phase returned immediately, so if the background deletion ran out of attempts nothing was left to try again until a restart. They keep the loop alive and it exits at the top, once nothing is pending. Only one reaper thread runs per directory, since asking for cleanup on every tick was starting a new one each time. A root directory that cannot be listed reads as "cannot tell" rather than "nothing there", which is what it was doing while deciding cleanup was complete. A linked environment gives the volume lock back rather than holding it for six hours waiting for a retirement that is never going to happen, and says so at warning level once an hour instead of at debug. Test: a key the environment holds that is in neither view refuses the proof and is put back where the gates can see it. Verified by removing the distinction and watching it fail.
Two tests provoke a deletion failure with directory permissions, which is not how the same thing happens on Windows, and their bodies were gated while the variables they set up were not. That left unused variables on Windows and broke the build there. Both are gated whole now, and both assert what they were only conditionally checking before: the deletion is required to fail and the mark is required to survive it, rather than the assertions being skipped if the setup did not produce the failure.
…o hold The fourteenth round confirmed the exposure the previous round's fix created. A write announced itself by putting its key straight into the legacy-only set, which is not a note to self but the union view's authority: for the length of every write the node reported the chunk as held, counted it, offered it to neighbours, and while bridging could sign a commitment to it and price a quote from it. Under concurrent writes that is not a moment but a backlog. It also left a tail: a write whose halves both failed left a key with nothing behind it, and in the committed phase nothing copies such a key, so a node could sit at the possession gate for good. Announcements go in their own note now. It is deliberately not part of what the node holds, so nothing above sees it. It vetoes retirement, because what the environment holds is unsettled while it is there, and the driver resolves each entry against the disk once the work behind it has drained: the file store has the chunk, or the environment has it alone and the key is promoted, or neither and there was nothing to protect. Two paths proved a file's bytes wrong and then tried to fix it without saying so first. A repair or a quarantine can fail on capacity or I/O or be cancelled, and a chunk proven wrong that goes on looking healthy is one a cached pre-retirement pass still covers, so the legacy copy that would have repaired it gets deleted. Both record it before acting on it. The test for the in-flight note covers the veto, the reconciliation, and that the two sets are treated differently. It does not cover the window itself: observing what the node claims part-way through a write needs the write paused, which is more machinery than the property is worth.
The doc job denies warnings, and a public method's documentation cannot link to a private one. Said in words instead.
…tters The fifteenth round went at the journal added last round and found three ways past it, all of them at the edges rather than in the middle. It was checked before the exclusive guard was taken. A write can announce itself under the shared guard, be cancelled so the guard is released, and leave its blocking half running past the drain that follows. The check now happens in the same critical section as the ownership and approved-shed checks, immediately before the handle is taken, which is the only moment the answer cannot change. Reconciliation read an unanswered question as an answer. A read of the environment that failed was treated exactly like one that found nothing, and the note was dropped either way, which loses the protection for a write that did land. The four outcomes are now four branches, and a failed read keeps the note and asks again next tick. Draining is not a barrier by itself, either. A second write for the same key could announce itself while reconciliation was deciding the first one's fate, be cancelled, and have its own blocking half outlive the single note they share. Reconciliation takes the exclusive guard before it snapshots anything, so nothing new can start while it works. It is only reached when something is waiting, which after a clean run is never. A delete now outlasts any queued write for the same key. One could otherwise land after the delete and be found by the next reconciliation, putting the key back on the copier's list and undoing a prune the node had decided on. A mark saying a chunk's bytes are wrong is cleared by anything that proves them right: a verifying read, a fresh publish, a re-read that finds a repair landed, and an exact comparison against a caller's own copy. It was only being cleared by a repair this store performed itself, so correct bytes could stay unclaimed while the node happily served them.
The sixteenth round found no blockers and one real gap in the previous round's fix. A delete waited out the environment half of a write nobody was waiting for, but not the file half, and either can be the one still running. A publish that landed afterwards recreated the file and its index entry, putting back a chunk the node had decided to prune, with the note already gone so nothing would notice. Both halves are drained now. The regression for it took three attempts to make honest. The first drove a real write and aborted it, which is a race about which half had started: on a quick machine the file half was parked as intended, and under load the abort landed first and the test passed having set up a different state than it described. It builds the state directly instead, and waits for the publish to be genuinely in flight rather than for a fixed delay. The gate it parks on is held from its own thread, so no blocking guard is held across an await. Also from this round: a test-only count of blocking work in flight, which is what lets that wait be a fact rather than a guess. Confirmed by the reviewer this round: the lock order has no reverse edge (shared guard before key lane everywhere, exclusive guard before key lane in reconciliation, verification takes lanes and never asks for the guard), a persistent read error retaining a note is the intended fail-closed behaviour rather than a leak, and every path that clears a mark first proves the bytes good or proves the file gone.
Four mechanisms are load-bearing in the implementation and were not in the design as written: a directory that says from the inside that it was retired, a chunk kept but not claimed, a verification proof that expires, and a write that announces itself before it starts. They share a cause worth naming rather than leaving as four separate notes: a fact established at one moment being acted on at another. Copying, verifying and retiring are hours apart by design, and every gap between them is somewhere the store can move. What works is making a belief carry its own expiry rather than checking again and hoping the check is close enough to the act.
The seventeenth round found the previous fix's scope assumption wrong, which is a better finding than another instance would have been. A delete waited for writes it could find in the dual-write journal, but the copier and the repair path write only the file and neither goes near that journal. Using it as a proxy for "is anything writing this key" was an assumption, not a fact, and a publish from either path could land after a delete and put back a chunk the node had decided to prune. The file store keeps its own record of what it is part-way through writing, registered before the work is spawned and cleared inside the work rather than by the caller. That is the whole point: the blocking half is not cancelled with the future, so anything the future was going to do afterwards is not a record of what happened. A delete waits on the key it is deleting, not on every write the store has in flight. Startup now fails when the store lock cannot be taken. It used to warn and carry on, which leaves two processes able to open one directory, each with its own index, its own view of what is in flight, and its own opinion about whether the environment may be deleted. A node that cannot take the lock has no way to know it is alone, and this is the one migration where being wrong about that destroys data. The unlocked sweep path that existed only for that case is gone with it. A legacy record too large for this build to store is removed and counted unusable, as a malformed one already was. The legacy API had no size bound and the file store does, so no amount of retrying resolves one, and a single such record would stop this node and every node sharing its disk from reclaiming space. Repair enforces the same ceiling, which it did not: it could install bytes the read path would refuse for ever. A repair's capacity reservation is released by the work rather than by its caller, so a caller that goes away no longer frees room the write is still about to use. Regression: a delete outlasts a file write that no journal knows about, which is the copier's exact shape. Both delete regressions fail three times out of three with the fix removed.
…oved The eighteenth round found no blockers and said the actor rewrite is not needed, which settles the open design question. What it found was the targeted fix being incomplete in three places. The registry recorded that a key was being written, not how many times. Cancellation releases the caller's lane while the blocking half survives, so a second write for the same key can start behind the first, and whichever finished first would clear the single entry and tell a waiting delete the key was free while the other was still queued. It counts now, and only the last one to finish wakes anyone. The store lock did not belong to the work that relies on it. The startup scan sweeps interrupted writes on the strength of being alone in the directory, and it runs on a thread that outlives the future that started it: a cancelled startup released the lock while that sweep carried on, into a directory another process could by then have opened. The lock is shared and every scan and every mutation holds a lease of its own. A successful repair recorded what it had proved after the await rather than in the work. A caller that stopped waiting left a healthy file excluded from everything the node claims to hold, and the capacity measurement believing the store was a chunk smaller than it is. The work does it now. The pre-retirement pass had the same gap from the other side: it reads raw, hashes the bytes itself, and returned without saying so, which could retire the environment while a chunk it had just proved good stayed unadvertised. Regression: waiting for a key waits for every write of it, not the first to finish. Fails three times out of three against the old registry. Also from this round, and worth recording rather than arguing with: refusing to start without the lock is right, and deleting an oversized legacy record is right, because production ingress already enforced that ceiling and preserving one for a hypothetical larger future limit would strand the migration now.
The nineteenth round found one thing and nothing else. Both stores sit on one disk, each measures the same free space, and neither knows what the other is about to spend. A PUT during the bridge passes the file store's capacity check, its legacy copy then extends the environment, and the file write is admitted against a measurement taken before that growth. Concurrent PUTs compound it, and the pair can cross the reserve together and fill the volume this whole exercise exists to free. From the moment it is adopted, the environment is pinned to what it already occupies. The machinery was already there for running out of disk; it is now held for the whole bridge regardless of how much room there appears to be, because the room is not this store's to spend while another is counting on it. It writes only from pages it already has, and refuses anything else, which the caller already handled by storing in files alone. What that costs is that the rollback copy is made only when the environment has room of its own. That is the right way round. The copy exists to make a fleet rollback survivable, not to be the write that has to succeed, and an environment this migration exists to delete should not be taking new disk to hold a second copy of something the file store already has. On a real node it usually will have room, because this migration exists precisely because deleting millions of chunks filled the free list and returned nothing to the filesystem. The test that asserted the rollback copy always lands now asserts what actually matters: four PUTs during the bridge, every one in the file store, and not one byte added to the environment.
7 tasks
The design recorded a cancelled awaiter dropping the per-key lock as a bounded risk. It was not: a publish landing after a delete undoes a prune, and a cancelled write into the legacy environment could leave a chunk that neither view protects, which is what retirement destroys. Both are fixed, so the entry now says what was done rather than what was tolerated.
Another ADR took 0012 on main while this branch was open, and the governance check refuses two documents sharing a number. Renumbered, with the title to match. No content change.
…g them Five things were listed as impossible from a workstation. Four of them were not; they needed harnesses rather than a fleet. **The disk actually comes back.** The claim the whole change exists to make good, and the one thing nothing checked. The unit tests proved the environment was removed, which is not the same thing and is exactly the mistake that started this work: the fleet deleted 2.29 million chunks, every counter agreed they were gone, and not one byte returned. So this measures the filesystem rather than the store's opinion of itself, before and after, and then reads every chunk back to be sure the space did not come back by losing data. **What survives a crash.** A real child process is killed part-way through writing and part-way through copying, and this one then opens the store and checks what is there: nothing claimed that cannot be served, nothing lost from both stores at once, and the store always opens. It does not cover losing the page cache, which is what a power cut adds and what no hosted runner can do. That half stays a fleet gate and the file says so. **Several nodes on one disk.** The volume lock excludes, passes on when released, and each node finishes holding its own chunks and only its own. **What one file per chunk costs at scale.** Startup scan time, the index's memory per chunk, and one inode per chunk: all three stated in the design and none of them measured. Regression gates rather than benchmarks, with the numbers printed so drift is visible before it trips anything. 100,000 chunks scan in 87ms locally. `ANT_SCALE_KEYS` raises the count for a bigger run. The durability tests also run on real ext4, XFS and btrfs volumes, built as loopback images in their own CI job, rather than only on whatever the runner happens to provide. btrfs earns its place there: it has been observed reordering writes around a rename, which is the operation the whole publish path is built on. One fixture bug worth recording: the first content generator folded the chunk number into a wrapping fill, so chunks 251 apart were byte-identical and content-addressed storage held one where the test believed it held two. It undercounted by a third and the assertion caught it.
The review of the first version found that most of them tested their own fixtures rather than the store. Three findings were blockers and all three were right. **The crash tests were not crashing inside anything.** The parent slept and hoped; on a quick machine the child had written every chunk before the kill arrived, so the test checked a clean shutdown while claiming to check a crash. There is a named failpoint now, compiled only under the test feature: the write stops with the bytes on disk and the rename not yet made, writes a marker to say it is there, and the parent kills it at that exact point. The migration child copies one chunk at a time and panics if it finishes, because a child that completes proves nothing. The dual-write test seeded the very addresses its child then wrote, and the write path skips the environment half for a key already on the copier's list, so no dual write happened at all. **The shared-volume tests never touched the lock they were about.** They called the copier, which does not take it; the driver does. Two real drivers run concurrently now, and the mutation the reviewer named, removing the lock from the driver entirely, turns the test red. **The reclamation test measured file lengths.** That is what a file claims, not what the filesystem has handed out, and it would pass while every block stayed allocated: unlink a file something still holds open and every name disappears while nothing is freed, which is a fair description of the bug that started this. It measures allocated blocks and the filesystem's own free space now, sampled before, at the two-copy peak, and after, with the store dropped first so no handle keeps blocks alive. Renaming the environment aside and not deleting it now fails the test. **The scale tests counted their own planted files.** The inode claim goes through `put`, so a store that wrote a sidecar per chunk is caught. The "does not read contents" claim is measured in bytes read from `/proc/self/io` rather than in elapsed time: the files were written moments earlier, so reading them back comes from the page cache and a timing comparison passed with a deliberate read of every file added to the scan. The loopback job is no longer called a durability job. Killing a process and reopening the same mounted filesystem keeps the page cache, so it exercises each filesystem's syscall, locking, rename and delete behaviour, not its behaviour under power loss. That still needs fault injection or real hardware. Every fix above was checked by making the regression and watching the test go red.
Two of them were tests that passed with the behaviour removed, which is worse than having no test. **The dual-write test proved nothing.** It seeded unrelated keys, then merely asserted the copier's list was non-empty, which those seeds satisfied whether or not a dual write had happened. It names the interrupted chunk now, and removing the environment write turns it red. That fix exposed a second thing worth recording: with the environment pinned to its current size for the whole bridge, a freshly seeded one has no free pages, so the second write never lands and the case cannot arise. The fixture now deletes half of what it seeds, which is what a real node looks like and precisely why this migration exists. **The migration crash test had only an upper bound.** It checked that not everything had been copied, so a copier that copied nothing passed: everything was still readable from the environment. The child says it is working only after a copy has actually landed, and the parent checks both bounds. **The mid-publish test iterated nothing.** The failpoint stopped the very first write, so the store had no chunks and the loop over what it claimed ran zero times. The failpoint takes a count now and lets twenty land first, so the crash happens to a store with real content in it. **The reclamation test never took the peak sample its comment described.** It does, between copying and retiring, and compares the recovery against the environment's measured size rather than against a multiple of the payload. Unlinking the environment while holding it open, which loses every name and frees nothing, now fails it. Also: the measuring tests run single-threaded in CI, because free space is filesystem-wide and resident memory is process-wide and four tests sharing either measure each other. Both crash handshakes have a deadline, so a failpoint that stopped working fails the job rather than hanging it. And the shutdown-drain test is now actually run by CI, having been written, wired into Cargo, and never invoked.
…s one Three failures, each real rather than a runner being slow. **A fixed delay decided whether the migration crash test tested anything.** On the runner the child had copied nothing in its 150 ms; on this machine it had copied some. It uses the same failpoint as the other children now: ten chunks copied, the eleventh interrupted, the rest untouched, the same every time. The progress handshake it used instead is gone with it. **btrfs does not report freed space immediately.** The test slept 200 ms and took one reading, which on ext4 was enough and on btrfs was not. It polls for the space to come back, with a deadline, and returns the last real reading so a genuine failure still fails on the number rather than on the wait. **Two things only CI's toolchain sees.** Clippy 1.98 rejects an unbounded range in a for loop where 1.95 did not, and making the file store public under the test feature put a doc link to a private item in front of rustdoc for the first time. Local checks now include `cargo doc --features test-utils`, which is what would have caught the second one here.
…old one btrfs charges very differently for four hundred small files than for one large one, so comparing what the disk still costs against what the environment used to occupy was a statement about filesystem overhead rather than about the migration. It compares against what the file store actually occupies now. The load-bearing assertion, that retiring hands back most of what the environment held, was passing on btrfs already. The numbers are printed as well as asserted: on this machine the environment held 14.1 MB, the file store holds 6.6 MB, and 14.1 MB came back.
…f Unix where they hold The four new harnesses were sequenced after the e2e testnet suite. That suite flakes on hosted runners for transport reasons unrelated to storage, and a failing step aborts the job, so on the last Windows run the harnesses did not execute on any platform at all. They are fast and deterministic, so they now run first and always report. The sweep test is Unix-only, because the leftover it sweeps only exists on Unix. Off Unix the store creates the chunk under its final name and flushes it, deliberately, since a rename there is not documented to reach the disk. There is no temporary file to find, and the equivalent hazard there is a real chunk name over short or wrong bytes, which the store's own tests cover on every platform. The scale harness runs on Linux only. It plants a hundred thousand files to measure what a restart costs, and that is a fleet question, where every node is Linux. Re-measuring it on the Windows runner would cost minutes of every run for an answer no node needs.
…y platform Off Unix the failpoint sat after the chunk was created under its final name and written, and before the flush. That is not the moment between the two halves of a dual write: the file is already there and already readable, so the crash test asserting that an interrupted chunk stays on the copier's list failed on Windows for a correct reason. It could not have proved anything about the missing flush either. Killing a process does not empty the page cache, so the bytes survive; only losing power loses them, which no test that kills a process can stage. Moved to before the file is created, which is the same point in the sequence as the Unix temporary-file-written-not-yet-renamed halt. Both scale ceilings were far looser than the measurements justify. The scan-time ceiling was a flat thirty seconds against a hundred milliseconds measured, so a ten-second stall passed; it is now fifty microseconds per key, which scales with a larger run. The bytes-read ceiling was a hundredth of the payload, which grows with chunk size and so permitted a 655-byte header read of every file in a test named for not reading contents; it is now a fixed 64 KiB against the 125 bytes measured, so any read that is per-chunk at all fails, and fails harder the larger the store. Also gate the failpoint out of shipped binaries. It is compiled only under test-utils, which is not a default feature and is not passed by the release workflow. CI now proves that instead of trusting it, by looking for the environment variable name in a default-feature build: the literal survives into the binary whenever the code that reads it is compiled, and it is present with the feature on.
The driver is documented as holding the volume from the first copy through retirement and not handing it back in between. Only the copying half had a test. Retirement is the heavier half: re-reading every chunk in the store to verify it, then deleting an environment. A driver that took the lock only for copying would run that pass while its neighbours on the same disk ran theirs, which is the pile-up the lock exists to prevent. The new test is shaped like the copying one, so the answer does not depend on catching a short window: an outsider takes the volume first, the node is put in the phase where retiring is the only work it has left, and it is watched for not doing it. Then the lock is released and it must retire, which is what keeps the test from passing against a node that never retires at all. Removing the lock from the retirement branch of the driver fails it. Copying and committing are done by hand rather than by waiting for the driver, because the driver reaches that phase by waiting out the shed hold, which is days. Both nodes are now checked for holding their own chunks and only their own. Counting just the one that went first would pass for a node that had picked up its neighbour's chunks as well. The bytes-read ceiling constant moves to the top of its file: clippy 1.98 rejects an item after a statement, and CI runs a newer clippy than this machine.
… if a harness runs nothing The new retirement test drives the store through a hook that exists only under test-utils, so the target now declares that feature and every job that runs it passes it. Without this the harness does not compile, which is how it failed on the loopback filesystem jobs. Each harness step now checks that it actually ran something. A test binary that reports no tests, or a target skipped because a feature was not passed, exits zero and reads as a pass, which is a harness quietly going dormant. Those steps run under bash explicitly, since the Windows runner would otherwise use PowerShell. The reclamation harness prints its measurements everywhere it runs, not only in the main test job. What that job is for is the number each filesystem gives back, and capturing the output meant ext4, XFS and btrfs each reported nothing but a pass.
…hat is left The validation section predated the four harnesses. It now says exactly what runs on every commit and what each mutation check confirmed, so the gates that remain are the ones a workstation genuinely cannot close. Two of them are narrowed rather than removed. The loopback filesystem jobs are not offered as closing the power-loss gate: killing a process keeps the kernel page cache, so removing every flush from the publish path would leave them green. What they do cover is the rest of what a filesystem decides. And the scale gate is now 1M and 10M keys rather than 100k, since 100k is answered in CI and printed, though where the curve stops being linear is still a question about a machine holding ten million files.
A crash during retirement is now staged, which is the most destructive moment in the migration: the environment renamed aside and marked, nothing yet deleted, and the process that wrote the mark killed there. The next start has to finish that deletion and never reopen the directory, because the node has already told the network it serves those chunks from the file store. Its recovery had unit tests that planted the mark by hand; what those cannot show is that the mark is really on disk at that moment. Refusing to believe the mark fails the new test. The startup scan's "names only, no stat per entry" claim had no protection. A flat ceiling cannot give it any: one stat per entry costs about three times a bare walk and stays well inside any ceiling loose enough not to flake on a shared runner, which is why the mutation passed. It is now measured against the machine instead of against a number. The same directory is walked twice in the same process, once reading names and once calling metadata on each entry, and the scan must land on the names-only side of the two. Runner speed cancels because it moves all three together. Adding the stat takes the scan from 88 ms to 296 ms against a 123 ms midpoint. The reclamation test read a signal that runner noise could swallow. Chunks are now 128 KiB rather than 16 KiB, which puts the environment at about 59 MB and the recovery threshold an order of magnitude clear of the drift, and the drift itself is measured and printed so a failure says whether the space did not come back or the machine was busy. Two tests promised more than they did. One that never started a migration driver now runs a real one for the whole of its wait, so that mapping lock contention to "available" no longer leaves it green. The other is renamed to what it checks, since it copies and never retires. The index memory test now says plainly what it can catch. Process-wide RSS and allocator reuse mean it finds an index costing several times what it should, not a small regression, and it should not be read as a byte-accurate account of one data structure.
…refusal The record said a node should refuse to delete the legacy environment on Windows until an operator explicitly overrode it after power-loss testing. That predates the two changes that removed the reason for it, and the code has shipped without a platform condition, so the record and the build disagreed. Publishing a chunk off Unix no longer renames at all: it creates the file under its final name and flushes it, which Microsoft documents as flushing the creation metadata with it. Retirement still renames the environment aside, and that rename is not durable there, but the mark now goes inside the directory rather than beside it. A power loss that reverts the rename brings the directory back under its live name still carrying its mark, and a marked directory under the live name is never opened or served from. A loss before the mark leaves it unmarked under either name, and an unmarked directory is always restored and reopened. All four states have tests. Replacing a policy with a mechanism is the better answer here: a switch nobody turns on is a migration that never finishes, and the fleet already deleted 2.29M chunks and got back nothing. The per-node override remains for an operator who wants retirement held off one machine, and forced power-loss testing is still an open gate on every platform. What that run is now checking is directory creation, which has no portable flush. Also corrects the claim that all four harnesses run on three platforms. The three that touch durability do; the fourth measures what one file per chunk costs at scale, which is a fleet question on a fleet that is Linux.
…nything The mark said yes or no and folded every other answer into no. Reading it can fail for reasons that are neither: a permission change, a descriptor limit, a filesystem that has gone away underneath the node. Folding those into "no mark" fails in the worst direction. A retired environment that reads as unmarked is put back under the live name and reopened, and its keys re-enter a commitment they have already left. It is now three states, and the two questions callers actually ask are asked separately. Deleting a directory requires a mark that was read; opening one requires a mark known to be absent. Neither treats "cannot tell" as a yes, and the six call sites each ask the one they mean. A path that is not there is still definitively unmarked, which is the ordinary case and asked on every tick. This is the same defect as the others this branch has been fixing, in a smaller place: a belief that fails open, acted on later as though it had been established.
…orrect stale docs The scan comparison took one sample of each of its three measurements, so a scheduling pause that landed on the scan and not on the two walks decided the result. Runner speed only cancels when it moves all three together, and a preemption does not. Three interleaved rounds and the median of each: headroom on this machine goes from about 25 ms to 52 ms, and the stat mutation still lands at 257 ms against a 123 ms midpoint. The index memory gate allowed 256 bytes a key against 52 measured, which is loose enough to let another 128 through unnoticed. Now 128. Five places said something the code does not do. The record described publishing as always a rename two lines above the table explaining that one platform does not rename at all, and put documented the same thing unconditionally. The Windows section said two uses of rename were gone when one of them is still there and is safe for a different reason. The retirement crash test claimed a killed process settles that the mark is on disk, when the page cache means it settles when the mark is written, not that it survives power loss. The shared-volume file carried a duplicated heading, and the failpoint check named a count that changes whenever a failpoint is added.
…t be classified An unreadable mark and an unopenable store are different problems with different answers, and one message for both sends an operator to the wrong place. A store this node cannot open needs a restart; a store nothing can classify usually needs a permission or a mount looked at, and the node will neither open nor remove it until that is fixed. The predicate itself now logs at debug rather than warn. It is asked on every tick, so a warn there would be a wall of the same line, and the retirement blocker is the message an operator is meant to read. Also converts the comparisons a mechanical edit left as `assert!(a == b)`, which the clippy CI runs rejects. This machine was two minor versions behind CI, which is how three of these reached it; it is now on the same toolchain and the whole lint, doc and test pass is clean there.
…l fell through Making the mark tri-state was only half of it. Two callers asked whether it permitted removal and let every other answer fall through to the opposite action, so "cannot tell" still reached the opening path. A start finding an environment it could not classify opened it, and the tombstone sweep renamed one back under the live name. A mark check that fails for a moment and succeeds the next is enough for that to resurrect a store that really had been retired. Both now match all three answers: read the mark and remove, know there is none and open, or do neither. Doing neither costs disk until somebody looks, which is the right price for not knowing. A node that cannot classify its environment is also work no amount of exclusive disk will finish. It now stands down from the shared volume instead of holding it to the six-hour cap while its neighbours wait, and it says so through the throttled operator warning rather than only at debug. The tests for this were staged by taking every permission off the directory, which staged too much: at mode 000 the operating system refuses the rename as well, so the tombstone test passed with its own protection removed. It also would have failed on any CI running as root, since root can read a mode-000 directory. The mark is now a symbolic link pointing at itself, so looking for it returns a loop while everything else about the directory keeps working, for every user. Disabling either branch turns both tests red.
… branches that check it The recovery asked whether the environment was there before asking what its mark said, and folded an undetermined answer into "nothing here". That skipped both of the branches added to stop exactly this: an environment whose presence could not be determined went straight to the opening path. The mark already tells the three apart. A path that is not there carries no mark and says so; a path that cannot be reached says it cannot be reached. So there is nothing for the extra question to add, and one less place for an answer to be lost on the way.
… whether or not there is a handle Two ways an environment nobody could classify could still be deleted. The mark is written with `create_new`, and a failure saying something is already at that name was taken as "the mark is there" and the environment deleted on the strength of it. What is at that name might be anything. Every other part of this file insists the name is not the evidence; this was the one place taking it. A mark already present is now accepted only when it reads back as one. And the classification was asked only when the node had lost its handle, so on the ordinary path it was never asked at all: a node holding its store open went through every gate, renamed the directory aside and deleted it, whatever the mark said or failed to say. It is now the first question the retirement blocker asks, before anything else and whether or not there is a handle. The start-up recovery also probed the mark twice. The answer can change between two probes, and a second answer of "cannot tell" after a first of "retired" dropped through to opening the very directory the first answer said not to open. One probe, matched exhaustively. An unanswerable "is the environment there" is no longer read as "it is not". The retirement blocker already read that failure as "there is one"; the classifier that decides whether the work needs a person read it the other way, so the node kept the shared volume for the six-hour cap and said nothing an operator would see. Two folds in the LMDB store, on the same theme. A delete whose growth could not be measured was charged nothing, so a copy-on-write delete could spend disk the budget never saw and the reserve stopped meaning anything; it is now charged the whole slack, which costs at worst one assisted delete. And sizing the map read every metadata failure as an empty database, which on a node with a large one produces a map too small to open it; only a missing file means empty now.
…was measuring nothing CI reported the index costing zero bytes per chunk, and the test passed. Resident memory is process-wide and the allocator hands back what earlier work freed, so opening a store in a process that has already opened and dropped one of the same size grows the resident set by nothing at all. Adding the three reopens the scan comparison needs is what tipped it over: from that point the test measured the allocator rather than the index. It now runs in a child that has done nothing else and so has no freed heap to reuse, and it refuses a reading of zero. A gate that cannot tell the difference between an index that costs nothing and a measurement that happened not to be taken is not a gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear issue
V2-1033
Risk tier
Compatibility
{root}/chunks.mdbto{root}/chunks/<xy>/<64-hex>.A node that runs this and then downgrades finds its chunks in a directory the old build
ignores. While the legacy store still exists every new chunk is written to it as well, so
a downgrade at that stage loses nothing. After a node has removed its legacy store there
is no downgrade. This release does remove it, once every gate below is satisfied, because
removing it is the only thing that returns disk.
ChunkStore/ChunkStoreConfigare added;LmdbStorage/LmdbStorageConfigare still exported.
StorageStats,MIBandGIBmove fromstorage::lmdbtostorage.Semver impact
Test evidence
One caveat on the lint evidence: this machine runs clippy 1.95 and CI runs 1.98. A lint was
renamed between them, so a local
-D warningspass is not by itself proof CI will pass.What CI proves on every commit
Four harnesses, run on Linux, macOS and Windows. The three that touch durability run again
on ext4, XFS and btrfs loopback volumes, because how a filesystem handles a rename, a lock,
a deletion and the space behind it is its own business and btrfs in particular accounts for
it differently.
migration_reclaims_diskmigration_crash_safetymigration_shared_volumemigration::run; the lock held through retirement as well as copyingtake_volume_lockfrom either branch of the driverstorage_scalestateach entry; write a sidecar per chunkTwo of those deserve saying out loud, because the obvious version of each test does not
catch them. A
statbehind every filename costs about three times a bare walk and stayswell inside any flat ceiling loose enough not to flake, so the scan is measured against the
machine instead: the same directory walked twice in the same process, once reading names and
once calling
metadataon each entry, and the scan has to land on the names-only side ofthe two. Adding the
stattakes it from 88 ms to 296 ms against a 123 ms midpoint. And theretirement crash is the most destructive moment in the migration, because by the time the
mark is written the node has already told the network it serves those chunks from the file
store: the next start has to finish the deletion, never reopen the directory.
Measured rather than asserted, and printed in the log so drift shows before it trips a gate:
the startup scan of 100,000 chunks takes 102 ms (1,019 ns/key), the in-memory index costs
52 bytes per chunk, opening the store reads 125 bytes whatever the chunks contain, and
putwrites exactly one directory entry per chunk.Two things these deliberately do not prove, and are not offered as proving. Killing a
process keeps the kernel page cache, so the loopback jobs would stay green with every flush
removed from the publish path: real power-loss behaviour is still a fleet gate. And 100,000
keys is what a hosted runner can do in reasonable time, so where the curve stops being linear
at 1M and 10M is still a question about a machine, not about the code.
Each harness step fails if it runs no tests, and a separate check proves a shipped binary
carries no failpoint: the environment variable name is in the binary with
--features test-utilsand absent without it.What the tests actually prove, rather than merely exercise: publish is exactly-once under
sixteen concurrent writers of one address; the index rebuilds from the filesystem with a
stable order across restarts; a file in the wrong shard, an uppercase name and a non-regular
file are all refused; an interrupted write is swept; a corrupt file is removed and served
from the legacy copy instead, with the key re-queued; the copier is resumable and cannot
resurrect a pruned chunk; committing narrows the commitment without narrowing what is
served; a read in flight blocks retirement and retirement completes as soon as that read
finishes, and the chunk stays readable throughout; the shipped configuration retires with
nothing set by hand; a fully built node holding a legacy store is actually migrating it; retirement is refused for each gate independently; verification repairs a file
that rotted and refuses to retire when it cannot; a node with no view of the network gives up
nothing; the commitment-delivery counter resets on rotation; suffix sharding reaches all 256
directories for a clustered key set where prefix sharding collapses to one.
Adversarial review: twenty-eight rounds of codex at xhigh, plus three independent
reviewers. The last round found nothing at any severity and concluded the branch is ready
to begin a gated rollout. The findings that mattered were all on the destructive path, and
all of them were cases where a node could end up with no copy of a chunk it was supposed to
have:
The last four rounds found the same defect three more times, at successively smaller
scales: a filesystem answer that is neither yes nor no, folded into one of them, and acted
on later as though it had been established.
retired environment whose mark could not be read looked live, went back under its own
name, and its keys re-entered a commitment they had already left. It is three states now,
and the two questions callers ask are asked separately: deleting needs a mark that was
read, opening needs one known to be absent, and neither takes "cannot tell" for a yes.
permitted and let every other answer fall through to the opposite action, so "cannot tell"
still reached the opening path.
create_new, and a failure saying something is already at thatname was taken as proof the mark was there. What is at that name might be anything. Every
other part of this code insists the name is not the evidence; this was the one place
taking it.
ordinary path it was never asked: a node holding its store open went through every gate,
renamed the directory aside and deleted it. It is now the first question asked, before
anything else.
answer of "cannot tell" after a first of "retired" dropped through to opening the very
directory the first answer said not to open.
Two of the same shape in the LMDB store: a delete whose growth could not be measured was
charged nothing, so a copy-on-write delete could spend disk the growth budget never saw;
and sizing the map read every metadata failure as an empty database, which on a node with a
large one produces a map too small to open it.
And from the earlier rounds:
not yet reached the copy that would replace it. Reads and retirement are now a shared and
exclusive guard rather than a race, verified by removing the guard and watching the test
fail.
because a name already on disk returned early without flushing. Every successful return now
flushes, and the same applies to creating a shard directory and to moving the legacy store
aside before deleting it.
the protocol handler and the file store answered from names alone. Both compare the length
first now.
group, so the last two holders could each conclude the other had it.
failed to start ran on forever with two stores. A test now builds a real node with a legacy
store and asserts it is migrating it; deleting the spawn turns it red.
directory back with its contents already deleted and the node would fail to start on it.
The removal is now recorded durably before anything moves, with the same create-and-flush
that publishes a chunk. A start that finds that record opens the environment to decide:
one that opens is intact and is kept, with the migration starting again from copying; only
one that cannot be opened is treated as the remains of an interrupted removal.
file store answered "already have it" from names. A good copy offered to repair a damaged
chunk was thanked and discarded, and nothing offered another. Both read and compare now,
and repair from the offer.
store at another volume leaves a link there, and retirement would have deleted the
contents of a directory that is not this node's to delete. A linked environment is copied
out of and never retired.
for half an hour, because retirement is usually deferred by a gate unrelated to the files.
A kept chunk that stopped being readable in that window was invisible: ordinary requests
were still served from the legacy copy. The store counts the times a chunk stops being
servable, a proof records that count, and retirement refuses one the store has outrun.
Several of those were introduced by earlier rounds' own fixes, which is the argument for
running the loop rather than stopping at the first clean round. Holding a legacy handle across
every read closed a race but meant retirement would never see the environment unreferenced
on a busy node; taking the resulting guard on reads alone let writes and deletes starve the
drain instead; holding it through the deletion would have stalled every chunk request on the
node for as long as
remove_dir_alltook. The guard is shared for reads, writes and deletes,exclusive for retirement, and released once the handle is out and the directory is renamed
away.
Four findings are accepted rather than fixed, each with its reasoning, in the ADR.
New dependency
none
ADR
https://github.com/grumbach/ant-node/blob/storage/file-chunk-store-and-lmdb-retirement/docs/adr/ADR-0013-file-based-chunk-store-and-lmdb-retirement.md
What the review changed, and what it left
The last five rounds found no blockers. What they did find, repeatedly, was one shape of
defect: a fact established at one moment being acted on at another. Copying, verifying
and retiring are hours apart by design, and the physical work runs on blocking threads that
outlive the futures that started them, so a caller that goes away does not undo what it
started. The answer that worked was to make each belief carry its own expiry rather than to
re-check and hope the check lands close enough to the act: the retired directory carries its
own mark, a verification proof carries the store's health count and is refused once the store
outruns it, a write announces itself before it starts and is cleared by the worker rather
than the caller, and a delete waits out whatever is already writing its key.
Two decisions were taken against a reviewer's first suggestion and are worth stating:
until something reads it, and the first read settles it. Persisting means another file to
keep in sync with its own failure modes. The pre-retirement pass reads every chunk before
anything is deleted, which closes the window during the migration itself.
class above. Its key property, that the worker owns the bookkeeping rather than the caller,
is applied where the defects actually land. The final round agreed the rewrite is
unnecessary.
Mitigation / rollback
storage.migration.enabled = falsestops the copier at any point; the node keeps readingboth stores and simply never frees the old one's disk.
ANT_MIGRATION_RETIRE_LEGACY=0leavesthe copier running but holds the removal off, per node, without a separate build.
Before a node removes its legacy store, reverting is a higher-semver build of the previous
code, and every chunk written meanwhile is in the legacy store too. After a node has
removed it, there is no rollback for that node. That is what the staged rollout, the 72-hour
shed hold, the wave stagger and the four-hour retention delay are all for: at any point the
fleet is a mix of nodes that have removed it and nodes that have not, and the ones that have
not are the rollback.
The problem
LMDB returns a deleted page to its own free list and never to the filesystem. Last week the
fleet deleted 2.29 million chunks and got back zero bytes. Operators read that as a bug and
are tempted to wipe node directories, which costs the network real replicas. Compaction does
not help: it needs free space equal to the live data, which is exactly what a full node does
not have, and it leaves us on LMDB. Disk comes back exactly once, when
chunks.mdbgoes.Why the last two hex characters and not the first
A node holds keys it is among the closest to, so its holdings share roughly
log2(N / close_group_size)leading bits with its own node ID, and that prefix grows asthe network grows.
At today's fleet a two-hex prefix is already about two directories. Prefix sharding does not
degrade, it fails, and it fails later for the nodes that grow into it. The address is a
BLAKE3 output and close-group membership constrains only its leading bits, so the trailing
byte is uniform by construction at any size. IPFS shipped the same fix for a different reason
and their
_READMEstill says so.Why no index database
A persistent index cannot remove reconciliation. Commit the index first and a crash leaves a
phantom key; rename the file first and a crash leaves an unindexed file. Repairing either
means reading the filesystem anyway, so the filesystem may as well be the authority, and then
nothing can drift. Ceph FileStore's tracker #17177 is what the other choice looks like.
How a full node is handled
A node that fits its payload copies everything and then removes the old store. It is never
unable to serve.
A node that does not fit copies closest-first, commits to what it can hold while still
serving everything it ever committed to, and only then gives the rest up. Nothing is given up
until three things hold, in this order:
neighbour sync that carried it, because until then they audit it against the set it used
to hold;
challenge over a nonce it has never seen, and is itself currently publishing a
commitment.
Point 3 is the pruner's own evidence. The cheap
VerificationRequestis deliberately notused: it carries a self-reported
presentflag, and a node that silently lost a chunk stillanswers yes.
Close groups migrate in waves derived from a hash of each node's ID, about two of seven
at a time. If every holder went at once, none could prove to the others that a copy survived
and the group would deadlock. A host-wide lock separately serialises nodes sharing a volume,
which is a different question: one machine's disk rather than one chunk's replicas.
Before the old store goes, every chunk both stores hold is re-hashed and rewritten from the
legacy copy if they disagree, because a filename is not proof the bytes behind it are good.
What this costs, plainly
control.
ANT_MIGRATION_RETIRE_LEGACY=0holds a node before the removal if one is neededto keep both copies.
could not be written. The environment is pinned at its current size for the whole bridge,
so a write it cannot satisfy from its own free list is refused and the chunk lives in
files alone. That is deliberate: two stores measuring one disk can otherwise spend the
same free space twice and fill the volume this exists to free. A node that must keep an
old-binary rollback available should have retirement held off.
disk, and will tell its operator to add storage. That is the right answer under "no data
loss", but the migration will not complete unattended everywhere. How large that population
is, is the main thing to measure once this is out.
count, so a node that has just proved it is short of disk quotes cheaper and then refuses
the store on capacity. A wasted round trip, not a mispayment. The fix belongs to the quote
path.
heedcannot be dropped yet.