Skip to content

Exercise ownership through real partitions and owner VM loss - #35

Open
jeregrine wants to merge 1 commit into
mainfrom
test/distributed-ownership-failures
Open

Exercise ownership through real partitions and owner VM loss#35
jeregrine wants to merge 1 commit into
mainfrom
test/distributed-ownership-failures

Conversation

@jeregrine

Copy link
Copy Markdown
Member

Problem

In-process storage faults and a connected peer do not establish what happens when a real owner VM disappears or loses its quorum. The suite lacked direct checks that acknowledged state survives those failures, that a minority cannot acknowledge a durable write, and that old ownership tokens remain fenced after healing.

Fix

Add a focused, opt-in three-node EKV-backed ownership lane:

  • Concurrent starts across all three nodes must converge on one owner; concurrent synchronous increments must return distinct values and agree with consistently read durable state.
  • Isolate the owner from the majority, require its attempted write to receive no success acknowledgement, recover on the majority, and verify the previously acknowledged state. After healing, require the old process to be gone and its captured ETag to fail a conditional write without changing the recovered state.
  • Abruptly halt the owner VM without terminate callbacks, recover on the remaining quorum, and continue from the last acknowledged counter value.

Peer control runs over stdio independently of Erlang distribution. Per-node cookies prevent accidental reconnection during partitions; no host network rules are modified. Nodes and worktree-local data directories are uniquely named and cleaned up on exit. A dedicated permanent counter keeps the recovery contract explicit.

Supporting information

These are narrow, black-box failure scenarios, not a general history/linearizability checker or a deterministic simulator. An interrupted call is recorded as indeterminate rather than assumed uncommitted. This does not replace future generated operation histories or broader fault schedules.

The lane is tagged distributed and integration, keeping it out of the default suite. Its storage is entirely local EKV, but main's global test helper still contacts LocalStack; the independent fixture-isolation change in #33 removes that unrelated dependency. CI must explicitly select the distributed lane to exercise it.

Only tests, test support, and documentation change; no production behavior is relaxed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant