[WIP] CAS draft (adopting to CI/CD, not for review / merge) - #2073
Draft
filimonov wants to merge 4422 commits into
Draft
[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073filimonov wants to merge 4422 commits into
filimonov wants to merge 4422 commits into
Conversation
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
…ent_addressed stateless lanes) Config Workflow check failed with 'Workflows are outdated' for master.yml, pull_request.yml, pull_request_community.yml, release_builds.yml. Regenerated via 'python3 -m praktika yaml'. The regeneration adds the two CAS stateless jobs to the generated workflows: 'Stateless tests (arm_binary, content_addressed storage, parallel)' and 'Stateless tests (arm_binary, content_addressed s3 storage, parallel)' (the rustfs-backed lane). CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=927ea142c9cb14759623861eb004261d0b4b1c8f&name_0=PR&name_1=Config+Workflow PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
…teless lane The lane's start_rustfs expected a pre-extracted binary at ci/tmp/rustfs and failed on CI runners where nothing provisions it (the workflow wipes ci/tmp on every run). Download the static musl build for the runner architecture from the RustFS GitHub release (1.0.0-beta.9) when the binary is absent, mirroring how setup_minio.sh downloads minio/mc. Validated locally: the beta.9 binary passes the conditional-operation semantics the CA pool requires (second 'If-None-Match: *' PUT -> 412, wrong-etag conditional DELETE -> 412, right-etag DELETE succeeds), and download_rustfs provisions an executable binary end-to-end. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
Fast test fails at cmake generation: 'Target "dbms" links to ch_contrib::crc32c but the target was not found' — the fast-test job initializes a limited submodule list that does not include contrib/crc32c, so the unconditional add_contrib is skipped while the dbms link line still references the target. The dependency is dead: it was wired in for per-block CRC32C in the early CAS run-file format (5f1272c), which was later replaced by the text record-stream codecs; no source file includes the library today. Restore the pre-CAS state: crc32c is built only for google-cloud-cpp, and dbms does not link it. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=835251f81cb5af73ad9eaa3a835f50f0c8b678db&name_0=PR&name_1=Fast+test PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
Fast test builds without SSL and failed on the unconditional 'openssl/evp.h' include in CasBlobHashingWriteBuffer.cpp. Wrap the OpenSSL-backed Sha256 hashing write buffer and the one-shot digest in '#if USE_SSL'; on non-SSL builds selecting blob_hash = 'sha256' now fails closed with SUPPORT_IS_DISABLED. CityHash128 and XXH3-128 blob hashes are unaffected. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
… test regression) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Fast+test PR: #2073 A CAS parser commit grouped `RELOAD_DICTIONARY`/`RELOAD_MODEL`/ `RELOAD_FUNCTION` with `CONTENT_ADDRESSED_GARBAGE_COLLECTION` into a format case that prints only the optional disk, dropping the reload targets: `SYSTEM RELOAD MODEL my_model` formatted as `SYSTEM RELOAD MODEL` (failed 04117_parser_system_query_variants and 04124_parser_system_query_extra in Fast test). Fold all four types back into the generic target-printing case (table / target_model / target_function / disk else-if chain) — for the CA GC command the disk branch produces the identical output. Both stateless tests verified locally via clickhouse-local against their references. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
…als (arm_tidy, T13 batch 1) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Removes default arguments from all virtual/override methods flagged by `google-default-arguments` (147 sites: `CasBackend.h` interface, `IObjectStorage.h`/`S3ObjectStorage.h`, all backend implementers, test helpers/fixtures) and adds non-virtual convenience overloads on the base classes that forward the previous default values. Derived classes gain `using` declarations to unhide the base overloads. Qualified parent-implementation calls in test fault backends switched to the explicit 3-arg form — the 2-arg form would now route through the base forwarder and re-enter the derived override virtually (double fault injection; caught by the battery). Bulk edits produced by codex (gpt-5.6-luna) per the T13 brief; overload visibility and qualified-call fixes plus verification by Claude. Battery 919/919 green. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Semantics-preserving conformance for the remaining flagged classes: readability-container-contains, readability-isolate-declaration, google-runtime-int (AWS SDK retry-API overrides keep `long` with targeted NOLINT — the override contract owns the type), readability-duplicate-include, cppcoreguidelines-init-variables, cert-msc, modernize-raw-string-literal, modernize-use-starts-ends-with, bugprone-empty-catch (comments only — no new behavior), googletest naming, bugprone-argument-comment, bugprone-optional-value-conversion, bugprone-misplaced-widening-cast (CasTypes.h site audited: not a real precision bug — the value is range-validated to 0-5; cast made explicit without value change). CasRefCowMap's own `contains` keeps its `find` with NOLINT (self-recursion). Bulk edits by codex (gpt-5.6-luna) per the T13 brief (.superpowers/sdd/task-13-batch2-report.md); one over-removed include (PartFolderAccess.h) restored and verification by Claude. Battery 919/919. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 21, 2026
…n't abort the server (STID 0883) CI PR#2073 (content_addressed storage lanes) crashed with "Too large size (9223372036854775870) passed to allocator" (LOGICAL_ERROR, server abort) running the regression test 04070_no_crash_extreme_compress_block _size. Root cause: an extreme max_compress_block_size (2^63-1) flows into ContentAddressedTransaction::writeFile's buf_size and, unclamped, reaches the CaContentWriteBuffer base-class allocation (Memory::alloc), where checkSize (>= 0x8000000000000000) fires. The ordinary MergeTree writers clamp compress -block sizes to 256 MiB (MergeTreeWriterSettings::MAX_COMPRESS_BLOCK_SIZE) for exactly this reason; the CAS write path received the value unclamped. Fix: clamp buf_size and adaptive_write_buffer_initial_size to 256 MiB at the CAS allocation site (both CaContentWriteBuffer ctors), mirroring the ordinary clamp. New gtest CasContentWriteBuffer.ExtremeBufferSizeIsClampedNotPassedToAllocator reproduces the exact crash number without the clamp (verified RED) and passes with it. CA gtest gate 1057/1057. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=latest&name_0=PR Related: #2073
filimonov
added a commit
that referenced
this pull request
Jul 21, 2026
…cle (UNMOUNT/FSCK) Design for STID 3982-3b48 (CI PR#2073). Five parts: (1) MountLeaseKeeper's vanished-backing-store case throws FILE_DOESNT_EXIST (not LOGICAL_ERROR) so a background renewal thread never aborts the server + new CasMountLeaseLost counter; (2) SYSTEM CONTENT ADDRESSED UNMOUNT <disk> (shutdown() the pool, auto-remount on next access) as a clean alternative to rm -rf under a live mount; (3) online SYSTEM CONTENT ADDRESSED FSCK <disk> via read-only runFsck(Pool&); (4) rewrite the no-leftovers test teardown to GC RUN -> FSCK -> UNMOUNT -> rm -rf; (5) rename the offline clickhouse-disks fsck applet to ca-fsck for consistency. Auto-teardown-on-DROP stays out of scope. Related: #2073
filimonov
added a commit
that referenced
this pull request
Jul 21, 2026
…ence Task 7 added the SYSTEM_CONTENT_ADDRESSED_FSCK AccessType but only updated the AccessType.h macro list, not the 01271_show_privileges reference — the test enumerates every privilege, so the new row (right after MOUNT, matching the AccessType declaration order) was missing and fasttest failed with a one-line diff. 01271 was not in the local gtest/FSCK-access test runs, so the gap surfaced only in CI. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=d780fa7473807203916125e96cf73a9774c84464&name_0=PR&name_1=Fast%20test PR: #2073 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
filimonov
added a commit
that referenced
this pull request
Jul 24, 2026
Add two DESIRABLE items under §3 (GC correctness/observability) surfaced by triaging the CAS-s3 stateless run (Altinity PR #2073, run 30019911967): - [RECOVERED-INDEGREE-ATTRIBUTION] the "delete_pending recovered in-degree — structurally impossible … investigate" warning is a false alarm (dedup-adopt vs condemn TOCTOU, spared, no data loss). Downgrade the GC log to a ProfileEvent + Debug and move the real adopt-without-resurrect detector to the writer's edge-commit. - [CONDEMN-GRACE-WINDOW] a cool-down before condemning a just-zeroed blob to kill hot-dedup churn (tiny system-log blobs) at the source; flagged higher-risk (condemn-timing/ack-floor, TLA-gated, protocol-veto). Docs-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jul 24, 2026
…GABRT) CI-confirmed on Altinity PR #2073 (run 30019911967, asan_ubsan CAS-s3 stateless): a mount-lease renewal PUT that times out client-side (ambiguous — may have applied server-side) gets soft-retried with a stale token, mismatches against its own bumped body, and falls through MountLeaseKeeper's classifier (same uuid/epoch, unfenced — none of the 3 cases match) into the base class's generic LOGICAL_ERROR, aborting the server under ASan. A third variant of the STID 3982-3b48 family (parts 1a/1b covered vanished/absent-at-release). Also notes: amd_msan/amd_tsan CAS-s3 stateless jobs in the same run hit the 6h job timeout with zero artifacts — unknown if same crash-loop or a plain hang. Docs-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jul 27, 2026
The unsharded CA-s3 sanitizer lanes do not fit the 6h GitHub job timeout, and the kill lands after the test loop but before result upload, so the lane reports NO artifacts at all and praktika's aggregate stays RUNNING forever. In run 30203355812 (sha e2d04bf): tsan finished all 10990/10990 tests at 5h59m and was killed 16s later during teardown; msan was at 5824/10990 (53%, steady progress, no hang) when killed; asan_ubsan passed at 5.4h — too close. This is the full explanation of the "msan/tsan CA-s3 cancelled at exactly 6h with zero artifacts" pattern seen in three consecutive runs. Shard asan_ubsan and tsan 2 ways and msan (the slowest, ~11h projected) 3 ways, following the existing "amd_tsan, s3 storage, parallel, N/M" convention; ci/jobs/functional_tests.py already parses the N/M batch token generically. Workflow YAML regenerated with praktika yaml. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
filimonov
added a commit
that referenced
this pull request
Jul 27, 2026
…irectory The refusal in prepareAdoptFromManifest throws LOGICAL_ERROR, which aborts the whole process in debug/sanitizer builds instead of behaving like a catchable exception, so the EXPECT_THROW form killed unit_tests_dbms on all three sanitizer CI lanes (asan_ubsan/tsan/msan, 0s failures with "Logical error: 'Relink target ... does not address a content-addressed part directory of a live table'"). Split it the same way as the CasWiringOpsDeathTest precedent in this file: EXPECT_THROW stays for plain release builds, and a CasWiringExchangeDeathTest EXPECT_DEATH variant proves the same refusals positively abort under DEBUG_OR_SANITIZER_BUILD. Verified: build_asan runs the death test (1 test, OK), build (release) runs the EXPECT_THROW variant (9 tests from 2 suites, all OK). Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
filimonov
added a commit
that referenced
this pull request
Jul 27, 2026
The pre-scrape cleanup removed only filesystem_caches/*/status (one level), but dynamic cache disks created by tests nest their path — the binary CA-s3 lane died on filesystem_caches/disks/cache_03517/status with "Cannot lock file ... Another server instance in same directory is already running" (StatusFile.cpp flock, EWOULDBLOCK) raised from FileCache::initialize when clickhouse-local initialized the disk map. Make the glob recursive so any depth is covered, same mechanism as before: removing the file lets clickhouse-local create and lock a fresh inode even when the not-fully-stopped server still holds the old one. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
filimonov
added a commit
that referenced
this pull request
Jul 27, 2026
…lanes Tag it no-content-addressed-storage: the coverage is the inline content-addressed disk the test creates itself, so it stays meaningful on every ordinary lane. On lanes whose DEFAULT MergeTree storage is content-addressed, system.remote_data_paths (no disk_name pushdown — the applyFilters TODO in StorageSystemRemoteDataPaths.cpp) also walks the huge shared default pool with the whole run's data, and on the S3 (RustFS) variant that walk exceeds the 600s test timeout (recurred in two consecutive runs, plus a "Some queries hung" ride-along). The BACKLOG pushdown item stays open as the real fix; also note the ConnectionGroup Disk-session pressure signal from the same run. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
filimonov
added a commit
that referenced
this pull request
Jul 27, 2026
Same class as 5f292e0: PreparedPartWrite's one-shot-terminal rejection throws LOGICAL_ERROR, which aborts the whole process in debug/sanitizer builds (Exception.cpp's handle_error_code) instead of behaving like a catchable exception, so CasPartFolderAccess.PreparedPartWriteRejectsASecondTerminal killed unit_tests_dbms on all three sanitizer CI lanes (0s failures). The file's author knew the class (the NETWORK_ERROR/MEMORY_LIMIT_EXCEEDED choices further down cite it) — these two spots were the leftovers: - PreparedPartWriteRejectsASecondTerminal: split per the CasWiringOpsDeathTest precedent — expectThrowsCode stays for plain release builds, a CasPartFolderAccessDeathTest EXPECT_DEATH twin proves the same rejections positively abort under DEBUG_OR_SANITIZER_BUILD. - PreparedPartWriteMoveTransfersTheTerminalDuty: the single moved-from abort check is guarded in place (EXPECT_DEATH forks, so the rest of the test continues in the parent). A sweep of all CAS gtest files for unguarded LOGICAL_ERROR throw expectations found no other gaps (blob_upload_pool, ref_writer, ref_install_safety, upload_detached, upload_fanout all carry guards). Verified: build_asan 39/39 with both death tests OK; build (release) 30/30 with the EXPECT_THROW variants. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=8417137aeb842f463066fd68a75b60ffeae30d63&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
filimonov
added a commit
that referenced
this pull request
Jul 28, 2026
CasRefInstallSafetyDeathTest.DenyGuardStopsAnAllocation dispatched on DEBUG_OR_SANITIZER_BUILD, but DENY_ALLOCATIONS_IN_SCOPE itself is gated on MEMORY_TRACKER_DEBUG_CHECKS (MemoryTracker.h: defined only under !NDEBUG). Sanitizer builds define NDEBUG, so there the guard compiles to static_assert(true) and the death test "failed to die" on all three sanitizer CI lanes — visible verbatim in the CI error output. (These lanes only now reached this test at all: the earlier LOGICAL_ERROR aborts in gtest_ca_wiring / gtest_cas_part_folder_access killed the process before it, so each fix un-shadows the next tail failure.) Gate the death test on MEMORY_TRACKER_DEBUG_CHECKS instead, and drop the throw-only EXPECT_ANY_THROW branch as dead code: MEMORY_TRACKER_DEBUG_CHECKS implies !NDEBUG implies DEBUG_OR_SANITIZER_BUILD, so whenever the guard exists its LOGICAL_ERROR aborts — death is the only observable outcome (the old comment claimed the opposite implication). Verified: build_debug runs the death test (OK, guard fires); build_asan compiles it out, remaining CasRefInstallSafety* 21 tests x5 repeats green. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=07f8398acddff2c676aded5f804f122984845a4a&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
filimonov
added a commit
that referenced
this pull request
Jul 28, 2026
RCA of the fence cascades on the CA-s3 sanitizer lanes (run for 07f8398, msan 2/3 lane): clickhouse-server logged 351 bursts of 'Connection refused' (e.code() = 111) to rustfs at localhost:11121 spread over the whole run — the service stayed up between bursts, so this is fd/accept exhaustion, not a crash. rustfs was launched WITHOUT the open-files-limit raise that start_azurite performs for exactly this failure mode, while the server under parallel sanitizer load holds 10k+ active Disk-group S3 sessions. Most bursts were absorbed by the CAS request retries (max_attempts=16 / 90s deadline); the one at ~02:00-02:01 outlived (a) in-flight writes' 90s budgets (Code 210 UNCERTAIN) and (b) the mount-lease renewal confirm window (TTL 30s), so the mount fence tripped (fail-closed by design) and every writing test failed with Code 668/210 from 02:02:52 until remount completed ~02:05:00, with straggler failures to ~02:10 — ~40 of the run's 66 test failures. Also add rustfs.log to the uploaded artifacts: it was already written to ci/tmp/rustfs.log but never uploaded (azurite/kafka/minio logs are), which was exactly the missing evidence for this class of triage. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=07f8398acddff2c676aded5f804f122984845a4a&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
ilejn
reviewed
Jul 28, 2026
|
|
||
| ```sql | ||
| SYSTEM CONTENT ADDRESSED GC RUN [ON CLUSTER cluster_name] [disk_name] | ||
| ``` |
filimonov
added a commit
that referenced
this pull request
Jul 30, 2026
… razor RefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts failed on 5 of 6 sanitizer-lane runs across two CI rounds (asan/tsan/msan, 'refLaneWedgedForTest — Actual: false, Expected: true') while passing every quiet local run. Reproduced locally 14/14 by adding full CPU load with the test otherwise unchanged and filtered alone — so not test-ordering contamination, pure timing. Root cause: the test set attempt_timeout_ms == operation_deadline_ms == 100, which turns the request controller's pre-send gate (putIfAbsentControlled: 'now + attempt_timeout > deadline' returns Unresolved WITHOUT sending) into a zero-width race — it only passes when no millisecond tick elapses between the deadline capture and the gate. This test uniquely burns that window encoding the ~1700-op removal chunk, so on a loaded or sanitizer-slow machine the gate fired first, the injected ambiguous fault was never reached, the sweep failed CLEAN — nothing sent, nothing ambiguous, and the product CORRECTLY did not wedge the lane. The test had over-specified the failure mode it would meet. Ambiguity is guaranteed by max_attempts = 1 alone; widen operation_deadline_ms to 5000 so the PUT is always actually sent and the wedge is deterministic on any machine. Verified: 20/20 green (5 repeats x 4 sweep tests) on ASan under half-core CPU load; the sibling wedge tests keep the tight budget deliberately — their capture-to-gate window is empty. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=834c9517f56a83927b0c941ac08d8dfd82460579&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
svb-alt
added
antalya
antalya-26.6
roadmap
strtgbb
requested changes
Jul 31, 2026
filimonov
added a commit
that referenced
this pull request
Aug 3, 2026
… (PR 2073 review) Addresses the three review recommendations from strtgbb on #2073 - All 10 CAS `Job.ParamSet` entries move out of `JobConfigs.functional_tests_jobs` into a new `AltinityJobConfigs.cas_functional_tests_jobs`. Every workflow that picked them up implicitly -- including `release_builds` (`binary` filter), `release_branches` and `backport_branches` (`asan` filters) -- now concatenates the two lists in the original order, so the job sets are unchanged. Proven by an empty `diff -r` of the regenerated `.github/workflows` and by an identical dump of `workflow.jobs` names for every `ci/workflows/*.py` module before and after. - The CAS ParamSet comments are cut to the load-bearing constraints: RustFS rather than MinIO OSS for enforced conditional deletes, and sharded sanitizer lanes because an unsharded one exceeds the 6h GitHub job timeout. - The CAS comments in `ci/jobs/scripts/clickhouse_proc.py` are compressed the same way, keeping why the disks are opened read-only, why the substitution is keyed on the `<metadata_type>cas</metadata_type>` marker, and why `grep -R` and `sed --follow-symlinks` are required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
filimonov
added a commit
that referenced
this pull request
Aug 4, 2026
CASRefCheckpointJoin.EncodedCkptSizeIsIndependentOfCardinality and CASRefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts failed identically on all three sanitizer unit-test lanes (asan_ubsan, tsan, msan) at 9ad3e15, thrown from ordinary product code, not a sanitizer report -- both are the same class of bug `8f9e63c7a19` already fixed once for the second test: a CPU-bound encode step (committing `MANY_REFS`/ ~`ref_txn_max_ops` ref ops in one call) measured against a real-wall-clock deadline (`mount_lease_ttl_ms` for the first test, `operation_deadline_ms` for the second), which a slow-enough sanitizer build -- msan above all -- can outrun regardless of how wide the deadline is set. Both the mount fence and the ref-log request controller (`CasRefLedger`'s `controller_boot_ms_fn`) already read time through `PoolConfig::boot_ms_fn`, the same injectable seam a few lines below already uses (the `resumer` pool in the second test). Freeze it for the pools that do the CPU-bound work instead of widening the deadline further, removing the race rather than giving it more room. Not build-verified: the shared dev box was at 63 GB free disk (existing full sanitizer builds run 43-51 GB each) and near swap exhaustion, so no fresh build was attempted. Verified by tracing the exact production wiring (PoolConfig::boot_ms_fn -> CasRefLedger::controller_boot_ms_fn -> CasRequestController::now_ms, and CasMountRuntime::bootMsNow) and matching the pattern already in use for this file's `resumer` pool. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=9ad3e15b68823633cc1c2a1539292f20de3e7b5e&name_0=PR PR: #2073 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
…spec Review found two defects in rev.2's own construction. `GcMetaWriter` cannot be a direct member of `Gc`. Its pool size comes from `store->poolConfig()`, and a direct member is initialized before the constructor body, where `Gc::Gc` validates a null store, a zero `gc_id` and a zero `gc_stuck_removal_rounds`. It would dereference a null store before the first of those checks, breaking behaviour that `CASGCLease.CtorFailsClosedOnBadArguments` pins. It is held by `unique_ptr` and created in the constructor body, exactly as `meta_pool` is today and for the same documented reason. The destruction test deadlocked as sequenced: the pool's destructor joins its workers, so a thread that destroys the `Gc` and only then releases the backend latch blocks inside the join forever. The release is now sequenced by destruction order instead of by a second thread -- the harness member declared after the `Gc` is destroyed first and opens the latch, so destruction has provably begun before the job proceeds, and the release precedes the join on the same thread. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rev.3 replaced a deadlocking teardown test with a destruction-ordered latch release and called the result deterministic. It is not: the harness member that opens the latch is destroyed before the `Gc` it sits next to, so the released worker may finish before `~Gc` runs at all, and a closure capturing `this` would still pass. The claim is withdrawn rather than patched. Adding an observable notification inside the destruction boundary would mean writing a destructor purely for a test hook, on the very class whose lack of a hand-written destructor is the property being asserted -- and it would buy a detector that still cannot fail, since the pre-change access is to destructed-but-allocated storage that no sanitizer reports. The test is therefore stated as functional coverage: it exercises the pool's join against an in-flight job and asserts the write landed, and it claims nothing about ordering. The lifetime regression guard is named as the only thing that actually is one -- the absence of an entry point accepting a callable. Also corrects the B2a count: five of the eight branches need an in-window mutation, not four; fenced-during-adoption needs the seam too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four tasks against the rev.4 spec: extract `GcMetaWriter` and delete the callback-accepting entry point onto the meta pool, drain that pool on a round's throwing exit, reclassify the six `claim` conflicts to `ABORTED` with a test per branch, and gate the result on debug and ASan builds. The plan marks which steps have a real red-then-green cycle and which do not: the round-exit drain does, asserted as "the round must not return while a meta job is held", which is the only formulation that does not race the job's completion. The ownership test is stated as functional coverage, since the pre-change access is to destructed-but-allocated storage that no sanitizer reports; its guard is a grep for the deleted API. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found the plan not executable as written; every finding checked out against the code. Two harness blockers. `deleteConfirmedMeta` returns immediately when no meta object exists, so the confirmed-delete test waited forever for a job that never reached the latch -- the meta is now seeded first, and the spin loop is replaced by a bounded wait that fails loudly. And the latch blocked every blob-meta mutation from construction, including the fixture's own part write, so it now starts disarmed and is armed once the fixture is built. The drain test's fixture was also unsound: the outcome log is written from settled entries, not from fresh condemns, so a round that merely condemns writes none and the fault never fires. The fixture now drives rounds until `previewDeletes` reports a pending delete and faults the round that actually deletes -- the one that schedules a confirmed-meta delete and writes an outcome log, in that order. The test additionally separates the two ways a timeout can lie, since blocking in the pre-existing protocol wait looks the same as blocking in the drain. Three coverage and compliance gaps: the ownership tests asserted no effect at all and now assert durable Condemned meta, registry visibility after a drain, and actual removal; the six conflict tests used a code-only assertion for six branches that now share one code, and get a message-substring helper; and the file lacked the required frontmatter and heading anchors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers item 10 of the CAS pre-release checklist -- findings B3 and B4 of the second 2026-08-05 umbrella review -- plus the third recommendation triage made and the item text omits, a `~Pool` that cannot terminate the process. That one is in scope because the drain increases its exposure: `~Pool` today runs late and rarely on the shutdown thread, and afterwards runs there on every shutdown, so fixing the timing alone would raise the frequency of the crash it removes. Teardown unpublishes the pool before waiting on anything, since `poolAccess` hands out a strong reference under `pointer_mutex` and a wait taken while the pool is still published proves nothing. The guarantee is stated narrowly and honestly: a successful drain establishes that no TRACKED detached work holds the pool, not that `~Pool` runs on the shutdown thread, because a `PoolPtr` handed out earlier can outlive shutdown. The stop is observed at four points, not one. `settleSnapshotPublish` re-admits its own successor under the same lock it decrements, so a stop checked only at the dispatch entry point leaves a chain that outruns the bounded wait and makes expiry the normal outcome. The event sinks hold a weak reference and drop counted events on an expired one, and the two CAS-owned `Context` accessors adopt `getZooKeeperLog`'s shape -- holding `mutex_shared_context` across the nested lock and the copy, since a bare ternary would leave the race it appears to remove. In `~Pool` all three phases are guarded and `drained` defaults to false: the clean-release marker is a successor's proof that no in-flight conditional PUT can still land, so a swallowed drain failure must never be able to forge it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six defects, each checked against the code before accepting. The farewell ordering was described backwards. `MountLeaseKeeper::terminate` does its `putOverwrite` FIRST and emits `MountRelease` last, so the marker is already durable when the emit crashes; the original review's "the crash also eats the farewell write" described an older tree. Overstating the cost would misdirect the next reader, so the claim is removed and the correction stated. The RAII pin as specified could not compile: `startThreadFromGlobalPool` takes `std::function<void()>`, so a captured pin must be copy-constructible. It becomes a copyable single-release guard -- an aliasing `shared_ptr` whose control block owns the `Pool` reference and releases the count once, on the last copy -- with `pin_owner` renamed to `try_pin_detached_work` so that a refusal after stop is part of its signature. The counter and its test disagreed: after `resetSharedContext` on a live `Context` the weak reference still locks and the accessor merely returns nothing, so no "expired" counter fires. The three outcomes are now tabulated and only a genuinely expired reference is counted -- an unconfigured system log is the ordinary steady state and counting it would make the counter mean nothing. Two sections contradicted each other about `~Pool`; explicit teardown is now described as the usual place the last reference is released, never the guaranteed one. `~Pool` under `pointer_mutex` was listed as out of scope while the new sequencing actually closes it, so it is claimed as a consequence instead. And a bare weak `Context` could not tell "never supplied" from "already released", so the member is an optional weak reference: `nullopt` disables the integration, an expired supplied reference fails startup rather than silently degrading. Also: real anchor links instead of literal tags, one verified commit instead of per-claim line numbers, an Observability section naming both counters, an idempotent teardown helper, and tests specified without sleeps and with the `std::terminate` check as a death test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five corrections, each checked against the code. `pin_owner` is not a `Pool` method: it is `CasRefLedger`'s injected `std::function<std::shared_ptr<void>()>` member. `Pool` gains `tryPinDetachedWork` and the ledger's callback is renamed to `try_pin_detached_work`; its existing return type already carries the refusal contract. Admission is also specified exception-safe -- build the token unarmed, then check, increment and arm under the mutex -- since a count leaked by an allocation failure makes every later drain run to its deadline and report a timeout that never resolves. `Context::shared` is a raw `ContextSharedPart *`, so a strong `ContextPtr` never owned the shared part and the neighbouring comment about ownership is not what the CAS reference violates. What it defeats is the surrounding intent to destroy the `Context` explicitly, which the spec and its test now say instead. The guarantee no longer implies ordinary `PoolPtr` holders are ordered against `shutdown` -- nothing drains them; they are short-lived by expectation and explicitly outside the guarantee. The fail-soft rationale no longer claims a frequency increase: `~Pool` runs once either way, and the point is that this design moves it into managed shutdown while a timeout keeps the old late path reachable. Observability said two warnings carry the same facts as two counters. They do not: the drain timeout has both, an expired context has only a counter -- the logging machinery is what has gone away -- and the clean-release warning belongs to a different condition. A timeout is also restated as the guarantee not being ESTABLISHED, since it is conditional on a successful drain. The destructor test becomes a subprocess exit test with `EXPECT_EXIT`: after the change the subprocess exits normally, which is not what `EXPECT_DEATH` expresses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rk spec Two architectural gaps, three unsound details, verified against the code. The ledger had no way to read the stop flag. Two of the four checkpoints -- `runtime_still_admitted` and `settleSnapshotPublish` -- must ask whether stopping has begun without taking a pin, and taking one to read a flag would create the very work the caller is avoiding. The ledger gains a second, read-only injected callback wired to a new `Pool::detachedWorkStopping`. A refused or failed pin now rolls back the publisher's admission. `admitSnapshotPublishUnderStateLock` increments `pending_snapshot_publishes` before the dispatch, and only the dispatched thread decrements it, so any path that launches no thread strands the count -- and remount quiescence, the cache budget and `dropNamespace` all block on it reaching zero. The stop must also cancel in-flight recovery. `ensureRefTableRecovered` runs before `runtime_still_admitted` exists and retries transient failures for `recovery_retry_budget_ms`, 120 seconds by default, against a drain deadline measured in seconds. The existing per-runtime `recovery_cancel_requested` is reused rather than duplicated, with one difference stated: self-remount clears the flag, shutdown must leave it latched. `startup` now resolves the weak context once and holds it, since two separate resolutions can straddle an expiry and leave a mount half-configured. The fail-soft guards must guard their own logging: `tryLogCurrentException` allocates, and memory pressure is exactly when a teardown phase throws. The drain deadline is a derived bound only for the budget-controlled snapshot writes; for the raw diagnostic GET it is insurance. Tests added for the two destructor paths the design introduces: implicit destruction without `shutdown`, and destruction after it, for idempotence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The detached-work design is rebuilt around one entry point, `Pool::tryDispatchDetached`, instead of exporting a pin and a stopping predicate. The rollback does not disappear -- a single-flight reservation taken under `state_mutex` necessarily precedes a fallible launch, so compensation is required -- but it becomes one RAII guard released into the launched task, so refusal-after-stop, allocation failure and launch failure share a single path. The task receives a stop token, which removes the need for a second read-only callback: both remaining checkpoints run inside the task. A failed dispatch now never fails its caller. `dispatchSnapshotPublisher` is reached from read-side maintenance and from settlement's re-dispatch, and the existing catch states the contract: a background publish must never fail an otherwise-successful read or mutation. rev.4's "let it propagate" contradicted that. Recovery cancellation becomes latch-only for shutdown. `cancelRecoveriesAndAwaitQuiescence` waits on `recovery_cv` with no deadline and then clears the flag, so calling it from a bounded drain would block before the bounded wait -- preventing the very timeout warning and counter that report the failure -- and would unlatch a terminal cancellation. The publish step is factored out; shutdown latches and lets the registry's single bounded wait do the waiting. Recovery reads are also direct backend calls, so the request budget does not bound them either; that is now stated. The reset-context test moves into a subprocess exit test for the same reason the destructor test already was: pre-change it dereferences null and takes the binary down. A test for terminal cancellation unparking a publisher inside recovery is added -- otherwise the new critical branch has no coverage. All three documents now use the mandated strict gate filter `CAS*`, and the item-9 suites are renamed to match. `Cas*` would have dragged in `CascadeWriteBuffer` and, worse, masked exactly the misnaming it was hiding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reservation accounting. `settleSnapshotPublish` decrements and re-admits under one `state_mutex` hold precisely so no observer sees a transient zero, so a copyable guard that also decremented would double-decrement a successful publish or erase a re-admitted reservation -- false quiescence, then a negative count. The guard becomes scope-local and non-copyable, armed only around the fallible dispatch and disarmed on success, so exactly one path owns each decrement. Being local, it also covers the allocation that happens while constructing the task argument, which the dispatcher itself cannot: that throw occurs in the caller's expression, before the method body is entered, so "never throws" was a promise the signature could not keep. Reference release ordering. A zero in-flight count did not imply the `Pool` reference was gone: a strong reference living in the callable's captures dies with the `std::function`, after the count may already have reached zero, so shutdown could observe zero and `~Pool` still run on the worker. The registry state is now separately owned and outlives the `Pool`, and a lease releases its `PoolPtr` BEFORE decrementing. Recovery cancellation. Reusing `recovery_cancel_requested` was wrong twice over: `cancelRecoveriesAndAwaitQuiescence` unconditionally clears it after an unbounded wait, and the remount worker is still live during the drain, so a latched terminal cancellation can be lost; and setting it would not unpark a sleeping publisher anyway, since `recovery_retry_sleep_fn` exits early only on a dropped fence, which a terminal shutdown does not cause -- 30s of backoff against a deadline of seconds. The stop token is threaded into recovery instead, including its retry sleep's slice loop, which is already built to be interruptible and merely polls the wrong predicate. Shutdown no longer touches remount's mechanism at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keep `GcMetaWriter::drain` as the throwing protocol barrier and add `drainOnExitNoThrow` for the scope guard. Guard the cleanup diagnostic so an unwinding round exception is preserved. Move the confirmed CAS unit-suite escapes under the strict `CAS*` gate and pin the cleanup `noexcept` signature at compile time.
The task lease must be a copyable handle that completes once: the task travels through `std::function<void()>` and is copied, so a move-only lease does not compile and a plainly copyable one releases per copy. A `shared_ptr` control block performs the release order on last destruction. Relatedly, neither task body may keep a `PoolPtr` of its own -- the diagnostic lambda's `self` capture dies with the `std::function`, outside the lease's control, which is precisely how a released count can coexist with a live reference. Settlement in the launched task becomes unconditional. It is reached today by a bare call after a handler that calls `tryLogCurrentException`, which allocates and can throw under memory pressure; the reservation is then stranded forever and `quiesceRefTablesForRemount` and `dropNamespace` wait on a count that never reaches zero. A stranded reservation is worse than the crash this spec is about, because nothing reports it. The diagnostic dispatch keeps a caller-side try/catch even though it needs no rollback guard: it is called from a fail-closed path, and an allocation failure while building the task must not replace the exception the caller was already raising. Recovery must observe the stop token in four places, not one. Two sit above every checkpoint previously listed: `recovery_cv.wait` is called with no predicate and no deadline while another recovery is in progress, and a fresh recovery reads the checkpoint before its first admission check. Also corrects `waitForSnapshotPublishSettle` to its real name, `waitForSnapshotPublishSettleForTest`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six tasks against rev.7 of the spec. The three properties that decide correctness each get a failing-first test rather than more prose: a zero in-flight count implies the pool reference is already released; exactly one path owns each reservation decrement; and a teardown unparks a publisher wherever it can park, including a concurrent-recovery wait with no predicate and a backoff sleep that today only watches the mount fence. The plan marks which steps have a real red and of what kind -- task 1 opens with a build failure rather than a test failure, and the two context and destructor cases are subprocess exit tests because pre-change they take the whole binary down. Suite naming is called out as a gate rather than a style rule: the CAS gate is exactly `CAS*`, and a suite that does not match never runs in it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remove the dead `CountingBackendShape` exclusion now that the suite is registered as `CASCountingBackendShape`, and close the two matching backlog records with strict gate evidence.
The previous version was an outline. This one carries the test code, the hook definitions and the implementation bodies, and separates every build, log analysis, test run and commit into its own step so no stage can run a stale binary or commit after a failure. Test logs are analyzed by a subagent exactly as build logs are. Four technical corrections came out of review. A predicate on `rt.recovery_cv` cannot wake anyone, because the drain notifies a different condition variable -- the concurrent-recovery wait becomes a bounded slice wait, mirroring the sliced backoff beside it, and a token-stopped attempt is terminal so the transient classifier cannot re-drive the walk. The first-request test no longer demands the impossible: a stop cannot interrupt an in-flight GET, so the publisher is parked at a pre-request hook and the assertion is that the request is never issued; an already-started request is a bounded timeout by design. Admission allocates before it counts, so an allocation failure cannot strand a count that nothing will decrement, and the release-ordering test asserts at an injected boundary hook rather than racing a post-hoc use_count. And the timeout counter is asserted only from the task that introduces it, with `src/Common/ProfileEvents.cpp` in the same commit as each counter's first use. Task boundaries are also redrawn: dispatch rewiring is its own task rather than a step, so the publisher's rollback guard and unconditional settlement are reviewable on their own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…; six duplicates) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
…ntry and the pinned beta S3 image Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
…beta rustfs image) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
Task order is now a stated safety property: the fail-soft `~Pool` lands BEFORE the teardown sequencing that makes `~Pool` routine, so no intermediate commit reaches a throwing teardown phase more predictably while it is still unguarded. The teardown helper guards each phase separately, since one `try` around all of them would let a throw from stopping the GC skip the part-access release and the drain itself. Three symbols in the previous version did not exist. Recovery gets its own token-aware sleep seam rather than the shared single-parameter `setCasRetrySleepForTest`, which also feeds the request controller and has a dozen call sites; the waiters accessor is `refRecoveryWaitersForTest`; and the publisher's error hook is an injected ledger callback, because the ledger's own `config` member is a `RefLedgerConfig` and a `PoolConfig` field is unreachable from there. The metadata-storage fixture reference is corrected to `gtest_cas_operation_gate.cpp`, which actually builds one. The two handshake tests were races: opening a gate from another thread could beat the drain, so a correct implementation could record `false` and a broken one pass. Both now run the drain on its own thread and poll an observable stop latch before releasing the worker. The publish-failure test faults the backend unconditionally instead of leaving it optional -- with a healthy backend the error handler is never entered and the test proves nothing. The token now threads through `runRecoveryWalkOnce` and `checkRecoveryStillAdmitted`, not only the first request, since the walk issues more I/O after that check. A deliberately expired drain asserts the timeout counter, which no previous test could observe. Task 3 also adds its seams in a separate, green build step before the behaviour tests, so its runtime red is observable rather than masked by a compile error. The command templates no longer append `; echo "$?"`: that makes the command's exit status the echo's, so a failed build reports success to anything gating on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
content addressable storage - draft PR
Documentation entry for user-facing changes
TBD.
Exclude tests:
Regression jobs to run: