DAOS-19451 cont: enable checksum by default on non-v0 pools by ryon-jensen · Pull Request #18885 · daos-stack/daos

ryon-jensen · 2026-08-19T02:45:14Z

DAOS-17946 changed the default container properties to crc32 checksum and server-side checksum verification, but only updated the v0 default property table in src/container/srv_layout.c. cont_prop_entries_default, which is the table used for every pool with a global version >= 1, was left untouched, so containers created on any modern pool still defaulted to checksum off.

Apply the same defaults to cont_prop_entries_default: crc32 checksum, server-side verification on and RF3. Note that the container redundancy factor is always inherited from the pool when it is not explicitly specified, so the RF3 entry is only there to keep the two tables consistent.

The regression went unnoticed because the test harnesses hid it: daos_test forced checksum off on every container it created and the ftest container utility hardcoded "cksum:off,srv_cksum:off", both added by DAOS-17946. Drop those overrides and update the co_properties assertions so the defaults are actually exercised.

Latent checksum bugs uncovered by the new default

Turning checksums on reactivated a large amount of code that no test had been
exercising, and CI surfaced a series of pre-existing defects that are unrelated
to the default itself but block it:

Area	Defect
`cont`	`dc_cont_props_init()` became fallible, but `cont_open_complete()` had already linked the container into the pool list and never unlinked it on failure, so the error path left a dangling entry.
`object`	Both `iod_csums` decode error paths leaked the partially decoded array.
`object`	`LOCAL_STACK_NUM` is 2, so a CPD update writing more than two iods heap-allocates the `pcsums` array with NULL elements; `obj_get_iods_offs()` dereferenced one unconditionally and crashed the engine.
`object`	Echo RPCs short-circuit before `obj_fetch_create_maps()`, so the client map-count assertion fired; plus a zero-length iov guard.
`common`	A size-query fetch sets every `iov_len` to zero, tripping `idx->iov_offset < len` with `0 < 0` in `daos_sgl_get_bytes()`.
`object`	A fetch that transferred no data was still verified.
`object`	The EC single-value checksum layout was completed in shared, per-request state from a per-shard reply, so concurrent shard callbacks raced (see below).
`object`	`DAOS_CSUM_CORRUPT_DISK` fakes silent on-disk corruption, but was injected before `obj_verify_bio_csum()`. With server-side verify now on, the write was rejected and nothing reached disk, breaking the scrubber tests. Moved the injection after verification.

The subtlest one is the EC single-value layout used to verify a fetch.

On a fetch vos_fetch_begin() overwrites iod_size with the stored record size
before obj_singv_ec_rw_filter() runs, so the server lays the reply out and
checksums it according to the size it replies with, and each shard replies its
own size. The client request was built from the size the caller asked with,
which for DAOS_REC_ANY is only a guess taken from the sgl buffer size, and may
be an over-estimate otherwise.

Two problems followed:

Shared state between shard callbacks. dc_rw_cb_csum_verify() runs per
shard but receives orw_iod_array.oia_iods, which for an EC request aliases
the shared reasb_req->orr_iods, and it completed the equally shared
orr_singv_los[] in place. dc_shard_update_size() writes that same
shared iod_size from each reply, so one shard callback could overwrite the
size and layout another shard was about to verify against.
Stale layout. When the requested and replied sizes fall on opposite sides
of the even-distribution threshold, or merely imply different cell sizes, the
reply does not describe the record the request was segmented for.

Either way verification reported a false corruption, which fed
obj_ec_fail_info_insert() and surfaced as -DER_DATA_LOSS once the parity
limit was exceeded. EC17 reproduces both, logging the straddling pair:

dc_shard_singv_size_conflict() old_size 4103, tgt idx 1 off 2056, new_size 4095, conflict.

The fix derives the layout from the size this shard replied — the value
already used as the record length the checksum is calculated over — and returns
it through a caller-owned struct rather than completing the shared one. When the
replied size implies a different layout than the request was built for, it
returns -DER_FETCH_AGAIN: obj_reasb_io_fini() restores the user iods for
that error and dc_shard_update_size() has already stored the replied size in
them, so the retry is assembled for the layout the server actually used and is
verified normally. Verification is deferred, never skipped or downgraded, so a
real checksum error is still reported.

Verified on a six-engine single-host configuration, which is enough ranks for
the EC suite to run: four consecutive full daos_test -I runs pass 30/30 (the
failure reproduced roughly every second run beforehand), daos_test -X passes
29/29, and daos_test -T passes the distributed TX suite.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

daosbuild3 · 2026-08-19T03:18:32Z

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/1/testReport/

github-actions · 2026-08-19T07:09:41Z

Ticket title is 'container checksum is off by default'
Status is 'In Progress'
Labels: 'test_2.8.0rc'
Errors are Unknown component
https://daosio.atlassian.net/browse/DAOS-19451

daosbuild3 · 2026-08-19T23:20:59Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/2/execution/node/1522/log

daosbuild3 · 2026-08-20T07:27:34Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/3/execution/node/1525/log

daosbuild3 · 2026-08-20T20:09:33Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/4/execution/node/1524/log

daosbuild3 · 2026-08-21T08:39:00Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/5/testReport/

daosbuild3 · 2026-08-21T19:08:51Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/6/testReport/

daosbuild3 · 2026-08-21T22:40:04Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/6/execution/node/1632/log

daosbuild3 · 2026-08-22T00:06:33Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/6/execution/node/1622/log

daosbuild3 · 2026-08-23T01:15:39Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/12/execution/node/1623/log

daosbuild3 · 2026-08-23T04:31:48Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/12/testReport/

daosbuild3 · 2026-08-23T19:22:58Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/13/testReport/

daosbuild3 · 2026-08-24T04:58:54Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/14/testReport/

daosbuild3 · 2026-09-01T09:58:15Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/21/execution/node/1672/log

daosbuild3 · 2026-09-01T10:13:45Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/21/execution/node/1725/log

daosbuild3 · 2026-09-01T18:56:05Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/22/testReport/

daosbuild3 · 2026-09-04T05:20:59Z

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18885/24/display/redirect

daosbuild3 · 2026-09-04T12:46:25Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/25/testReport/

daosbuild3 · 2026-09-04T23:46:49Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/26/testReport/

daosbuild3 · 2026-09-08T21:05:41Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/27/testReport/

daosbuild3 · 2026-09-09T19:52:23Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/28/execution/node/1759/log

daosbuild3 · 2026-09-09T19:53:08Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/28/execution/node/1637/log

phender · 2026-09-10T20:32:58Z

@@ -367,7 +367,7 @@ def __init__(self, pool, daos_command, label_generator=None, namespace=CONT_NAME
        self.dir_oclass = BasicParameter(None)
        self.file_oclass = BasicParameter(None)
        self.chunk_size = BasicParameter(None)
-        self.properties = BasicParameter(None, "cksum:off,srv_cksum:off")
+        self.properties = BasicParameter(None)


This particular change is being addressed as part of https://daosio.atlassian.net/browse/DAOS-19452, in particular in https://daosio.atlassian.net/browse/DAOS-19488 with #18851.

daosbuild3 · 2026-09-11T00:34:18Z

Test stage Functional Cluster Box Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18885/31/testReport/

daosbuild3 · 2026-09-11T03:27:18Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18885/31/execution/node/1719/log

DAOS-17946 changed the default container properties to crc32 checksum and server-side checksum verification, but only updated the v0 default property table in src/container/srv_layout.c. cont_prop_entries_default, the table used for every pool with a global version >= 1, was left untouched, so containers created on any modern pool still defaulted to checksum off. Apply the same checksum defaults to cont_prop_entries_default: crc32 checksum and server-side verification on. The v0 table also defaults the redundancy factor to RF3, but that is deliberately not replicated here. A container inherits the redundancy factor from the pool whenever it is not specified, so the entry would change nothing in practice while leaving a surprising default sitting in the table. The regression went unnoticed because the test harnesses hid it: daos_test forced checksum off on every container it created and the ftest container utility hardcoded "cksum:off,srv_cksum:off", both added by DAOS-17946. Drop those overrides and update the co_properties assertions so the defaults are actually exercised. Enabling the default reactivated a large amount of checksum code that no test had been exercising, surfacing a series of pre-existing defects that block it: - cont: unlink the container from the pool list when dc_cont_props_init() fails, so the error path does not leave a dangling entry. - object: free partially decoded checksums on both iod_csums decode error paths. - object: fix an engine crash in CPD update. LOCAL_STACK_NUM is 2, so an update writing more than two iods heap-allocates pcsums with NULL elements that obj_get_iods_offs() dereferenced unconditionally. - object: fix client checksum verify for echo RPCs, which short-circuit before obj_fetch_create_maps() and tripped the map-count assertion, plus a zero-length iov guard. - common: skip zero-length iovs in daos_sgl_get_bytes(). A size-query fetch sets every iov_len to zero, tripping idx->iov_offset < len as 0 < 0. - object: do not verify fetches that transferred no data. - object: inject DAOS_CSUM_CORRUPT_DISK after obj_verify_bio_csum(). It fakes silent on-disk corruption, but with server-side verify now on the write was rejected before reaching disk, breaking the scrubber tests. - cont, object: report the return code with DF_RC when container property initialisation or key checksum verification fails. NLT suppresses fault-injection errors that carry DER_NOMEM, so these paths were reported as new warnings once enabling checksums made them reachable. The subtlest defect is the EC single-value layout used to verify a fetch. On a fetch vos_fetch_begin() overwrites iod_size with the stored record size before obj_singv_ec_rw_filter() runs, so the server lays the reply out and checksums it according to the size it replies with, and each shard replies its own size. The client request was built from the size the caller asked with, which for DAOS_REC_ANY is only a guess taken from the sgl buffer size and may be an over-estimate otherwise. Two problems followed. dc_rw_cb_csum_verify() is called per shard but receives orw_iod_array.oia_iods, which for an EC request aliases the shared reasb_req->orr_iods, and it completed the equally shared orr_singv_los[] in place, so one shard callback could overwrite the layout another shard was about to verify against. Independently, when the requested and replied sizes fall on opposite sides of the even distribution threshold, or merely imply different cell sizes, the reply does not describe the record the request was segmented for, and verifying it reported a false corruption. That fed obj_ec_fail_info_insert() and surfaced as -DER_DATA_LOSS once the parity limit was exceeded. Derive the layout from the size this shard replied, which is already used for the record length the checksum is calculated over, and return it through a caller owned struct instead of completing the shared one, so concurrent shard callbacks cannot interfere with each other. When the replied size implies a different layout than the request was built for, return -DER_FETCH_AGAIN. obj_reasb_io_fini() restores the user iods for that error and dc_shard_update_size() has already stored the replied size in them, so the retry is assembled for the layout the server actually used and is verified normally. Verification is deferred, never skipped or downgraded, so a real checksum error is still reported. That layout fix alone was not sufficient. The singv size conflict that forces the retry is derived from reasb_req->orr_fetch_stat, which holds the sizes replied by all shards, but it was only evaluated on whichever branch handled the reply in hand, and the parity/small-index branch additionally required every other shard to have replied success. A -DER_REC2BIG from another shard is the long singv overwritten by short singv case, which is deliberately ignored when the short value lives on a single target, so the conflict was never evaluated and the callback went on to verify against a stale layout. Evaluate the conflict from the shared fetch statistics on every callback, and accept -DER_REC2BIG as a state in which it still applies. EC17 reproduced once in four runs before this change and passed six consecutive runs after it, with no checksum mismatch logged in any of them. Features: checksum Test-tag: daos_test pr test_daos_container checksum Signed-off-by: Ryon Jensen <ryon.jensen@hpe.com>

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from d2e8bf5 to e842977 Compare August 21, 2026 01:45

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from 7c9f696 to 0b011e1 Compare August 23, 2026 12:21

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from 0b011e1 to 3d6584c Compare August 23, 2026 22:11

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch 5 times, most recently from fa0550e to 495761b Compare August 31, 2026 19:07

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from 495761b to 6769097 Compare September 1, 2026 10:11

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from 6769097 to bba33d3 Compare September 1, 2026 19:19

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from bba33d3 to db6608d Compare September 4, 2026 15:33

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from db6608d to 16cc4a5 Compare September 8, 2026 13:49

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch 2 times, most recently from 6c72ccb to 8fc84b3 Compare September 9, 2026 19:50

phender reviewed Sep 10, 2026

View reviewed changes

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from 8fc84b3 to 7266c80 Compare September 11, 2026 07:28

ryon-jensen force-pushed the ryon-jensen/DAOS-19451 branch from 7266c80 to 1a600f8 Compare September 11, 2026 07:36

ryon-jensen marked this pull request as ready for review September 14, 2026 17:05

ryon-jensen requested review from a team as code owners September 14, 2026 17:05

ryon-jensen requested review from NiuYawei, daltonbohning, gnailzenh, johannlombardi, liuxuezhao and phender September 14, 2026 17:05

Uh oh!

Conversation

ryon-jensen commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Latent checksum bugs uncovered by the new default

Steps for the author:

After all prior steps are complete:

Uh oh!

daosbuild3 commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Aug 19, 2026

Uh oh!

daosbuild3 commented Aug 20, 2026

Uh oh!

daosbuild3 commented Aug 20, 2026

Uh oh!

daosbuild3 commented Aug 21, 2026

Uh oh!

daosbuild3 commented Aug 21, 2026

Uh oh!

daosbuild3 commented Aug 21, 2026

Uh oh!

daosbuild3 commented Aug 22, 2026

Uh oh!

daosbuild3 commented Aug 23, 2026

Uh oh!

daosbuild3 commented Aug 23, 2026

Uh oh!

daosbuild3 commented Aug 23, 2026

Uh oh!

daosbuild3 commented Aug 24, 2026

Uh oh!

daosbuild3 commented Sep 1, 2026

Uh oh!

daosbuild3 commented Sep 1, 2026

Uh oh!

daosbuild3 commented Sep 1, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 8, 2026

Uh oh!

daosbuild3 commented Sep 9, 2026

Uh oh!

daosbuild3 commented Sep 9, 2026

Uh oh!

phender Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Sep 11, 2026

Uh oh!

daosbuild3 commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

ryon-jensen commented Aug 19, 2026 •

edited

Loading

github-actions Bot commented Aug 19, 2026 •

edited

Loading

phender Sep 10, 2026 •

edited

Loading