Conversation
`zarr.open` looks for an array first and opens a group when it doesn't find one. Both steps read `zarr.json` and `.zattrs`, so a format-detecting open of a group made seven requests where five would do, a cost paid on every call against a remote store. Split the array lookup into `_probe_array_metadata`, which reports "no array here" as a value instead of an exception and hands back the documents it read. `zarr.open` passes those to `AsyncGroup.open`, which then reads only the keys it is still missing. `get_array_metadata` keeps its signature and its exceptions, and becomes a thin wrapper over the probe. Assisted-by: ClaudeCode:claude-opus-5
github-actions
Bot
added
the
needs release notes
Assisted-by: ClaudeCode:claude-opus-5
github-actions
Bot
removed
the
needs release notes
Documentation build overview
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4366 +/- ##
==========================================
- Coverage 94.22% 94.20% -0.03%
==========================================
Files 92 92
Lines 12942 12908 -34
==========================================
- Hits 12195 12160 -35
- Misses 747 748 +1
🚀 New features to boost your workflow:
|
`zarr.json` wins whenever it is present, so whether the probe's metadata came from it is a fact about `docs`, not a second field that has to agree with `docs`. Also name the fetcher for what it reads: the array document set, never `.zgroup`. Assisted-by: ClaudeCode:claude-fable-5-1
A dataclass field set to None could mean the key was read and held nothing, or was never read at all, so the group open could only reuse documents under a guard at the call site. With NotRequired keys, presence is the signal: the group open takes whatever was actually read, and zarr.open passes the probe's documents along unconditionally. Assisted-by: ClaudeCode:claude-fable-5-1
The probe learns one of three things: nothing is here, an array is, or a Zarr format 3 group is. Say that with `node_type: Literal["array", "group"] | None` instead of deriving `is_array` and `from_zarr_json` from the documents after the fact. That makes the classification explicit. A `.zarray` is an array. A `zarr.json` is whatever its `node_type` says, with a missing `node_type` read as a group, which is the leniency `GroupMetadata.from_dict` already grants and so what `zarr.open` already did. Any other value is now rejected by the probe with `NodeTypeValidationError`, the same error the group open raised for it after the fallback, only earlier and with a clearer message. Assisted-by: ClaudeCode:claude-fable-5-1
…in functions Drop the `_ArrayProbe` class. `_fetch_array_metadata_docs` returns the `_MetadataDocs` TypedDict and `_array_metadata_from_docs` interprets it, raising exactly what `get_array_metadata` raised before; that function is now their composition. `zarr.open` reads the docs once ahead of its existing try/except, which is otherwise unchanged, and passes them to the group open on the fallback. Pin the two odd-document behaviors `zarr.open` already had: a zarr.json without a node_type opens as a group, and one with an unknown node_type ends in GroupNotFoundError. Assisted-by: ClaudeCode:claude-fable-5-1
With zarr.json already in hand, the format-detecting branch of AsyncGroup.open knows the format before it reads anything else. A format 3 group needs nothing more, so it now costs no reads at all on top of the array lookup; a format 2 group reads .zgroup, plus the consolidated document only when use_consolidated is not False, which is the rule the explicit format 2 branch already followed. The same rule now applies to the un-pre-fetched path, which used to read .zmetadata and then discard it. zarr.open on a group: format 3 goes from 5 reads to 3, format 2 with use_consolidated=False from 5 to 4. The one thing given up is the warning for a store holding both zarr.json and .zgroup on the zarr.open path, since .zgroup is no longer read when zarr.json is present; open_group with zarr_format=None still emits it. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
changed the title
zarr.open used to look for an array and, failing that, open a group, and the two steps read overlapping keys: seven requests for a group, five of them for a format 3 group whose single zarr.json already held everything. Now zarr.open builds the node itself from what it reads. _open_v3 reads zarr.json once; its node_type says array or group and the document holds everything needed to open either. Only if there is no zarr.json does _open_v2 read .zarray, .zgroup, .zattrs and, when it might be used, the consolidated document, in one concurrent round. The existing open_group fallback still handles create-or-raise, and the mode="w" quirk is kept: an existing array is returned, an existing group is overwritten. The pieces zarr.open needs come out of AsyncGroup.open without changing it: _resolve_use_consolidated settles use_consolidated against the store, the format 2 missing/discard check moves into _from_bytes_v2 so it mirrors _from_bytes_v3, and _from_dict_v3 builds a group from a parsed document. No signature that users see changes. Reads via zarr.open: a format 3 array or group costs one; a format 2 group five (four with use_consolidated=False), as before; a format 2 array five where it cost three, plus one round trip, the price of being the fallback. When both zarr.json and .zarray exist at a path, zarr.open takes the format 3 node without reading the other and no longer warns; open_array still does. zarr.open(mode="w", use_consolidated=...) now runs the consolidated metadata checks before overwriting instead of ignoring the request. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
changed the title
…data reader Three read implementations existed: get_array_metadata for arrays, AsyncGroup.open for groups, and the _read_metadata_v2/v3 helpers behind AsyncGroup.getitem for either kind with an explicit format. Each fused reading with interpreting, the first two each detected the format on their own, and zarr.open, with no primitive to build on, was trial-and-error over the specific openers with exceptions carrying "found something else." Now read_node_metadata reads a node's documents once and returns parsed metadata whichever kind and format the node is, trying Zarr format 3 first when the format is not given; _open_node adds the use_consolidated policy and builds the node; and open, open_array, open_group, AsyncArray.open, AsyncGroup.open and get_node are mode policy and a kind filter over that. AsyncGroup._from_bytes_v2/v3 and _from_dict_v3 are gone, and _build_node takes the array config. open has a written contract, replacing the "mode check seems wrong" TODO: with shape it behaves as open_array; otherwise the reading modes open whatever node is there, 'a' creating a group when there is none, and the creating modes create a group, 'w' replacing whatever is there. One node_type policy: a format 3 document's node_type must be array or group, else NodeTypeValidationError; finding the wrong kind is ContainsArrayError or ContainsGroupError, never "not found." Behavior changes, each pinned by a test: open(mode="w") without shape replaces an existing array with a group instead of returning it; open(shape=...) on a group raises ContainsGroupError instead of opening it; open(mode="r") on nothing raises NodeNotFoundError; a missing or unknown node_type raises everywhere (a missing one used to open as a group); wrong kind raises Contains*Error in every format and mode; open_array(mode="w") on a group replaces it; open_array applies config to an existing array; a path holding both formats opens as format 3 without a warning; requesting missing format 2 consolidated metadata raises with the format 3 message. Reads via zarr.open: a format 3 node costs one; a format 2 group five (four with use_consolidated=False); a format 2 array five where it cost three, the price of being the fallback. Assisted-by: ClaudeCode:claude-fable-5-1
read_v3_metadata and read_v2_metadata are the reusable pieces: each reads one format's documents and returns metadata or None. read_node_metadata is their composition, trying format 3 first when the format is not given, rather than inlining both and hanging the per-format wrappers off itself. Assisted-by: ClaudeCode:claude-fable-5-1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
What
zarr.openused to look for an array and, failing that, open a group, and the two steps read overlapping keys: seven requests for a group, five of them for a format 3 group whose singlezarr.jsonalready held everything. Fixing that exposed the shape of the code underneath, so this PR restructures how nodes are opened rather than patching the one path.How
Three read implementations existed:
get_array_metadata(arrays),AsyncGroup.open(groups), and the_read_metadata_v2/v3helpers behindAsyncGroup.getitem(either kind, explicit format only). Each fused reading with interpreting, and the first two each detected the format on their own.openhad no primitive to build on, so it was trial-and-error over the specific openers, with exceptions carrying "found something else."Now there is one reader and everything is a thin layer on it:
read_v3_metadata(store, path)andread_v2_metadata(store, path, *, consolidated_key=None)each read one format's documents once and return parsed metadata, whichever kind the node is, orNone; the v2 reader attaches a consolidated document to a group when asked to read one.read_node_metadata(store, path, zarr_format, *, consolidated_key=None)is their composition: withzarr_format=Noneit tries v3 first and reads the format 2 documents only if there is nozarr.json._build_node(now takingconfig), and for groups_apply_use_consolidatedenforces the caller'suse_consolidatedon what was read.AsyncGroup._from_bytes_v2/v3and_from_dict_v3are gone;AsyncGroup.from_dictis a plain constructor._open_node(store_path, *, zarr_format, use_consolidated, config)= read + consolidated policy + build, orNone. No mode policy, no exceptions for "nothing here."open,open_array,open_group,AsyncArray.open,AsyncGroup.open,get_nodeare mode policy and a kind filter over_open_node.get_array_metadatais kept for callers that want the document.openhas a written contract (in its docstring, replacing theTODO: the mode check below seems wrong!): withshapeit behaves asopen_array; otherwise the reading modes (r,r+,a) open whatever node is there,r/r+raisingNodeNotFoundErrorandacreating a group when there is none, and the creating modes (w,w-) create a group,wreplacing whatever is there.node_typepolicy: a format 3 document'snode_typemust bearrayorgroup, elseNodeTypeValidationError. Finding the wrong kind isContainsArrayError/ContainsGroupError, never "not found."Reads via
zarr.openmainuse_consolidated=False)The v2 array row is the price of format 2 being the fallback; format 3 is the default, so one read for any format 3 node is the number that matters.
Behavior changes
All deliberate, each pinned by a test:
zarr.open(mode="w")withoutshapecreates a group over whatever is there, including an existing array, which it used to return untouched.zarr.open(shape=...)on a group raisesContainsGroupError; it used to open the group.zarr.open(mode="r" | "r+")on nothing raisesNodeNotFoundErrorrather thanGroupNotFoundError(both areFileNotFoundError;GroupNotFoundErroris the narrower class, so code catching only that no longer catches this).node_typeraisesNodeTypeValidationErroreverywhere. A missingnode_typeused to open as a group throughopen/open_group; an unknown one raisedGroupNotFoundError.ContainsArrayError(format 2 used to sayFileNotFoundError); opening a group as an array raisesContainsGroupError(format 3 used to sayNodeTypeValidationError, format 2ArrayNotFoundError).open_array(mode="w")on a group replaces it in both formats; format 3 used to raise.open_array(config=...)on an existing array now appliesconfig; it was silently ignored.ValueError(".zmetadata"); a consolidated document with nometadataobject raisesMetadataValidationErrorinstead ofKeyError.Tests
test_open_mode_contractis the contract as a table: 30 cells of what-is-there × mode × with/withoutshape, checking what comes back and whether it was opened (attributes kept) or created.test_open_reads_only_what_the_node_needs(84 cases) asserts the exact keys read, each once, and that the node is whatopen_array/open_groupreturn. Small tests pin each error type above.This supersedes d-v-b#192.
🤖 Generated with Claude Code