Statically-typed configuration object (donfig retained as the config reader)

[d-v-b]

🤖 AI text below 🤖

Zarr's configuration is currently an untyped nested dictionary. This PR introduces a dataclass-backed schema, typed attribute access, typed reads for known dotted-string keys, and helpful errors for unknown programmatic keys. Donfig remains responsible for environment/YAML parsing, file discovery, and precedence.

External configuration is processed in two stages:

• Collection recognizes schema-derived configuration names and explicit environment controls (ZARR_CONFIG, ZARR_ROOT_CONFIG, and ZARR_BENCHMARK_CLEAR_CACHE). Controls remain available to their consumers and never become config fields. Unknown environment names and YAML keys warn and are skipped. The codecs namespace remains open to custom codec names.
• Config creation takes the collected mapping, applies typed defaults, and validates keys and values without reading external state. For example, ZARR_ARRAY__ORDER=Q raises a field-specific error. Programmatic config.set() continues to validate keys while deferring value validation to use sites.

The state remains process-global, including scoped overrides. Exiting with config.set(...) restores only the keys changed by that context. Hyphen/underscore key aliases, nested keyword overrides such as array__order="F", custom codec selections, and manager deep-copying are supported. Collection now recognizes the benchmark cache-control flag directly, replacing the temporary benchmark warning filter.

Compatibility changes are documented in changes/4101.changed.md: config.defaults returns a dict; structured subtrees become immutable dataclasses with mapping-style reads; whole structured-subtree assignment is rejected; clear() and merge() are not retained; and pprint() no longer accepts formatting arguments.

Tests compare supported API and ingestion behavior against donfig using schema-derived Hypothesis strategies. Additional tests cover collection ownership, unknown names, malformed values, construction independent of the environment, and the open codec namespace. The targeted configuration/consumer run with ZARR_BENCHMARK_CLEAR_CACHE=1 passed 262 tests with 5 skipped. All repository hooks passed, including full mypy. CI is rerunning for the latest commit.

[codecov]

Codecov Report

❌ Patch coverage is 95.17426% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.24%. Comparing base (9f0ae4e) to head (7a26481).

Files with missing lines	Patch %	Lines
src/zarr/core/config.py	95.17%	18 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4101      +/-   ##
==========================================
+ Coverage   94.22%   94.24%   +0.02%     
==========================================
  Files          92       92              
  Lines       12942    13303     +361     
==========================================
+ Hits        12195    12538     +343     
- Misses        747      765      +18     

Files with missing lines	Coverage Δ
src/zarr/core/config.py	95.22% <95.17%> (-4.78%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

I'm adding some useful error messages:

  5 Traceback (most recent call last):
  4   File "<stdin>", line 1, in <module>
  3   File "/home/d-v-b/wt/statically-typed-config/src/zarr/core/config.py", line 483, in get
  2     raise _unknown_key_error(key, current) from None
  1 KeyError: "'arraya' is not a valid configuration key. Did you mean 'array'?"

[d-v-b]

flagging this as nice, but delicate (polite term for brittle). These overloads have to be kept synced up with the actual structure of the configuration, which is defined elsewhere. This PR adds tests that confirm that these overloaded signatures match the field name + type of the config object. I'm not sure how we can do do any better than that, if we want a string-based API like this to be statically typed.

an alternative that we can add later would be a non-string-based API, where we define an object that acts like a cursor / lens over our config.

[d-v-b]

donfig returned defaults as a list, we just return a dict

[maxrjones]

Thanks for your work on this @d-v-b!

I like that this brings validation of configuration settings along with suggestions for fixing wrong configuration inputs. I'm neutral about donfig vs. just pyyaml as a dependency. I don't think replacing donfig is strictly necessary for getting strict type checking, but also donfig doesn't bring a whole lot of extra value as a dependency such that it's motivating to keep it.

There are some regressions relative to donfig that I think should be addressed or at a minimum included in a changelog fragment under the changed category.

The hybrid state model leads to cross-thread permanent-set shadowing

import threading
from zarr.core.config import ZarrConfigManager
cfg = ZarrConfigManager(); cfg.set({"async.concurrency": 1})
t=threading.Thread(target=lambda: cfg.set({"async.concurrency": 999})); t.start(); t.join()
print("branch main sees:", cfg.get("async.concurrency"))
# -> branch main sees: 1
import threading
from donfig import Config
c = Config("z", defaults=[{"async": {"concurrency": 10}}]); c.set({"async.concurrency": 1})
t=threading.Thread(target=lambda: c.set({"async.concurrency": 999})); t.start(); t.join()
print("donfig main sees:", c.get("async.concurrency"))
# -> donfig main sees: 999

Subtree .get() returns a dataclass

from zarr.core.config import config
print("type:", type(config.get("array")).__name__)
config.get("array")["order"] # used to work

Previous config file locations are no longer picked up

See inlined comment + https://donfig.readthedocs.io/en/latest/configuration.html#yaml-files

[d-v-b]

I like that this brings validation of configuration settings along with suggestions for fixing wrong configuration inputs. I'm neutral about donfig vs. just pyyaml as a dependency. I don't think replacing donfig is strictly necessary for getting strict type checking, but also donfig doesn't bring a whole lot of extra value as a dependency such that it's motivating to keep it.

the core of this PR is defining our config as a dataclass that we manage. how could we do that, and still use donfig? iirc, donfig just exposes an untyped dict-like API. So there would be no way to relate the contents of the donfig config with our normative dataclass. Maybe I'm missing some donfig feature or something?

[maxrjones]

I like that this brings validation of configuration settings along with suggestions for fixing wrong configuration inputs. I'm neutral about donfig vs. just pyyaml as a dependency. I don't think replacing donfig is strictly necessary for getting strict type checking, but also donfig doesn't bring a whole lot of extra value as a dependency such that it's motivating to keep it.

the core of this PR is defining our config as a dataclass that we manage. how could we do that, and still use donfig? iirc, donfig just exposes an untyped dict-like API. So there would be no way to relate the contents of the donfig config with our normative dataclass. Maybe I'm missing some donfig feature or something?

I think if you consider the goal being managing our config internally as dataclasses, donfig must go. If the goal is provide typing and suggestions for configuration settings, that can be built on top of donfig since types can be made more strict.

[d-v-b]

If the goal is provide typing and suggestions for configuration settings, that can be built on top of donfig since types can be made more strict.

Can you outline how this would work?

[d-v-b]

maybe this aligns with your thinking, but I'm going to explore keeping donfig as the config reader, and use the dataclass as the source of truth for the structure of the config

[d-v-b]

i'd like to move forward on this effort, since improving our config API is a necessary condition for adding a lot of the new features we have in planning.

@maxrjones what's your feeling today on a donfig dep? At present I lean toward whatever gets this PR out the door, but long term I'm not convinced that donfig offers anything that we couldn't vendor ourselves. LMK what you think.

[d-v-b]

we can also move the "should we depend on donfig" discussion elsewhere. i'll make an issue.