refactor(transport)!: share fan-out execution across services

[thodson-usgs]

Stacked on DOI-USGS#350, and opened here on the fork deliberately: refactor/phase-2-transport-boundaries only exists on the fork, so this is the only base that yields a 1-commit diff. Targeting DOI-USGS:main today would carry DOI-USGS#350's three commits along with it — the duplicated-history shape that got DOI-USGS#345 closed.

When DOI-USGS#350 merges, retarget this at DOI-USGS:main (it will then be a clean single commit). Independent of DOI-USGS#351, which touches waterdata/api.py only.

The distinction

Chunking is how you divide the data structurally; fan-out is how you distribute the work operationally.

The two are orthogonal, and only the first is protocol knowledge: dividing a query needs the byte budget, the CQL2 grammar, and which parameters are list-valued. Distributing the pieces needs none of it.

The package had not drawn that line. ChunkPlan (division) and ChunkedCall (distribution) sat side by side in dataretrieval/ogc/ as siblings — and ADR 0006 grouped them together deliberately, which was right while a byte plan was the only thing anyone fanned out over.

Measuring OGC domain vocabulary in each class's code, docstrings and comments stripped:

ChunkPlan     7 distinct -> CQL, axes, axis, axis_chunks, bytes, filter_lang, url_limit
ChunkedCall   0 distinct -> []

What that cost

Water Use fans out for an entirely different reason — the NWDC accepts one location= per request, and its URLs run ~63 bytes against an 8000-byte budget. Unable to reach an OGC-internal executor, wateruse._fan_out re-implemented the semaphore, the gather, and the cancellation-beats-HTTP-error precedence rule, with a comment at wateruse.py:398 naming ChunkedCall._run as the original. One subtle rule, two copies, synchronized by prose.

The duplicate also lost three things:

	OGC	Water Use (before)
resume after 429	.call.resume()	none — one exhausted location discarded all completed ones
progress line	yes	no
concurrency knob	API_USGS_CONCURRENT	module global MAX_CONCURRENT_REQUESTS = 4

The resume gap is the one that bites: a multi-state county pull is hundreds of requests against a 1000/hr quota.

What this does

Moves execution down, leaves planning up. transport/fanout.py::FanOut drives any FanOutPlan — a Protocol of exactly the three members the executor already used (total, canonical_url, iter_sub_args()).

Structural rather than nominal because the two implementations share an interface and no implementation at all: ChunkPlan derives sub-requests from a byte budget over multi-value axes; a Water Use plan lists locations the caller already named separately. Neither has anything the other could inherit, so an ABC would be ceremony. ChunkPlan needed no edits — it already satisfied the protocol.

Water Use sheds ~75 lines and gains resume, progress, and the shared setting.

Breaking changes

1. A Water Use fan-out interrupted by 5xx/429 now raises ServiceInterrupted/QuotaExhausted, not ServiceUnavailable/RateLimited. Both remain DataRetrievalError so broad handlers are unaffected, but a narrow except ServiceUnavailable around a Water Use call must widen. This is convergence with the OGC getters — and it is precisely what makes the failure resumable.
2. wateruse.MAX_CONCURRENT_REQUESTS is removed in favor of API_USGS_CONCURRENT and wateruse.DEFAULT_CONCURRENT_REQUESTS.

Not breaking: ChunkInterrupted is a permanent alias of the renamed FanOutInterrupted — the same class object, not a deprecation — so except ChunkInterrupted keeps working. The rename is one name, because QuotaExhausted and ServiceInterrupted were already fan-out-neutral.

Concurrency: one setting, per-service defaults

API_USGS_CONCURRENT is general; a service declares a default for when it is unset (Water Use 4, package-wide 32). An explicitly set env var outranks a service default, never the reverse — a service able to override it would make API_USGS_CONCURRENT=1 a lie, which is the original defect.

Layout

From	To
ogc/chunking.py::ChunkedCall	transport/fanout.py::FanOut
ogc/retry.py::_classify_chunk_error	interruptions.py (beside the classes it produces)
ogc/interruptions.py	interruptions.py (top-level leaf)

Stays in ogc: ChunkPlan, multi_value_chunked, parallel_chunks, _OGC_URL_BYTE_LIMIT. Compatibility aliases (ChunkedCall, get_active_client, _chunked_client) remain importable from ogc.chunking; _chunked_client is the same ambient object transport publishes, not a copy.

interruptions.py is a top-level leaf for the reason ADR 0006 gives for combining/progress/credentials: adapters need it whether or not they went through transport, and an exception taxonomy is not HTTP execution policy.

Verification

• 685 tests pass (5 new), ruff clean, mypy --strict clean, pre-commit clean.
• New fitness functions: wateruse may contain no asyncio.gather/Semaphore/TaskGroup; both plans satisfy FanOutPlan (including that iter_sub_args() is stable across passes and agrees with total, since resume keys by position); the Water Use plan does not inherit ChunkPlan; no interruption taxonomy inside transport.
• New adapter tests: Water Use resume re-issues only unfinished locations, progress ticks, concurrency precedence.

One deviation from plan: I predicted the moves would change zero OGC tests. They changed three call sites — 9 patches of _chunking.asyncio.sleep (the backoff sleep is issued by transport.retry; patching it through the chunker only ever worked because asyncio is a shared module object) and one client-factory patch. No assertion changed; each now names the module that actually owns the behavior.

Known costs

• Resume re-issues a failed location's entire page walk, so pages fetched before the failure are fetched again. Already true for OGC — a partial walk never enters the completion map — so a cost, not a correctness problem.
• Water Use frames carry huc12_id, not id, so _combine_chunk_frames concatenates without deduplicating. Correct (locations partition by construction), but the dedup safety net does not apply there.

Open question

.completed_chunks / .total_chunks are left as-is. They are read rather than caught, and the message text already says "sub-requests", so renaming them would churn ~30 assertions for cosmetics. Happy to do it if you want the vocabulary uniform.

Supersedes one clause of ADR 0006; see the new ADR 0007.

🤖 Generated with Claude Code

[thodson-usgs]

Superseded by DOI-USGS#355, which is the same commit rebased onto main now that DOI-USGS#350 has merged. This one existed only because its base branch lived on the fork; with phase 2 in main the work can target upstream directly.