feat(transport)!: page large results in parallel by offset; remove parallel_chunks

[thodson-usgs]

Summary

A multi-page Water Data result used to cost one round trip per page, in strict
sequence. It no longer does. Where the API honors offset, every page's URL is
computable up front, so the pages are fetched concurrently — the request count
is unchanged, only their timing. Since the USGS quota is volume-based
(x-ratelimit-limit, default 1000/hr), the speedup is free of quota.

The same change removes parallel_chunks(n), which bought parallelism the
other way — splitting a request that already fit into more sub-requests. That
approach spent extra quota, and did nothing at all for a single-site query,
which has no multi-value axis to split. Overlapping the pages a request was
going to fetch anyway subsumes it.

Measured, fully warm, against the live API:

Query	Before	After	Speedup
8 sites × 2 years (16,000 rows, 8 pages)	4.6 s	2.2 s	2.1×
1 site, full daily history (~19,000 rows, 10 pages)	6.3 s	1.5–2.0 s	3.1–4.2×

The second row is the case that had no answer before: ChunkPlan provably
returns 1 sub-request for a single site regardless of parallel_chunks(n)
(verified at n=1, 8, and 32), so a single-site deep history was sequential and
there was no knob that changed it.

Why cursors are the slow part

OGC API - Features Part 1 defines paging as a next link relation. The
link's target is opaque — Water Data returns ?cursor=<token> — so page N+1's
URL literally does not exist until page N has come back and been parsed:

GET items?limit=2000               ──► page 1 ──► parse ──► next=?cursor=abc
GET items?cursor=abc               ──► page 2 ──► parse ──► next=?cursor=def
GET items?cursor=def               ──► page 3 ──► ...

Ten pages, ten serialized round trips. Latency dominates: each page is a small
payload behind a ~0.5 s round trip.

offset breaks the chain. It is not in the OGC spec (more on that below), but
Water Data supports it, and with it the whole URL set is a closed-form
computation — offset = i × limit:

GET items?limit=2000&offset=0     ─┐
GET items?limit=2000&offset=2000  ─┤ all in flight at once
GET items?limit=2000&offset=4000  ─┤
GET items?limit=2000&offset=6000  ─┘

Same four requests. One round trip of wall clock instead of four.

How it knows when to stop

Here is the wrinkle: the walk does not know how many pages there are. OGC
API - Features Part 1 makes numberMatched optional — "each page may
include information about the number of selected and returned features" — and
Water Data omits it. A page carries numberReturned (its own row count) but no
total. So the page count cannot be computed in advance; it has to be probed.

That is why the walk fetches in waves of width rather than one flat
fan-out. Each wave issues width requests at offsets continuing where the last
wave stopped, then inspects the results for a stop signal. A too-clever flat
fan-out of 32 requests for a 3-page result would burn 29 requests to learn
nothing; a wave of 8 burns at most 7, and only on the final wave.

Worked example: 25 rows, limit=10, width=4

wave 1 ── all four issued concurrently ─────────────────────────────
  offset=0    ──►  10 rows   (== limit → full page, more may follow)
  offset=10   ──►  10 rows   (== limit → full page, more may follow)
  offset=20   ──►   5 rows   (<  limit → SHORT PAGE, this is the end)
  offset=30   ──►   0 rows   (past the end — speculative, discarded)
                             ▲
                             └─ stop_index = 2

_stop_index walks the wave in offset order and returns the index of the
page that ends the walk — index 2 here. Pages after it are dropped, so the
overshoot at offset=30 is harmless: it was already paid for in parallel with
the pages that mattered, and its rows are never concatenated.

Result: 25 rows from 4 requests — 3 useful pages plus the single probe that
proved page 3 was the last one. The sequential walk needed those same 3 requests
plus a 4th to discover the end (next absent), so the request count is
identical; only the wall clock changed. tests/waterdata_offset_paging_test.py::test_page_count_is_not_inflated_by_parallelism
pins exactly that count.

Returning an index rather than a bool is what makes the earliest terminal page
win. If offset=20 is short but a speculative offset=40 also looks terminal,
index 2 still ends the walk — a later page cannot resurrect the pages between.

The four stop conditions, in precedence order

_stop_index is the single source of truth; the module docstring documents the
same order.

1. A short page — fewer than limit rows. The last page by construction:
   the server had no more rows to give. This is the normal exit. The page is
   kept, rows included.

2. An empty page — zero rows. The previous page ended exactly on a limit
   boundary and this offset is past the end. Everything before it is kept; the
   empty page is not. (With 20 rows at limit=10, no page is ever short — the
   empty page at offset=20 is the only signal available.)

3. The row cap — max_rows is reached, so further pages would be discarded
   anyway. A wave can overshoot the cap, so the cap is re-applied to the
   combined frame (result.head(row_cap)) rather than to a wave boundary;
   otherwise max_rows=25 would return whatever a wave happened to land on.

4. The offset ceiling — Water Data rejects offset > 40000 with HTTP 400
   InvalidQuery. This is not an end-of-data signal, so it must not end the
   walk, or a deep pull would silently truncate. Instead the offsets stop and a
   sequential cursor continuation takes over. Offsets have a ceiling; cursors
   don't. The hybrid is fast over the parallelizable prefix and complete over
   the rest.

Waves continue until one of these fires, so a 10-page result at width=4 is
three waves (4 + 4 + 2 useful), not one flat guess.

The ceiling seam needs a rewind

Worth spelling out, because the obvious implementation is subtly wrong — and it
was wrong here until an integration test caught it.

At the ceiling, the next offset the walk would need is by definition past the
ceiling. So it can't seed the continuation either — that request would earn the
same HTTP 400 the offset walk just avoided (offset=40010 > 40000). The walk
therefore rewinds one page: it drops the last page it fetched and re-seeds
the cursor walk at offsets[-1], the largest offset the service still accepts,
following next links from there. One page is re-fetched per deep query, in
exchange for a seam with neither a gap (missing rows) nor an overlap
(duplicates).

Risks, and what they cost

offset is a non-standard extension. Part 1 defines only limit and the
next relation. Worse, an unrecognized query parameter is conventionally
ignored rather than rejected — so a server that drops offset support would
answer every offset with page 1, and a naive walk would concatenate the same
rows N times and report success. Silent duplication is the worst failure
mode available to this design, so it is checked for directly: _offset_ignored
compares the first two full-length pages of the first wave, and identical frames
raise OffsetUnsupported. That happens before any rows are returned, so the
fallback re-walk cannot double-count. The query then completes via standard
next-link paging — slower, correct, and needing no extensions. A false
positive (two genuinely identical pages) costs a fallback, not an error.

The design does not assume the offset ceiling a priori in a way that can
break silently. max_offset is declared per service on OgcDialect, default
None, and None means "don't use offsets at all" — so the conservative path
is what an undeclared service gets. If USGS lowers the ceiling, requests past
the new limit fail loudly with the existing typed HTTP error rather than
truncating. If they raise or remove it, the current value just leaves some
speed on the table.

What could still regress: a server that honors offset inconsistently
across pages (rather than not at all) would slip past a check that samples the
first wave. Nothing in the API's behavior suggests that, and the alternative —
validating every page against its neighbors — would mean holding the whole
result to compare it. Flagging it as the known gap rather than papering over it.

What was removed

Removed	Why
parallel_chunks(n) (public, dataretrieval and waterdata)	Split a fitting request into more sub-requests. Quota-positive, and a no-op when there's no multi-value axis.
ChunkPlan.max_chunks	The dial's entry point into planning. Now a TypeError.
ChunkPlan._refine()	~65 lines of fan-out-driven splitting, with its own cap-overshoot arithmetic.

Byte-driven chunking is untouched and remains a correctness requirement —
the OGC edge WAF caps request bytes at ~8200 and returns HTTP 414 above it. Only
the parallelism half of the chunker is gone. tests/waterdata_chunking_test.py
now pins that split explicitly: test_byte_driven_chunking_survives_the_removal
and test_unchunkable_still_raised_without_the_dial.

Migration

# before
with waterdata.parallel_chunks(32):
    df, md = waterdata.get_daily(monitoring_location_id=sites, parameter_code="00060")

# after — delete the wrapper; the pages inside it are now overlapped
df, md = waterdata.get_daily(monitoring_location_id=sites, parameter_code="00060")

API_USGS_CONCURRENT now bounds the page-fetch wave width as well as
sub-request fan-out — one env var for everything in flight. API_USGS_CONCURRENT=1
pages strictly sequentially, via standard cursors (not offsets with a wave of
one). unbounded is clamped to a finite width here, because a wave is
speculative and an unbounded one would issue arbitrarily many past-the-end
requests to find a single short page.

Testing

ruff check clean, ruff format --check clean, mypy dataretrieval/ clean on
43 files, pytest tests/ 671 passed.

• 11 new unit tests (tests/transport_test.py) for the service-neutral
  walk: each stop condition, no gap or overlap across waves, the ceiling
  hand-off, warn-and-truncate without a continuation, refusing a server that
  ignores offset, accepting distinct equal-length pages, offset clipping, and
  page-failure wrapping.
• 7 new end-to-end tests (tests/waterdata_offset_paging_test.py) through
  get_daily, fully mocked. These exist because conftest.py pins
  API_USGS_CONCURRENT=1 suite-wide, so the existing 671 tests never touched
  the parallel path — the new feature had zero end-to-end coverage. The module
  immediately earned its place: it caught the ceiling-seam bug described above,
  where the tail walk was seeded one page too far and would have failed a real
  deep pull with HTTP 400.

Disclosed as incomplete: a head-to-head of offset paging against the removed
_refine fan-out on a large real multi-site pull was started and never
finished — it ran into the 1,000-request anonymous quota (HTTP 429,
retry-after: 290). The two measurements in the table above are complete and
repeated; that third comparison is not, and is not being claimed.

Docs

README.md and docs/source/userguide/errors.rst replace their
parallel_chunks sections with "large downloads are paged in parallel
automatically", covering quota neutrality, the API_USGS_CONCURRENT=1 escape
hatch, and both fallbacks. ADR 0006 gains the second page-walk strategy — why
both live in transport, and why the strategy choice is a dialect decision rather
than a transport one. NEWS.md carries the breaking-change entry.

🤖 Generated with Claude Code