Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,9 @@ link checking.
* Group public download functions by data portal. For example, modern Water
Data functions belong in `dataretrieval.waterdata`; legacy NWIS functions
remain quarantined in `dataretrieval.nwis` during deprecation.
* Treat a change to a service's documented return shape or metadata type as a
public compatibility change; update contract tests and architecture
documentation and follow the deprecation process where required.
* Preserve the dependency direction documented in
[`docs/source/architecture`](docs/source/architecture/index.rst): public
facades depend on service/protocol adapters, which depend on stable shared
Expand Down
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
**08/03/2026:** Split the Water Data implementation into focused time-series, metadata, measurements, reference, samples, and CQL collection-family modules behind the unchanged `waterdata.api` facade. Active service modules now declare explicit exports; public Water Data imports, signatures, function identities, deprecations, and return contracts are protected by executable contract snapshots. OGC ambient context and schema/queryables execution are separated from request construction, adapter-to-adapter reach-through is prohibited by architecture tests, and service-specific output shapes are documented rather than forced into one model.

**08/02/2026:** Added an internal API-neutral transport layer for guarded HTTP clients, host-scoped authentication, cursor pagination, bounded retry, response aggregation, progress, and sync-over-async dispatch. Water Use and the non-OGC Statistics API now consume transport directly instead of private OGC execution helpers; WQP, NLDI, and StreamStats opt into bounded transient retry while deprecated NWIS behavior remains unchanged. OGC retains CQL2, request construction, feature shaping, chunk planning, resumable calls, and interruption types, with compatibility imports at previous private paths. Failed pagination and fan-out still raise rather than returning partial data, and no public signatures or return shapes changed.

**08/02/2026:** Phase 1 OGC boundary stabilization: the `dataretrieval.ogc` package now exposes a deliberate, small facade (`OgcDialect`, `prepare_request_args`, `get_ogc_data`, `fetch_ogc_request`) that service adapters (NGWMN, Water Data's generic wrapper) import. Internal request-construction helpers moved to a new `ogc.requests` module; the dialect type and endpoint constants live in the leaf `ogc.policy` module. `ogc.shaping` no longer depends on `ogc.engine` at all, and the complete runtime OGC import graph is now acyclic. `_default_headers` now accepts a target URL and adds `X-Api-Key` only for `api.waterdata.usgs.gov`; shared sync and async clients also strip the key before following any cross-host redirect, including external rating-asset downloads. `waterdata.utils` no longer bulk-re-exports private OGC symbols, and consumers import implementation helpers from their canonical modules. No public API, return-value, or deprecation changes.
Expand Down
15 changes: 14 additions & 1 deletion dataretrieval/ngwmn.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,21 @@
from dataretrieval.ogc import OgcDialect, get_ogc_data, prepare_request_args
from dataretrieval.utils import BaseMetadata

__all__ = [
"get_sites",
"get_water_level",
"get_lithology",
"get_well_construction",
"get_providers",
]


# The Water Data API base URL, from the credentials leaf rather than OGC policy
# internals: it names the same authority the API key is scoped to.
# internals: it names the same authority the API key is scoped to. Spelling the
# host out here instead would put a second copy of it in the package, which
# ``tests/architecture_test.py::test_credential_policy_has_one_definition``
# rejects: the code that attaches the API key and the code that strips it at
# redirect time must not be able to disagree about which host is authorized.
BASE_URL = WATERDATA_BASE_URL

# The National Ground-Water Monitoring Network exposes its own OGC API at a
Expand Down
9 changes: 9 additions & 0 deletions dataretrieval/nldi.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@

from dataretrieval.utils import _query_with_retry

__all__ = [
"get_flowlines",
"get_basin",
"get_features",
"get_features_by_data_source",
"search",
]


try:
import geopandas as gpd
except ImportError as err:
Expand Down
20 changes: 20 additions & 0 deletions dataretrieval/ogc/context.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
"""Per-call OGC request state, scoped rather than passed.

The base URL, dialect, and row cap apply to a whole call but are read deep
inside request construction, several frames below whoever set them. Threading
them through every signature would put protocol plumbing in the getters, so they
travel as context variables -- which also makes them safe under the concurrent
fan-out, where a thread-global would not be.
"""

from dataretrieval.ogc.policy import DEFAULT_DIALECT, OGC_API_URL, OgcDialect
from dataretrieval.utils import Ambient

# Optional cap on rows accumulated by one paginated request.
_row_cap: Ambient[int | None] = Ambient("ogc_row_cap", None)

# OGC base URL targeted by request construction and schema lookup.
_ogc_base_url: Ambient[str] = Ambient("ogc_base_url", OGC_API_URL)

# Per-call request and response dialect.
_dialect: Ambient[OgcDialect] = Ambient("ogc_dialect", DEFAULT_DIALECT)
3 changes: 1 addition & 2 deletions dataretrieval/ogc/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
import dataretrieval.progress as _progress
from dataretrieval.credentials import without_embedded_credentials
from dataretrieval.ogc.chunking import get_active_client
from dataretrieval.ogc.context import _row_cap
from dataretrieval.ogc.errors import _raise_for_non_200
from dataretrieval.ogc.policy import (
BASE_URL, # noqa: F401 — compatibility alias
Expand All @@ -52,7 +53,6 @@
_NO_NORMALIZE_PARAMS,
_as_str_list,
_check_monitoring_location_id,
_check_ogc_requests,
_construct_api_requests,
_construct_cql_request,
_cql2_param,
Expand All @@ -61,7 +61,6 @@
_normalize_str_iterable,
_ogc_base_url,
_ogc_query_params,
_row_cap,
_switch_arg_id,
_switch_properties_id,
prepare_request_args,
Expand Down
65 changes: 10 additions & 55 deletions dataretrieval/ogc/requests.py
Original file line number Diff line number Diff line change
@@ -1,58 +1,25 @@
"""OGC request preparation, construction, and schema/queryables lookup.
"""OGC argument normalization and HTTP request construction.

This module owns the machinery for building OGC API requests (both GET and
POST/CQL2 paths), the ambient base-URL and dialect state that request builders
read, and the queryables/schema request helper used by empty-result shaping.

It depends on :mod:`~dataretrieval.ogc.policy` (the dialect type and endpoint
constants), :mod:`~dataretrieval.ogc.dates`, :mod:`~dataretrieval.ogc.errors`,
and :mod:`~dataretrieval.utils` (shared HTTP primitives). It must NOT import
engine or shaping.
Ambient request state lives in :mod:`dataretrieval.ogc.context`; queryables and
schema execution live in :mod:`dataretrieval.ogc.schema`. Neither is re-exported
from here -- importing the schema helper only to forward it would give this
module an edge to the one part of OGC that executes HTTP, which is exactly what
request *construction* is supposed to be free of.
"""

from __future__ import annotations

import json
import logging
import re
from collections.abc import Iterable, Mapping
from typing import Any, cast
from typing import Any

import httpx

from dataretrieval.ogc.context import _dialect as _dialect
from dataretrieval.ogc.context import _ogc_base_url as _ogc_base_url
from dataretrieval.ogc.dates import _DATE_RANGE_PARAMS, _format_api_dates
from dataretrieval.ogc.errors import _raise_for_non_200
from dataretrieval.ogc.policy import DEFAULT_DIALECT, OGC_API_URL, OgcDialect
from dataretrieval.transport.http import (
HTTPX_DEFAULTS,
)
from dataretrieval.transport.http import (
default_headers as _default_headers,
)
from dataretrieval.transport.http import (
get as _get,
)
from dataretrieval.utils import Ambient

logger = logging.getLogger(__name__)

# ---------------------------------------------------------------------------
# Ambient per-call state
# ---------------------------------------------------------------------------

# Optional cap on the rows one paginated call accumulates before it stops
# following ``next`` links (``None`` = uncapped). Set by :func:`get_reference_table`
# to preview large tables without downloading every page.
_row_cap: Ambient[int | None] = Ambient("ogc_row_cap", None)

# OGC base URL the shared request builder (:func:`_construct_api_requests`)
# targets — the main Water Data API or, for NGWMN collections, their own base.
_ogc_base_url: Ambient[str] = Ambient("ogc_base_url", OGC_API_URL)

# Per-call OGC dialect the request builder reads for CQL2-vs-GET routing and
# date-only formatting (default: a plain OGC API).
_dialect: Ambient[OgcDialect] = Ambient("ogc_dialect", DEFAULT_DIALECT)

from dataretrieval.transport.http import default_headers as _default_headers

# ---------------------------------------------------------------------------
# Monitoring location ID validation
Expand Down Expand Up @@ -221,18 +188,6 @@ def _construct_cql_request(
)


def _check_ogc_requests(
endpoint: str, req_type: str = "queryables"
) -> tuple[dict[str, Any], httpx.Response]:
"""Send an HTTP GET request to the OGC endpoint for queryables/schema."""
if req_type not in ("queryables", "schema"):
raise ValueError(f"req_type must be 'queryables' or 'schema', got {req_type!r}")
url = f"{_ogc_base_url.get()}/collections/{endpoint}/{req_type}"
resp = _get(url, headers=_default_headers(url), **HTTPX_DEFAULTS)
_raise_for_non_200(resp)
return cast("dict[str, Any]", resp.json()), resp


# ---------------------------------------------------------------------------
# Argument normalization helpers
# ---------------------------------------------------------------------------
Expand Down
30 changes: 30 additions & 0 deletions dataretrieval/ogc/schema.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
"""Asking an OGC service to describe itself.

Queryables and collection schemas: which properties a collection accepts, and
what columns it returns. Separate from request construction because answering
these questions means *issuing* a request, and building one must not.
"""

from __future__ import annotations

from typing import Any, cast

import httpx

from dataretrieval.ogc.context import _ogc_base_url
from dataretrieval.ogc.errors import _raise_for_non_200
from dataretrieval.transport.http import HTTPX_DEFAULTS
from dataretrieval.transport.http import default_headers as _default_headers
from dataretrieval.transport.http import get as _get


def _check_ogc_requests(
endpoint: str, req_type: str = "queryables"
) -> tuple[dict[str, Any], httpx.Response]:
"""Retrieve one collection's queryables or response schema."""
if req_type not in ("queryables", "schema"):
raise ValueError(f"req_type must be 'queryables' or 'schema', got {req_type!r}")
url = f"{_ogc_base_url.get()}/collections/{endpoint}/{req_type}"
response = _get(url, headers=_default_headers(url), **HTTPX_DEFAULTS)
_raise_for_non_200(response)
return cast("dict[str, Any]", response.json()), response
2 changes: 1 addition & 1 deletion dataretrieval/ogc/shaping.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ def _deal_with_empty(
if return_list.empty:
if not properties or all(pd.isna(properties)):
# Import from requests module (no engine dependency).
from dataretrieval.ogc.requests import _check_ogc_requests
from dataretrieval.ogc.schema import _check_ogc_requests

schema, _ = _check_ogc_requests(endpoint=service, req_type="schema")
properties = list(schema.get("properties", {}).keys())
Expand Down
2 changes: 2 additions & 0 deletions dataretrieval/streamstats.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
from dataretrieval.transport.http import HTTPX_DEFAULTS
from dataretrieval.utils import _get_with_retry

__all__ = ["download_workspace", "get_sample_watershed", "get_watershed", "Watershed"]


def download_workspace(workspaceID: str, format: str = "") -> httpx.Response:
"""Function to download a StreamStats workspace.
Expand Down
Loading