Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 28 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@

## What is dataretrieval?

`dataretrieval` simplifies the process of loading hydrologic data into Python.
Like the original R version
[`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval), it retrieves major
U.S. Geological Survey (USGS) hydrology data types available on the Web, as well
as data from the Water Quality Portal (WQP), the National Ground-Water
Monitoring Network (NGWMN), and the Network Linked Data Index (NLDI).
`dataretrieval` simplifies loading hydrologic data into Python. Like the
original R version
[`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval), it retrieves the
major U.S. Geological Survey (USGS) hydrology data types available on the Web.
It also retrieves data from the Water Quality Portal (WQP), the National
Ground-Water Monitoring Network (NGWMN), and the Network Linked Data Index
(NLDI).

Check the [NEWS](NEWS.md) for all updates and announcements.

Expand Down Expand Up @@ -41,7 +42,7 @@ pip install git+https://github.com/DOI-USGS/dataretrieval-python.git

Access USGS water-monitoring data.

**Important:** Users are strongly encouraged to obtain an API key for higher
**Important:** We strongly encourage you to obtain an API key for higher
rate limits. [Register for an API key](https://api.waterdata.usgs.gov/signup/)
and set it as an environment variable:

Expand Down Expand Up @@ -110,14 +111,14 @@ print(f"Retrieved {len(df)} continuous gage height measurements")

By default the getters split a multi-value request only as far as the server's
~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated**
pull that is needlessly conservative: every sub-request pages through its own
results, so dividing the query into more, smaller sub-requests lets those pages
be fetched **in parallel**. `parallel_chunks(n)` opts a single call into that
finer split, fanning it out into `n` sub-requests. It pays off only when the
result is large enough to span many pages *and* the query has a multi-value
argument to divide (such as a list of monitoring locations); on a small query —
or one with nothing to split — it just adds requests, so it is a deliberate,
scoped `with` block, never the default.
pull, that default is needlessly conservative: every sub-request pages through
its own results, so dividing the query into more, smaller sub-requests lets
those pages be fetched **in parallel**. `parallel_chunks(n)` opts a single call
into that finer split, fanning it out into `n` sub-requests. The finer split
pays off only when the result is large enough to span many pages *and* the query
has a multi-value argument to divide, such as a list of monitoring locations. On
a small query — or one with nothing to split — it only adds requests, so
`parallel_chunks` is a deliberate, scoped `with` block, never the default.

```python
from dataretrieval import waterdata
Expand All @@ -134,17 +135,17 @@ with waterdata.parallel_chunks(32): # fan out into 32 sub-requests
)
```

`n` is the number of sub-requests to fan the call out into. It is capped by how
many values there are to split, and each sub-request costs a request against
your hourly [rate limit](https://api.waterdata.usgs.gov/signup/); since how many
run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32), the
useful range is roughly `2` up to that value.
`n` is the number of sub-requests to fan the call out into, capped by how many
values there are to split. Each sub-request costs a request against your hourly
[rate limit](https://api.waterdata.usgs.gov/signup/). How many run *at once* is
capped separately by `API_USGS_CONCURRENT` (default 32), so the useful range is
roughly `2` up to that value.

Benchmark — a fixed 271-site subset of Ohio stream gages
(`get_daily`, `parameter_code="00060"`), with a small fixed page size
(`limit=250`) so every run fetches roughly the same number of pages (isolating
the effect of parallelism). Each `n` was run against its own cold 1-year time
window so no run is served from the server's data-window cache:
the effect of parallelism). Each `n` ran against its own cold 1-year time
window, so no run is served from the server's data-window cache:

| `n` | parallelism | pages | wall-clock | speedup |
| ---- | ----------- | ----- | ----------------------- | ------- |
Expand All @@ -153,10 +154,10 @@ window so no run is served from the server's data-window cache:
| `32` | 32 | 54 | 1.2 s | ~8× |

The gain comes from overlapping each sub-request's per-page latency and
server-side work, so the exact multiplier scales with how many pages the pull
spansa larger pull (more pages) has more parallelism to exploit. The extra
sub-requests each cost quota, so reserve a large `n` for pulls you know are
large.
server-side work. The exact multiplier therefore scales with how many pages the
pull spans: a larger pull (more pages) has more parallelism to exploit. The
extra sub-requests each cost quota, so reserve a large `n` for pulls you know
are large.

Visit the
[API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)
Expand Down Expand Up @@ -327,7 +328,7 @@ directory, including Jupyter notebooks demonstrating advanced usage patterns.

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for
development guidelines.

## Acknowledgments
Expand Down
12 changes: 6 additions & 6 deletions dataretrieval/codes/states.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
"""State code lookups and normalization, keyed by full state name.

``state_codes`` maps each state name to its two-letter postal abbreviation
(e.g. ``"Alabama": "al"``); ``fips_codes`` maps it to its two-digit FIPS
code (e.g. ``"Alabama": "01"``). :func:`to_state` normalizes a state
identifier -- a full name, postal code, or two-digit / ``US:``-prefixed FIPS
code (or an iterable of them) -- to a chosen representation, raising
``ValueError`` on an unrecognized value. Coverage is the 50 states plus the
District of Columbia.
(e.g. ``"Alabama": "al"``); ``fips_codes`` maps the same names to their
two-digit FIPS codes (e.g. ``"Alabama": "01"``). :func:`to_state` normalizes
a state identifier -- a full name, postal code, or two-digit /
``US:``-prefixed FIPS code (or an iterable of them) -- to a chosen
representation. An unrecognized value raises ``ValueError``. Coverage is the
50 states plus the District of Columbia.
"""

from __future__ import annotations
Expand Down
4 changes: 1 addition & 3 deletions dataretrieval/codes/timezones.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
"""
Time zone information
"""
"""Time zone information."""

tz_str = """-1200 Y
-1100 X NUT SST
Expand Down
24 changes: 12 additions & 12 deletions dataretrieval/combining.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@
These utilities assemble the output of a chunked/fan-out call from its
individual per-sub-request results. They have no event-loop, retry, or
network state — they're pure data transforms shared by protocol-specific
chunk execution, service fan-out, and
cursor-driven pagination.
chunk execution, service fan-out, and cursor-driven pagination.

Separated from :mod:`dataretrieval.ogc.planning` so that module stays
focused on *what* to split, while this module owns *how* to reassemble.
Expand Down Expand Up @@ -101,13 +100,14 @@ def _merge_response(
elapsed: timedelta,
url: str | httpx.URL | None = None,
) -> httpx.Response:
"""Fold several responses into one: a shallow copy of ``base`` whose
``.headers`` are rebuilt as a fresh ``httpx.Headers`` from ``headers_from``,
``.elapsed`` set to ``elapsed``, and ``.url`` overridden when ``url`` is
given. ``base`` and ``headers_from`` are never mutated, and the fresh
``httpx.Headers`` means downstream mutations don't back-propagate into any
underlying response — so callers may re-fold idempotently. This is the one
low-level merge behind both pagination
"""Fold several responses into one shallow copy of ``base``.

The copy's ``.headers`` are rebuilt as a fresh ``httpx.Headers`` from
``headers_from``, ``.elapsed`` is set to ``elapsed``, and ``.url`` is
overridden when ``url`` is given. ``base`` and ``headers_from`` are never
mutated, and the fresh ``httpx.Headers`` means downstream mutations don't
back-propagate into any underlying response — so callers may re-fold
idempotently. This is the one low-level merge behind both pagination
(:func:`~dataretrieval.transport.pagination.paginate`) and the chunked /
fan-out aggregation (:func:`_combine_chunk_responses`)."""
merged = copy.copy(base)
Expand Down Expand Up @@ -196,9 +196,9 @@ def _combine_chunk_responses(
if len(responses) == 1 and canonical_url is None:
return responses[0]

# Headers come from the response with the lowest reported remaining quota;
# ``_lowest_remaining`` returns the lone response as-is
# for a single-element list). ``_merge_response`` re-sums elapsed onto a
# Headers come from the response with the lowest reported remaining quota
# (``_lowest_remaining`` returns the lone response as-is for a
# single-element list). ``_merge_response`` re-sums elapsed onto a
# fresh copy, so repeated calls (e.g. via ``ChunkedCall.partial_response``
# during resume) stay idempotent.
elapsed = sum((_safe_elapsed(r) for r in responses), start=timedelta())
Expand Down
2 changes: 1 addition & 1 deletion dataretrieval/credentials.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def accepts_api_key(target_url: str | httpx.URL | None) -> bool:
url = target_url if isinstance(target_url, httpx.URL) else httpx.URL(target_url)
except (httpx.InvalidURL, TypeError):
return False
return url.scheme == "https" and url.host == _AUTHORIZED_API_KEY_HOST
return bool(url.scheme == "https" and url.host == _AUTHORIZED_API_KEY_HOST)


def without_embedded_credentials(url: httpx.URL) -> httpx.URL:
Expand Down
40 changes: 23 additions & 17 deletions dataretrieval/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

Every service module (``nwis``, ``wqp``, ``nldi``, ``waterdata``,
``streamstats``) raises a subclass of :class:`DataRetrievalError` when a request
fails, so one ``except dataretrieval.DataRetrievalError`` catches them all --
including connection-level failures (timeouts, DNS, refused connections), which
fails, so one ``except dataretrieval.DataRetrievalError`` catches them all. That
includes connection-level failures (timeouts, DNS, refused connections), which
are wrapped as :class:`NetworkError` with the underlying ``httpx`` exception on
``__cause__``.

Expand All @@ -15,8 +15,8 @@
:func:`error_for_status` maps a status to its type.

This module has no third-party runtime dependencies -- ``httpx`` is imported only
for type checking -- so any module can import it without pulling in pandas / httpx
and without risking an import cycle.
for type checking. Any module can therefore import it without pulling in pandas
or httpx, and without risking an import cycle.
"""

from __future__ import annotations
Expand Down Expand Up @@ -94,8 +94,10 @@ def __setstate__(self, state: dict[str, Any] | None) -> None:


def _new_error(cls: type[DataRetrievalError]) -> DataRetrievalError:
"""Build a blank :class:`DataRetrievalError` for unpickling, bypassing
``__init__``; pickle then calls ``__setstate__`` to restore its state."""
"""Build a blank :class:`DataRetrievalError` for unpickling.

Bypasses ``__init__``; pickle then calls ``__setstate__`` to restore state.
"""
return cls.__new__(cls)


Expand All @@ -110,8 +112,8 @@ class HTTPError(DataRetrievalError):
(429 / 5xx) is the retryable subset, and is itself an ``HTTPError``. The one
exception to "a status is an ``HTTPError``" is a request the service rejects
as too long: it surfaces as :class:`URLTooLong` (a :class:`RequestTooLarge`),
*not* an ``HTTPError`` -- so catch :class:`DataRetrievalError` to be certain
of spanning every failure. See :func:`error_for_status` for the full mapping.
*not* an ``HTTPError``. Catch :class:`DataRetrievalError` to be certain of
spanning every failure. See :func:`error_for_status` for the full mapping.

Parameters
----------
Expand All @@ -127,8 +129,9 @@ def __init__(self, message: str, *, status_code: int) -> None:


class TransientError(HTTPError):
"""A 429 or 5xx the server may serve on a later try -- :class:`RateLimited`
for 429, :class:`ServiceUnavailable` for 5xx.
"""A 429 or 5xx the server may serve on a later try.

:class:`RateLimited` covers 429 and :class:`ServiceUnavailable` covers 5xx.

This only classifies the condition; it does not itself retry. Whether to
retry is up to the calling path: a single-shot request raises it for the
Expand Down Expand Up @@ -219,8 +222,8 @@ class Unchunkable(RequestTooLarge):

Raised by the Water Data chunker when even the smallest reducible plan
(every list axis at one atom per sub-request, the filter at one clause per
sub-request) still exceeds the server's byte limit -- so unlike
:class:`URLTooLong`, automatic splitting has already been tried and
sub-request) still exceeds the server's byte limit. Unlike
:class:`URLTooLong`, then, automatic splitting has already been tried and
exhausted. Shrink the input lists, simplify the filter, or split the call
manually.
"""
Expand All @@ -230,9 +233,10 @@ class Unchunkable(RequestTooLarge):


class NetworkError(DataRetrievalError):
"""The request never completed a round-trip to the service -- a DNS
failure, refused connection, or timeout -- so no HTTP response arrived to
classify.
"""The request never completed a round-trip to the service.

A DNS failure, refused connection, or timeout stopped it, so no HTTP
response arrived to classify.

Wraps the underlying ``httpx`` transport exception, preserved on
``__cause__``. Worth retrying (:attr:`~DataRetrievalError.retryable` is
Expand All @@ -246,8 +250,10 @@ class NetworkError(DataRetrievalError):


class ConfigurationError(DataRetrievalError, ValueError):
"""A ``dataretrieval`` setting -- an environment variable, a policy field --
holds a value that can't be used, so no request was issued.
"""A ``dataretrieval`` setting holds a value that can't be used.

The setting may be an environment variable or a policy field; either way,
no request was issued.

It is a :class:`DataRetrievalError` so ``except`` around a retrieval catches
it rather than letting a bare ``ValueError`` escape a request path, and a
Expand Down
9 changes: 4 additions & 5 deletions dataretrieval/ngwmn.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
"""National Ground-Water Monitoring Network (NGWMN) getters.
"""Retrieve data from the National Ground-Water Monitoring Network (NGWMN).

The NGWMN exposes its data through a dedicated OGC API
(``https://api.waterdata.usgs.gov/ngwmn/ogcapi``) with five collections:
``sites``, ``waterLevelObs``, ``lithologyObs``, ``constructionObs``, and
``providers``. Each getter below delegates to the shared OGC facade
(:func:`~dataretrieval.ogc.get_ogc_data`) with
``base_url=NGWMN_OGC_API_URL``, so multi-value chunking, pagination,
retry/resume, and result shaping all behave exactly as they do for the main
Water Data getters.
(:func:`~dataretrieval.ogc.get_ogc_data`) with ``base_url=NGWMN_OGC_API_URL``.
Multi-value chunking, pagination, retry/resume, and result shaping therefore
behave exactly as they do for the main Water Data getters.

Unlike the main Water Data collections, NGWMN aggregates monitoring locations
from many agencies, so ``monitoring_location_id`` values use other agency
Expand Down
40 changes: 28 additions & 12 deletions dataretrieval/nldi.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
"""Retrieve hydrologic network features from the Network Linked Data Index (NLDI).

The getters below navigate the hydrologic network from an origin -- a feature
source and id, a ``comid``, or a lat/long point -- and return flowlines, basins,
or registered features as a ``geopandas.GeoDataFrame``, or as raw JSON when
``as_json=True``. This module requires geopandas.

See https://api.water.usgs.gov/nldi/linked-data for the API reference.
"""

from __future__ import annotations

from json import JSONDecodeError
Expand Down Expand Up @@ -48,7 +58,7 @@ def _features_to_gdf(feature_collection: dict[str, Any]) -> gpd.GeoDataFrame:

NLDI can legitimately return no features (e.g. a feature with nothing
upstream), and :func:`_query_nldi` returns ``{}`` when a 200 response
carries no JSON body. ``GeoDataFrame.from_features`` raises on those
carries no JSON body. ``GeoDataFrame.from_features`` raises on both cases
(there's no geometry column to attach the CRS to), so return an empty
GeoDataFrame with the correct CRS instead of crashing.
"""
Expand All @@ -68,8 +78,10 @@ def get_flowlines(
trim_start: bool = False,
as_json: bool = False,
) -> gpd.GeoDataFrame | dict[str, Any]:
"""Gets the flowlines for the specified navigation either by comid or feature
source in WGS84 lat/long coordinates as GeoDataFrame containing a polyline geometry.
"""Get the flowlines for a navigation, either by comid or by feature source.

Flowlines are returned in WGS84 lat/long coordinates as a GeoDataFrame
containing a polyline geometry.

Parameters
----------
Expand Down Expand Up @@ -133,8 +145,10 @@ def get_basin(
split_catchment: bool = False,
as_json: bool = False,
) -> gpd.GeoDataFrame | dict[str, Any]:
"""Gets the aggregated basin for the specified feature in WGS84 lat/lon
as GeoDataFrame or as JSON containing a polygon geometry.
"""Get the aggregated basin for the specified feature.

The basin is returned in WGS84 lat/lon as a GeoDataFrame or as JSON,
containing a polygon geometry.

Parameters
----------
Expand Down Expand Up @@ -190,9 +204,10 @@ def get_features(
stop_comid: int | None = None,
as_json: bool = False,
) -> gpd.GeoDataFrame | dict[str, Any]:
"""Gets all features found along the specified navigation either by
comid or feature source as points in WGS84 lat/long coordinates - a GeoDataFrame
containing a point geometry.
"""Get all features along a navigation, either by comid or by feature source.

Features are returned as points in WGS84 lat/long coordinates - a
GeoDataFrame containing a point geometry.

Parameters
----------
Expand Down Expand Up @@ -295,8 +310,10 @@ def get_features(
# TODO: This function can cause a timeout error for some data sources
# - maybe we shouldn't provide this function?
def get_features_by_data_source(data_source: str) -> gpd.GeoDataFrame:
"""Gets all features found for the specified data source as
points in WGS84 lat/long coordinates as GeoDataFrame containing a point geometry.
"""Get all features for the specified data source.

Features are returned as points in WGS84 lat/long coordinates as a
GeoDataFrame containing a point geometry.

Parameters
----------
Expand Down Expand Up @@ -336,8 +353,7 @@ def search(
long: float | None = None,
distance: int = 50,
) -> dict[str, Any]:
"""Searches for the specified feature in NLDI and returns the results
as a dictionary.
"""Search NLDI for the specified feature and return the results as a dict.

Parameters
----------
Expand Down
Loading