diff --git a/README.md b/README.md index 88b076ab2..41b3299c6 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,13 @@ ## What is dataretrieval? -`dataretrieval` simplifies the process of loading hydrologic data into Python. -Like the original R version -[`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval), it retrieves major -U.S. Geological Survey (USGS) hydrology data types available on the Web, as well -as data from the Water Quality Portal (WQP), the National Ground-Water -Monitoring Network (NGWMN), and the Network Linked Data Index (NLDI). +`dataretrieval` simplifies loading hydrologic data into Python. Like the +original R version +[`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval), it retrieves the +major U.S. Geological Survey (USGS) hydrology data types available on the Web. +It also retrieves data from the Water Quality Portal (WQP), the National +Ground-Water Monitoring Network (NGWMN), and the Network Linked Data Index +(NLDI). Check the [NEWS](NEWS.md) for all updates and announcements. @@ -41,7 +42,7 @@ pip install git+https://github.com/DOI-USGS/dataretrieval-python.git Access USGS water-monitoring data. -**Important:** Users are strongly encouraged to obtain an API key for higher +**Important:** We strongly encourage you to obtain an API key for higher rate limits. [Register for an API key](https://api.waterdata.usgs.gov/signup/) and set it as an environment variable: @@ -110,14 +111,14 @@ print(f"Retrieved {len(df)} continuous gage height measurements") By default the getters split a multi-value request only as far as the server's ~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated** -pull that is needlessly conservative: every sub-request pages through its own -results, so dividing the query into more, smaller sub-requests lets those pages -be fetched **in parallel**. `parallel_chunks(n)` opts a single call into that -finer split, fanning it out into `n` sub-requests. It pays off only when the -result is large enough to span many pages *and* the query has a multi-value -argument to divide (such as a list of monitoring locations); on a small query — -or one with nothing to split — it just adds requests, so it is a deliberate, -scoped `with` block, never the default. +pull, that default is needlessly conservative: every sub-request pages through +its own results, so dividing the query into more, smaller sub-requests lets +those pages be fetched **in parallel**. `parallel_chunks(n)` opts a single call +into that finer split, fanning it out into `n` sub-requests. The finer split +pays off only when the result is large enough to span many pages *and* the query +has a multi-value argument to divide, such as a list of monitoring locations. On +a small query — or one with nothing to split — it only adds requests, so +`parallel_chunks` is a deliberate, scoped `with` block, never the default. ```python from dataretrieval import waterdata @@ -134,17 +135,17 @@ with waterdata.parallel_chunks(32): # fan out into 32 sub-requests ) ``` -`n` is the number of sub-requests to fan the call out into. It is capped by how -many values there are to split, and each sub-request costs a request against -your hourly [rate limit](https://api.waterdata.usgs.gov/signup/); since how many -run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32), the -useful range is roughly `2` up to that value. +`n` is the number of sub-requests to fan the call out into, capped by how many +values there are to split. Each sub-request costs a request against your hourly +[rate limit](https://api.waterdata.usgs.gov/signup/). How many run *at once* is +capped separately by `API_USGS_CONCURRENT` (default 32), so the useful range is +roughly `2` up to that value. Benchmark — a fixed 271-site subset of Ohio stream gages (`get_daily`, `parameter_code="00060"`), with a small fixed page size (`limit=250`) so every run fetches roughly the same number of pages (isolating -the effect of parallelism). Each `n` was run against its own cold 1-year time -window so no run is served from the server's data-window cache: +the effect of parallelism). Each `n` ran against its own cold 1-year time +window, so no run is served from the server's data-window cache: | `n` | parallelism | pages | wall-clock | speedup | | ---- | ----------- | ----- | ----------------------- | ------- | @@ -153,10 +154,10 @@ window so no run is served from the server's data-window cache: | `32` | 32 | 54 | 1.2 s | ~8× | The gain comes from overlapping each sub-request's per-page latency and -server-side work, so the exact multiplier scales with how many pages the pull -spans — a larger pull (more pages) has more parallelism to exploit. The extra -sub-requests each cost quota, so reserve a large `n` for pulls you know are -large. +server-side work. The exact multiplier therefore scales with how many pages the +pull spans: a larger pull (more pages) has more parallelism to exploit. The +extra sub-requests each cost quota, so reserve a large `n` for pulls you know +are large. Visit the [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html) @@ -327,7 +328,7 @@ directory, including Jupyter notebooks demonstrating advanced usage patterns. ## Contributing -Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for +Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for development guidelines. ## Acknowledgments diff --git a/dataretrieval/codes/states.py b/dataretrieval/codes/states.py index fe656232c..79e0a18dd 100644 --- a/dataretrieval/codes/states.py +++ b/dataretrieval/codes/states.py @@ -1,12 +1,12 @@ """State code lookups and normalization, keyed by full state name. ``state_codes`` maps each state name to its two-letter postal abbreviation -(e.g. ``"Alabama": "al"``); ``fips_codes`` maps it to its two-digit FIPS -code (e.g. ``"Alabama": "01"``). :func:`to_state` normalizes a state -identifier -- a full name, postal code, or two-digit / ``US:``-prefixed FIPS -code (or an iterable of them) -- to a chosen representation, raising -``ValueError`` on an unrecognized value. Coverage is the 50 states plus the -District of Columbia. +(e.g. ``"Alabama": "al"``); ``fips_codes`` maps the same names to their +two-digit FIPS codes (e.g. ``"Alabama": "01"``). :func:`to_state` normalizes +a state identifier -- a full name, postal code, or two-digit / +``US:``-prefixed FIPS code (or an iterable of them) -- to a chosen +representation. An unrecognized value raises ``ValueError``. Coverage is the +50 states plus the District of Columbia. """ from __future__ import annotations diff --git a/dataretrieval/codes/timezones.py b/dataretrieval/codes/timezones.py index 3f914217e..22907ebd9 100644 --- a/dataretrieval/codes/timezones.py +++ b/dataretrieval/codes/timezones.py @@ -1,6 +1,4 @@ -""" -Time zone information -""" +"""Time zone information.""" tz_str = """-1200 Y -1100 X NUT SST diff --git a/dataretrieval/combining.py b/dataretrieval/combining.py index 92e32f7aa..6be927607 100644 --- a/dataretrieval/combining.py +++ b/dataretrieval/combining.py @@ -3,8 +3,7 @@ These utilities assemble the output of a chunked/fan-out call from its individual per-sub-request results. They have no event-loop, retry, or network state — they're pure data transforms shared by protocol-specific -chunk execution, service fan-out, and -cursor-driven pagination. +chunk execution, service fan-out, and cursor-driven pagination. Separated from :mod:`dataretrieval.ogc.planning` so that module stays focused on *what* to split, while this module owns *how* to reassemble. @@ -101,13 +100,14 @@ def _merge_response( elapsed: timedelta, url: str | httpx.URL | None = None, ) -> httpx.Response: - """Fold several responses into one: a shallow copy of ``base`` whose - ``.headers`` are rebuilt as a fresh ``httpx.Headers`` from ``headers_from``, - ``.elapsed`` set to ``elapsed``, and ``.url`` overridden when ``url`` is - given. ``base`` and ``headers_from`` are never mutated, and the fresh - ``httpx.Headers`` means downstream mutations don't back-propagate into any - underlying response — so callers may re-fold idempotently. This is the one - low-level merge behind both pagination + """Fold several responses into one shallow copy of ``base``. + + The copy's ``.headers`` are rebuilt as a fresh ``httpx.Headers`` from + ``headers_from``, ``.elapsed`` is set to ``elapsed``, and ``.url`` is + overridden when ``url`` is given. ``base`` and ``headers_from`` are never + mutated, and the fresh ``httpx.Headers`` means downstream mutations don't + back-propagate into any underlying response — so callers may re-fold + idempotently. This is the one low-level merge behind both pagination (:func:`~dataretrieval.transport.pagination.paginate`) and the chunked / fan-out aggregation (:func:`_combine_chunk_responses`).""" merged = copy.copy(base) @@ -196,9 +196,9 @@ def _combine_chunk_responses( if len(responses) == 1 and canonical_url is None: return responses[0] - # Headers come from the response with the lowest reported remaining quota; - # ``_lowest_remaining`` returns the lone response as-is - # for a single-element list). ``_merge_response`` re-sums elapsed onto a + # Headers come from the response with the lowest reported remaining quota + # (``_lowest_remaining`` returns the lone response as-is for a + # single-element list). ``_merge_response`` re-sums elapsed onto a # fresh copy, so repeated calls (e.g. via ``ChunkedCall.partial_response`` # during resume) stay idempotent. elapsed = sum((_safe_elapsed(r) for r in responses), start=timedelta()) diff --git a/dataretrieval/credentials.py b/dataretrieval/credentials.py index 82021b4e4..ffb3df02f 100644 --- a/dataretrieval/credentials.py +++ b/dataretrieval/credentials.py @@ -61,7 +61,7 @@ def accepts_api_key(target_url: str | httpx.URL | None) -> bool: url = target_url if isinstance(target_url, httpx.URL) else httpx.URL(target_url) except (httpx.InvalidURL, TypeError): return False - return url.scheme == "https" and url.host == _AUTHORIZED_API_KEY_HOST + return bool(url.scheme == "https" and url.host == _AUTHORIZED_API_KEY_HOST) def without_embedded_credentials(url: httpx.URL) -> httpx.URL: diff --git a/dataretrieval/exceptions.py b/dataretrieval/exceptions.py index b40d62c4f..372171a09 100644 --- a/dataretrieval/exceptions.py +++ b/dataretrieval/exceptions.py @@ -2,8 +2,8 @@ Every service module (``nwis``, ``wqp``, ``nldi``, ``waterdata``, ``streamstats``) raises a subclass of :class:`DataRetrievalError` when a request -fails, so one ``except dataretrieval.DataRetrievalError`` catches them all -- -including connection-level failures (timeouts, DNS, refused connections), which +fails, so one ``except dataretrieval.DataRetrievalError`` catches them all. That +includes connection-level failures (timeouts, DNS, refused connections), which are wrapped as :class:`NetworkError` with the underlying ``httpx`` exception on ``__cause__``. @@ -15,8 +15,8 @@ :func:`error_for_status` maps a status to its type. This module has no third-party runtime dependencies -- ``httpx`` is imported only -for type checking -- so any module can import it without pulling in pandas / httpx -and without risking an import cycle. +for type checking. Any module can therefore import it without pulling in pandas +or httpx, and without risking an import cycle. """ from __future__ import annotations @@ -94,8 +94,10 @@ def __setstate__(self, state: dict[str, Any] | None) -> None: def _new_error(cls: type[DataRetrievalError]) -> DataRetrievalError: - """Build a blank :class:`DataRetrievalError` for unpickling, bypassing - ``__init__``; pickle then calls ``__setstate__`` to restore its state.""" + """Build a blank :class:`DataRetrievalError` for unpickling. + + Bypasses ``__init__``; pickle then calls ``__setstate__`` to restore state. + """ return cls.__new__(cls) @@ -110,8 +112,8 @@ class HTTPError(DataRetrievalError): (429 / 5xx) is the retryable subset, and is itself an ``HTTPError``. The one exception to "a status is an ``HTTPError``" is a request the service rejects as too long: it surfaces as :class:`URLTooLong` (a :class:`RequestTooLarge`), - *not* an ``HTTPError`` -- so catch :class:`DataRetrievalError` to be certain - of spanning every failure. See :func:`error_for_status` for the full mapping. + *not* an ``HTTPError``. Catch :class:`DataRetrievalError` to be certain of + spanning every failure. See :func:`error_for_status` for the full mapping. Parameters ---------- @@ -127,8 +129,9 @@ def __init__(self, message: str, *, status_code: int) -> None: class TransientError(HTTPError): - """A 429 or 5xx the server may serve on a later try -- :class:`RateLimited` - for 429, :class:`ServiceUnavailable` for 5xx. + """A 429 or 5xx the server may serve on a later try. + + :class:`RateLimited` covers 429 and :class:`ServiceUnavailable` covers 5xx. This only classifies the condition; it does not itself retry. Whether to retry is up to the calling path: a single-shot request raises it for the @@ -219,8 +222,8 @@ class Unchunkable(RequestTooLarge): Raised by the Water Data chunker when even the smallest reducible plan (every list axis at one atom per sub-request, the filter at one clause per - sub-request) still exceeds the server's byte limit -- so unlike - :class:`URLTooLong`, automatic splitting has already been tried and + sub-request) still exceeds the server's byte limit. Unlike + :class:`URLTooLong`, then, automatic splitting has already been tried and exhausted. Shrink the input lists, simplify the filter, or split the call manually. """ @@ -230,9 +233,10 @@ class Unchunkable(RequestTooLarge): class NetworkError(DataRetrievalError): - """The request never completed a round-trip to the service -- a DNS - failure, refused connection, or timeout -- so no HTTP response arrived to - classify. + """The request never completed a round-trip to the service. + + A DNS failure, refused connection, or timeout stopped it, so no HTTP + response arrived to classify. Wraps the underlying ``httpx`` transport exception, preserved on ``__cause__``. Worth retrying (:attr:`~DataRetrievalError.retryable` is @@ -246,8 +250,10 @@ class NetworkError(DataRetrievalError): class ConfigurationError(DataRetrievalError, ValueError): - """A ``dataretrieval`` setting -- an environment variable, a policy field -- - holds a value that can't be used, so no request was issued. + """A ``dataretrieval`` setting holds a value that can't be used. + + The setting may be an environment variable or a policy field; either way, + no request was issued. It is a :class:`DataRetrievalError` so ``except`` around a retrieval catches it rather than letting a bare ``ValueError`` escape a request path, and a diff --git a/dataretrieval/ngwmn.py b/dataretrieval/ngwmn.py index c2b437b96..2b4ead160 100644 --- a/dataretrieval/ngwmn.py +++ b/dataretrieval/ngwmn.py @@ -1,13 +1,12 @@ -"""National Ground-Water Monitoring Network (NGWMN) getters. +"""Retrieve data from the National Ground-Water Monitoring Network (NGWMN). The NGWMN exposes its data through a dedicated OGC API (``https://api.waterdata.usgs.gov/ngwmn/ogcapi``) with five collections: ``sites``, ``waterLevelObs``, ``lithologyObs``, ``constructionObs``, and ``providers``. Each getter below delegates to the shared OGC facade -(:func:`~dataretrieval.ogc.get_ogc_data`) with -``base_url=NGWMN_OGC_API_URL``, so multi-value chunking, pagination, -retry/resume, and result shaping all behave exactly as they do for the main -Water Data getters. +(:func:`~dataretrieval.ogc.get_ogc_data`) with ``base_url=NGWMN_OGC_API_URL``. +Multi-value chunking, pagination, retry/resume, and result shaping therefore +behave exactly as they do for the main Water Data getters. Unlike the main Water Data collections, NGWMN aggregates monitoring locations from many agencies, so ``monitoring_location_id`` values use other agency diff --git a/dataretrieval/nldi.py b/dataretrieval/nldi.py index 04a0b4774..e6a6c5910 100644 --- a/dataretrieval/nldi.py +++ b/dataretrieval/nldi.py @@ -1,3 +1,13 @@ +"""Retrieve hydrologic network features from the Network Linked Data Index (NLDI). + +The getters below navigate the hydrologic network from an origin -- a feature +source and id, a ``comid``, or a lat/long point -- and return flowlines, basins, +or registered features as a ``geopandas.GeoDataFrame``, or as raw JSON when +``as_json=True``. This module requires geopandas. + +See https://api.water.usgs.gov/nldi/linked-data for the API reference. +""" + from __future__ import annotations from json import JSONDecodeError @@ -48,7 +58,7 @@ def _features_to_gdf(feature_collection: dict[str, Any]) -> gpd.GeoDataFrame: NLDI can legitimately return no features (e.g. a feature with nothing upstream), and :func:`_query_nldi` returns ``{}`` when a 200 response - carries no JSON body. ``GeoDataFrame.from_features`` raises on those + carries no JSON body. ``GeoDataFrame.from_features`` raises on both cases (there's no geometry column to attach the CRS to), so return an empty GeoDataFrame with the correct CRS instead of crashing. """ @@ -68,8 +78,10 @@ def get_flowlines( trim_start: bool = False, as_json: bool = False, ) -> gpd.GeoDataFrame | dict[str, Any]: - """Gets the flowlines for the specified navigation either by comid or feature - source in WGS84 lat/long coordinates as GeoDataFrame containing a polyline geometry. + """Get the flowlines for a navigation, either by comid or by feature source. + + Flowlines are returned in WGS84 lat/long coordinates as a GeoDataFrame + containing a polyline geometry. Parameters ---------- @@ -133,8 +145,10 @@ def get_basin( split_catchment: bool = False, as_json: bool = False, ) -> gpd.GeoDataFrame | dict[str, Any]: - """Gets the aggregated basin for the specified feature in WGS84 lat/lon - as GeoDataFrame or as JSON containing a polygon geometry. + """Get the aggregated basin for the specified feature. + + The basin is returned in WGS84 lat/lon as a GeoDataFrame or as JSON, + containing a polygon geometry. Parameters ---------- @@ -190,9 +204,10 @@ def get_features( stop_comid: int | None = None, as_json: bool = False, ) -> gpd.GeoDataFrame | dict[str, Any]: - """Gets all features found along the specified navigation either by - comid or feature source as points in WGS84 lat/long coordinates - a GeoDataFrame - containing a point geometry. + """Get all features along a navigation, either by comid or by feature source. + + Features are returned as points in WGS84 lat/long coordinates - a + GeoDataFrame containing a point geometry. Parameters ---------- @@ -295,8 +310,10 @@ def get_features( # TODO: This function can cause a timeout error for some data sources # - maybe we shouldn't provide this function? def get_features_by_data_source(data_source: str) -> gpd.GeoDataFrame: - """Gets all features found for the specified data source as - points in WGS84 lat/long coordinates as GeoDataFrame containing a point geometry. + """Get all features for the specified data source. + + Features are returned as points in WGS84 lat/long coordinates as a + GeoDataFrame containing a point geometry. Parameters ---------- @@ -336,8 +353,7 @@ def search( long: float | None = None, distance: int = 50, ) -> dict[str, Any]: - """Searches for the specified feature in NLDI and returns the results - as a dictionary. + """Search NLDI for the specified feature and return the results as a dict. Parameters ---------- diff --git a/dataretrieval/nwis.py b/dataretrieval/nwis.py index 25e9cf8f4..479743aa1 100644 --- a/dataretrieval/nwis.py +++ b/dataretrieval/nwis.py @@ -129,27 +129,27 @@ def _parse_json_or_raise(response: httpx.Response) -> pd.DataFrame: def format_response( df: pd.DataFrame, service: str | None = None, **kwargs: Any ) -> pd.DataFrame: - """Setup index for response from query. + """Set up the index for a query response. - This function formats the response from the NWIS web services, in - particular it sets the index of the data frame. This function tries to - convert the NWIS response into pandas datetime values localized to UTC, - and if possible, uses these timestamps to define the data frame index. + Formats the response from the NWIS web services; in particular, it sets + the index of the data frame. It converts the NWIS response into pandas + datetime values localized to UTC and, where possible, uses those + timestamps to define the data frame index. Parameters ---------- df: ``pandas.DataFrame`` - The data frame to format + The data frame to format. service: string, optional, default is None - The NWIS service that was queried, important because the 'peaks' - service returns a different format than the other services. + The NWIS service that was queried. This matters because the 'peaks' + service returns a different format from the other services. **kwargs: optional - Additional keyword arguments, e.g., 'multi_index' + Additional keyword arguments, e.g. 'multi_index'. Returns ------- df: ``pandas.DataFrame`` - The formatted data frame + The formatted data frame. """ mi = kwargs.pop("multi_index", True) @@ -181,19 +181,17 @@ def format_response( def preformat_peaks_response(df: pd.DataFrame) -> pd.DataFrame: - """Datetime formatting for the 'peaks' service response. - - Function to format the datetime column of the 'peaks' service response. + """Format the datetime column of the 'peaks' service response. Parameters ---------- df: ``pandas.DataFrame`` - The data frame to format + The data frame to format. Returns ------- df: ``pandas.DataFrame`` - The formatted data frame + The formatted data frame. """ df["datetime"] = pd.to_datetime(df.pop("peak_dt"), errors="coerce") @@ -225,35 +223,33 @@ def get_discharge_peaks( ssl_check: bool = True, **kwargs: Any, ) -> tuple[pd.DataFrame, NWIS_Metadata]: - """ - Get discharge peaks from the waterdata service. + """Get discharge peaks from the waterdata service. Parameters ---------- sites: string or list of strings, optional, default is None - If the waterdata parameter site_no is supplied, it will overwrite the - sites parameter + USGS site number (or list of site numbers). If the waterdata parameter + site_no is supplied, it overwrites the sites parameter. start: string, optional, default is None - If the waterdata parameter begin_date is supplied, it will overwrite - the start parameter (YYYY-MM-DD) + Starting date of record (YYYY-MM-DD). If the waterdata parameter + begin_date is supplied, it overwrites the start parameter. end: string, optional, default is None - If the waterdata parameter end_date is supplied, it will overwrite - the end parameter (YYYY-MM-DD) + Ending date of record (YYYY-MM-DD). If the waterdata parameter + end_date is supplied, it overwrites the end parameter. multi_index: bool, optional - If False, a dataframe with a single-level index (datetime) is returned, - default is True + If False, return a dataframe with a single-level index (datetime). + Default is True. ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Returns ------- df: ``pandas.DataFrame`` - Time series data from the NWIS JSON + Time series data from the NWIS JSON. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. Examples -------- @@ -304,8 +300,7 @@ def get_gwlevels(**kwargs: Any) -> NoReturn: def get_stats( sites: list[str] | str | None = None, ssl_check: bool = True, **kwargs: Any ) -> tuple[pd.DataFrame, NWIS_Metadata]: - """ - Queries water services statistics information. + """Query the water services statistics service. For more information about the water services statistics service, visit https://waterservices.usgs.gov/docs/statistics/statistics-details/ @@ -313,26 +308,25 @@ def get_stats( Parameters ---------- sites: string or list of strings, optional, default is None - USGS site number (or list of site numbers) + USGS site number (or list of site numbers). ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Keyword Arguments - --------------------- + ----------------- statReportType: string - daily (default), monthly, or annual + daily (default), monthly, or annual. statTypeCd: string - all, mean, max, min, median + all, mean, max, min, median. Returns ------- df: ``pandas.DataFrame`` - Statistics data from the statistics service + Statistics data from the statistics service. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. .. todo:: @@ -366,23 +360,21 @@ def get_stats( def query_waterdata( service: str, ssl_check: bool = True, **kwargs: Any ) -> httpx.Response: - """ - Queries waterdata. + """Query the waterdata service. Parameters ---------- service: string Name of the service to query: 'peaks' or 'ratings'. ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Returns ------- request: ``httpx.Response`` - The response object from the API request to the web service + The response object from the API request to the web service. """ major_params = ["site_no", "stateCd"] bbox_params = [ @@ -412,8 +404,7 @@ def query_waterdata( def query_waterservices( service: str, ssl_check: bool = True, **kwargs: Any ) -> httpx.Response: - """ - Queries waterservices.usgs.gov + """Query waterservices.usgs.gov. For more documentation see https://waterservices.usgs.gov/docs/ @@ -426,31 +417,29 @@ def query_waterservices( service: string Name of the service to query: 'dv', 'iv', 'site', or 'stat'. ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Keyword Arguments - ---------------- + ----------------- bBox: string Bounding box of decimal latitude and longitude values, given as west longitude, south latitude, east longitude, north latitude, - separated by commas + separated by commas. startDT: string - Start date (e.g., '2017-12-31') + Start date (e.g. '2017-12-31'). endDT: string - End date (e.g., '2018-01-01') + End date (e.g. '2018-01-01'). modifiedSince: string - Used to return only sites where attributes or period of record data - have changed during the request period. String expected to be formatted - in ISO-8601 duration format (e.g., 'P1D' for one day, - 'P1Y' for one year) + Period during which site attributes or period-of-record data must have + changed for a site to be returned. Expected to be a string in ISO-8601 + duration format (e.g. 'P1D' for one day, 'P1Y' for one year). Returns ------- request: ``httpx.Response`` - The response object from the API request to the web service + The response object from the API request to the web service. """ if not any( @@ -480,8 +469,7 @@ def get_dv( ssl_check: bool = True, **kwargs: Any, ) -> tuple[pd.DataFrame, NWIS_Metadata]: - """ - Get daily values data from NWIS and return it as a ``pandas.DataFrame``. + """Get daily values data from NWIS and return it as a ``pandas.DataFrame``. .. note:: @@ -491,28 +479,27 @@ def get_dv( Parameters ---------- sites: string or list of strings, optional, default is None - USGS site number (or list of site numbers) + USGS site number (or list of site numbers). start: string, optional, default is None - If the waterdata parameter startDT is supplied, it will overwrite the - start parameter (YYYY-MM-DD) + Starting date of record (YYYY-MM-DD). If the waterdata parameter + startDT is supplied, it overwrites the start parameter. end: string, optional, default is None - If the waterdata parameter endDT is supplied, it will overwrite the - end parameter (YYYY-MM-DD) + Ending date of record (YYYY-MM-DD). If the waterdata parameter endDT + is supplied, it overwrites the end parameter. multi_index: bool, optional - If True, return a multi-index dataframe, if False, return a - single-index dataframe, default is True + If True, return a multi-index dataframe; if False, return a + single-index dataframe. Default is True. ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Returns ------- df: ``pandas.DataFrame`` - Time series data from the NWIS JSON + Time series data from the NWIS JSON. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. Examples -------- @@ -547,8 +534,7 @@ def get_dv( def get_info( ssl_check: bool = True, **kwargs: Any ) -> tuple[pd.DataFrame, NWIS_Metadata]: - """ - Get site description information from NWIS. + """Get site description information from NWIS. **Note:** *Must specify one major parameter.* @@ -558,13 +544,12 @@ def get_info( Parameters ---------- ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Keyword Arguments - ---------------- + ----------------- sites: string or list of strings A list of site numbers. Sites may be prefixed with an optional agency code followed by a colon. @@ -578,7 +563,7 @@ def get_info( bBox: string or list of strings A contiguous range of decimal latitude and longitude, starting with the west longitude, then the south latitude, then the east longitude, and - then the north latitude with each value separated by a comma. The + then the north latitude, with each value separated by a comma. The product of the range of latitude and longitude cannot exceed 25 degrees. Whole or decimal degrees must be specified, up to six digits of precision. Minutes and seconds are not allowed. @@ -611,9 +596,9 @@ def get_info( siteOutput: string ('basic' or 'expanded') Indicates the richness of metadata you want for site attributes. Note that for visually oriented formats like Google Map format, this - argument has no meaning. Note: for performance reasons, - siteOutput=expanded cannot be used if seriesCatalogOutput=true or with - any values for outputDataTypeCd. + argument has no meaning. For performance reasons, siteOutput=expanded + cannot be used if seriesCatalogOutput=true or with any values for + outputDataTypeCd. seriesCatalogOutput: bool A switch that provides detailed period of record information for certain output formats. The period of record indicates date ranges for @@ -623,9 +608,9 @@ def get_info( Returns ------- df: ``pandas.DataFrame`` - Site data from the NWIS web service + Site data from the NWIS web service. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. Examples -------- @@ -680,29 +665,28 @@ def get_iv( Parameters ---------- sites: string or list of strings, optional, default is None - If the waterdata parameter site_no is supplied, it will overwrite the - sites parameter + USGS site number (or list of site numbers). If the waterdata parameter + site_no is supplied, it overwrites the sites parameter. start: string, optional, default is None - If the waterdata parameter startDT is supplied, it will overwrite the - start parameter (YYYY-MM-DD) + Starting date of record (YYYY-MM-DD). If the waterdata parameter + startDT is supplied, it overwrites the start parameter. end: string, optional, default is None - If the waterdata parameter endDT is supplied, it will overwrite the - end parameter (YYYY-MM-DD) + Ending date of record (YYYY-MM-DD). If the waterdata parameter endDT + is supplied, it overwrites the end parameter. multi_index: bool, optional - If False, a dataframe with a single-level index (datetime) is returned, - default is True + If False, return a dataframe with a single-level index (datetime). + Default is True. ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Returns ------- df: ``pandas.DataFrame`` - Time series data from the NWIS JSON + Time series data from the NWIS JSON. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. Examples -------- @@ -761,32 +745,29 @@ def get_ratings( ssl_check: bool = True, **kwargs: Any, ) -> tuple[pd.DataFrame, NWIS_Metadata]: - """ - Rating table for an active USGS streamgage retrieval. + """Get the rating table for an active USGS streamgage. - Reads current rating table for an active USGS streamgage from NWISweb. + Reads the current rating table for an active USGS streamgage from NWISweb. Data is retrieved from https://waterdata.usgs.gov/nwis. Parameters ---------- site: string, optional, default is None - USGS site number. This is usually an 8 digit number as a string. - If the nwis parameter site_no is supplied, it will overwrite the site - parameter + USGS site number, usually an 8 digit number as a string. If the nwis + parameter site_no is supplied, it overwrites the site parameter. file_type: string, default is "base" - can be "base", "corr", or "exsa" + One of "base", "corr", or "exsa". ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Returns ------- df: ``pandas.DataFrame`` - Formatted requested data + Formatted requested data. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. Examples -------- @@ -816,23 +797,21 @@ def get_ratings( def what_sites( ssl_check: bool = True, **kwargs: Any ) -> tuple[pd.DataFrame, NWIS_Metadata]: - """ - Search NWIS for sites within a region with specific data. + """Search NWIS for sites within a region with specific data. Parameters ---------- ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - Accepts the same parameters as :obj:`dataretrieval.nwis.get_info` + Accepts the same parameters as :obj:`dataretrieval.nwis.get_info`. Returns ------- df: ``pandas.DataFrame`` - Formatted requested data + Formatted requested data. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A custom metadata object + A custom metadata object. Examples -------- @@ -867,8 +846,7 @@ def get_record( ssl_check: bool = True, **kwargs: Any, ) -> pd.DataFrame: - """ - Get data from NWIS and return it as a ``pandas.DataFrame``. + """Get data from NWIS and return it as a ``pandas.DataFrame``. .. note:: @@ -878,21 +856,21 @@ def get_record( Parameters ---------- sites: string or list of strings, optional, default is None - List or comma delimited string of sites. + List of sites, or a comma-delimited string of sites. start: string, optional, default is None - Starting date of record (YYYY-MM-DD) + Starting date of record (YYYY-MM-DD). end: string, optional, default is None - Ending date of record. (YYYY-MM-DD) + Ending date of record (YYYY-MM-DD). multi_index: bool, optional - If False, a dataframe with a single-level index (datetime) is returned, - default is True + If False, return a dataframe with a single-level index (datetime). + Default is True. wide_format : bool, optional - If True, return data in wide format with multiple samples per row and - one row per time, default is True + If True, return data in wide format, with multiple samples per row and + one row per time. Default is True. datetime_index : bool, optional - If True, create a datetime index. Default is True + If True, create a datetime index. Default is True. state: string, optional, default is None - full name, abbreviation or id + State full name, abbreviation, or id. service: string, default is 'iv' - 'iv' : instantaneous data - 'dv' : daily mean data @@ -906,14 +884,13 @@ def get_record( - 'ratings': get rating table - 'stat': get statistics ssl_check: bool, optional - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. **kwargs: optional - If supplied, will be used as query parameters + Additional query parameters, if supplied. Returns ------- - ``pandas.DataFrame`` containing requested data + ``pandas.DataFrame`` containing the requested data. Examples -------- @@ -1018,18 +995,17 @@ def get_record( def _read_json(json: dict[str, Any]) -> pd.DataFrame: - """ - Reads a NWIS Water Services formatted JSON into a ``pandas.DataFrame``. + """Read a NWIS Water Services formatted JSON into a ``pandas.DataFrame``. Parameters ---------- json: dict - A JSON dictionary response to be parsed into a ``pandas.DataFrame`` + A JSON dictionary response to be parsed into a ``pandas.DataFrame``. Returns ------- df: ``pandas.DataFrame`` - Time series data from the NWIS JSON + Time series data from the NWIS JSON. """ all_site_dfs = [] @@ -1136,13 +1112,13 @@ class NWIS_Metadata(BaseMetadata): Attributes ---------- url : str - Response url + Response url. query_time: datetime.timedelta - Response elapsed time + Response elapsed time. header: httpx.Headers - Response headers + Response headers. comments: str | None - Metadata comments, if any + Metadata comments, if any. Notes ----- @@ -1152,15 +1128,14 @@ class NWIS_Metadata(BaseMetadata): """ def __init__(self, response: httpx.Response, **parameters: Any) -> None: - """Generates a standard set of metadata informed by the response with specific - metadata for NWIS data. + """Generate the standard metadata set, plus NWIS-specific metadata. Parameters ---------- response: Response - Response object from httpx module + Response object from the ``httpx`` module. parameters: unpacked dictionary - Unpacked dictionary of the parameters supplied in the request + Unpacked dictionary of the parameters supplied in the request. """ super().__init__(response) @@ -1185,9 +1160,9 @@ def site_info(self) -> tuple[pd.DataFrame, BaseMetadata] | None: Returns ------- df: ``pandas.DataFrame`` - Formatted requested data from calling `nwis.what_sites` + Formatted requested data from calling `nwis.what_sites`. md: :obj:`dataretrieval.nwis.NWIS_Metadata` - A NWIS_Metadata object + A NWIS_Metadata object. """ if "site_no" in self._parameters: return what_sites(sites=self._parameters["site_no"]) diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index f15f226b5..08a6741bf 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -9,11 +9,11 @@ cartesian product of chunks. Requests that already fit get a trivial single-step plan — ``ChunkedCall`` has one code path either way. -Parallel chunks: the planner is conservative by default — it splits only as far as -the byte limit forces. A caller who knows their result is large can opt into a -finer split via the ``parallel_chunks(n)`` context manager, which fans the query -out into ``n`` parallel sub-requests; ``n`` drives :meth:`ChunkPlan._refine`. See -``parallel_chunks`` for the why and the when. +Parallel chunks: the planner is conservative by default — it splits only as +far as the byte limit forces. A caller who knows their result is large can opt +into a finer split via the ``parallel_chunks(n)`` context manager, which fans +the query out into ``n`` parallel sub-requests. ``n`` drives +:meth:`ChunkPlan._refine`; see ``parallel_chunks`` for the why and the when. This module owns the *execution* half — the event loop and bounded concurrency that drive a plan to completion (``ChunkedCall``) plus the @@ -28,7 +28,7 @@ Concurrency: ``multi_value_chunked`` fans every pending sub-request out under one ``asyncio.gather`` sharing a single ``httpx.AsyncClient``. An ``asyncio.Semaphore`` — not the client's connection pool, which is -merely sized to match — caps the sub-requests in flight at ``N``; see +merely sized to match — caps the sub-requests in flight at ``N``. See :meth:`ChunkedCall._run` for why the gate must be the semaphore rather than the pool. ``API_USGS_CONCURRENT`` resolves ``N``: an integer N > 1 allows N sub-requests in flight; ``1`` forces sequential dispatch (one @@ -159,7 +159,7 @@ def get_active_client() -> httpx.AsyncClient | None: Return the chunker's currently-published client, or ``None``. Used by the paginated-loop helpers (e.g. - :func:`dataretrieval.ogc.engine._client_for`) to reuse the + :func:`dataretrieval.transport.pagination._client_for`) to reuse the per-call connection pool. Returns @@ -187,26 +187,26 @@ def parallel_chunks(n: int) -> Iterator[None]: By default the Water Data / NGWMN getters chunk a request only as much as the server's ~8 KB URL-byte limit forces — the fewest sub-requests that - fit. That is the safe default, but it can be *needlessly* conservative: - because every sub-request paginates, splitting a large result further costs + fit. That is the safe default, but it can be *needlessly* conservative. + Because every sub-request paginates, splitting a large result further costs little or no extra quota *as long as each sub-request still spans many pages* — rows-per-chunk far exceeding the page size (ten states pulled as - one request then page nearly as many times as ten per-state requests - would). When a split leaves each sub-request only a page or two, its partial - final page is extra, so finer chunks do add some requests. This context - manager lets a caller who *knows* their pull is large ask for that finer - split — trading roughly the same pages for more, smaller sub-requests, which - gives smoother progress, more even concurrency, and a smaller unit of + one request page nearly as many times as ten per-state requests would). + When a split leaves each sub-request only a page or two, its partial final + page is extra, so finer chunks do add some requests. This context manager + lets a caller who *knows* their pull is large ask for that finer split. The + trade is roughly the same pages for more, smaller sub-requests, which gives + smoother progress, more even concurrency, and a smaller unit of retry/resume. - Because the library can't tell in advance whether a query is large (ten - states over a short window might fit in a single page, where extra chunks - would only burn quota), this is a *deliberate* per-call knob rather than an - automatic behavior or a process-wide environment variable — scoping it to a - ``with`` block keeps an aggressive setting from leaking into unrelated calls - and accidentally spending quota. Outside any block the getters use the - conservative default. Only the OGC getters (Water Data, NGWMN) read this; - wrapping a legacy NWIS call in the block is a harmless no-op. + This is a *deliberate* per-call knob rather than an automatic behavior or a + process-wide environment variable, because the library can't tell in + advance whether a query is large (ten states over a short window might fit + in a single page, where extra chunks would only burn quota). Scoping it to + a ``with`` block keeps an aggressive setting from leaking into unrelated + calls and accidentally spending quota. Outside any block the getters use + the conservative default. Only the OGC getters (Water Data, NGWMN) read + this; wrapping a legacy NWIS call in the block is a harmless no-op. Parameters ---------- @@ -217,17 +217,17 @@ def parallel_chunks(n: int) -> Iterator[None]: argument combined, not per argument), so several multi-value arguments cannot multiply past it. The cap is a ceiling, never exceeded: the actual count is bounded below by what the ~8 KB URL limit already - forces and above by ``n``, so an ``n`` larger than the input allows + forces and above by ``n``. So an ``n`` larger than the input allows simply yields one sub-request per value, and with several multi-value arguments the total may land somewhat below ``n`` because splits are - whole (the plan can't always divide evenly onto ``n``); ``n=1`` asks + whole (the plan can't always divide evenly onto ``n``). ``n=1`` asks for no extra fan-out. Each sub-request fetches at least one page, so it costs at least one request against your hourly rate limit — a larger ``n`` spends more - quota. And because how many sub-requests run *at once* is capped - separately by ``API_USGS_CONCURRENT`` (default 32), an ``n`` beyond that - adds quota without adding parallelism; the useful range is roughly ``2`` + quota. How many sub-requests run *at once* is capped separately by + ``API_USGS_CONCURRENT`` (default 32), so an ``n`` beyond that adds + quota without adding parallelism; the useful range is roughly ``2`` up to ``API_USGS_CONCURRENT``. Yields @@ -251,15 +251,15 @@ def parallel_chunks(n: int) -> Iterator[None]: independently, then the combined result is sorted and truncated to ``max_rows``. So a call with ``max_rows`` set returns a *different* (though still valid and deterministically sorted) row set inside a - ``parallel_chunks`` block than without one — the cap is drawn from the + ``parallel_chunks`` block than without one. The cap is drawn from the union of the sub-requests, not a single stream. Don't pair a tight ``max_rows`` preview with ``parallel_chunks`` if you need exactly the rows the un-fanned call would return. - Resumability: a single request either fully succeeds or fully fails, but a fanned-out call can fail partway (e.g. a mid-call rate-limit) and - raise a resumable :class:`~dataretrieval.exceptions.ChunkInterrupted` - (or ``QuotaExhausted``) carrying the completed sub-requests, which you - finish with ``exc.call.resume()``. + raise a resumable :class:`~dataretrieval.ogc.interruptions.ChunkInterrupted` + (or ``QuotaExhausted``) carrying the completed sub-requests. Finish the + call with ``exc.call.resume()``. - Cross-sub-request de-duplication keys on the feature ``id``; features with no ``id`` can't be deduped, so overlapping filter clauses split across chunks may yield duplicate rows. @@ -314,15 +314,15 @@ class ChunkedCall: Holds the in-flight state (per-sub-request frames and responses) and the async fetcher. A single :meth:`resume` entry point drives the call from wherever it is to completion — used both for the - first invocation (from :meth:`ChunkPlan.execute`) and for subsequent + first invocation (from :func:`multi_value_chunked`) and for subsequent retries after a :class:`ChunkInterrupted`. :meth:`_run` gathers every pending sub-request over one shared :class:`httpx.AsyncClient`, applies the failure-precedence rules, and - combines; :meth:`resume` drives it through an ``anyio`` blocking - portal so it works whether or not the caller is already inside an - event loop. Concurrency is bounded by a per-run ``asyncio.Semaphore`` - (see :meth:`_run`), so sequential dispatch + combines. :meth:`resume` drives :meth:`_run` through an ``anyio`` + blocking portal, so it works whether or not the caller is already + inside an event loop. Concurrency is bounded by a per-run + ``asyncio.Semaphore`` (see :meth:`_run`), so sequential dispatch (``API_USGS_CONCURRENT=1``) is just a degenerate gather. A ``ChunkedCall`` is created internally when a :class:`ChunkPlan` @@ -377,12 +377,12 @@ def __init__( self.finalize = finalize # Snapshot the ambient context at construction time — i.e. inside the # caller's ``with`` blocks (base URL, dialect, row cap, progress - # reporter). :meth:`resume` runs every drive inside this snapshot, so - # a *later* ``exc.call.resume()`` — which fires after those ``with`` - # blocks have exited and reset their ContextVars — still rebuilds - # sub-requests against the original API's base URL/dialect rather than - # the process defaults. ``build_request`` reads those ContextVars when - # it reconstructs each sub-request, so the snapshot must outlive them. + # reporter). :meth:`resume` runs every drive inside this snapshot. So a + # *later* ``exc.call.resume()`` still rebuilds sub-requests against the + # original API's base URL/dialect rather than the process defaults, even + # though it fires after those ``with`` blocks have exited and reset + # their ContextVars. ``build_request`` reads those ContextVars when it + # reconstructs each sub-request, so the snapshot must outlive them. self._ctx = copy_context() # Completed (frame, response) pairs keyed by sub-args index; sparse # (gathered sub-requests complete out of order — see class docstring). @@ -392,10 +392,10 @@ def __init__( def wrap_failure(self, exc: BaseException) -> ChunkInterrupted | None: """ - Build the matching :class:`ChunkInterrupted` carrying this - call when ``exc`` is a recognized transient transport failure; - return ``None`` for unrecognized failures so the caller can - re-raise. Encapsulates the + Wrap ``exc`` as the matching :class:`ChunkInterrupted` carrying this call. + + Returns ``None`` when ``exc`` is not a recognized transient transport + failure, so the caller can re-raise it. Encapsulates the ``classify → instantiate-with-call-state`` recipe so :class:`ChunkedCall`'s private fields stay private. @@ -428,16 +428,15 @@ def completed_chunks(self) -> int: return len(self._chunks) def _combine_raw(self) -> tuple[pd.DataFrame, httpx.Response]: - """Assemble the raw ``(frame, response)`` from completed sub-requests, - before :attr:`finalize` runs. + """Assemble the raw ``(frame, response)`` from completed sub-requests. - Frames concatenate in sub-args *index* order (``sorted`` keys — - deterministic, independent of parallel completion order). The - aggregated response takes its headers from the response with the - lowest reported ``x-ratelimit-remaining`` value. If no response - reports that header, it falls back to the last completed response; - ``self._chunks`` preserves completion order because the ``track`` - closure in :meth:`_run` is its only writer. + Runs before :attr:`finalize`. Frames concatenate in sub-args *index* + order (``sorted`` keys — deterministic, independent of parallel + completion order). The aggregated response takes its headers from the + response with the lowest reported ``x-ratelimit-remaining`` value. If no + response reports that header, the aggregate falls back to the last + completed response; ``self._chunks`` preserves completion order because + the ``track`` closure in :meth:`_run` is its only writer. Returns ------- @@ -538,9 +537,8 @@ def resume(self) -> tuple[pd.DataFrame, Any]: The finalized aggregate — a raw :class:`httpx.Response` (canonical URL, headers from the response with the lowest reported remaining quota, and summed response elapsed durations) by default, - or whatever - :attr:`finalize` produces (e.g. ``BaseMetadata`` for the OGC - getters). + or whatever :attr:`finalize` produces (e.g. ``BaseMetadata`` for + the OGC getters). Raises ------ @@ -554,10 +552,10 @@ def resume(self) -> tuple[pd.DataFrame, Any]: # Drive inside the snapshot taken at construction (see ``__init__``). # ``start_blocking_portal`` copies the *calling* context into its # worker thread, and running here means that calling context is the - # snapshot — so the base URL / dialect / row cap / progress reporter - # active when the call was created reach the rebuilt sub-requests, - # even when this is a resume fired long after the original ``with`` - # blocks exited. + # snapshot. So the base URL / dialect / row cap / progress reporter + # active when the call was created reach the rebuilt sub-requests, even + # when this is a resume fired long after the original ``with`` blocks + # exited. return self._ctx.run(self._resume_in_context) def _resume_in_context(self) -> tuple[pd.DataFrame, Any]: @@ -573,16 +571,16 @@ def _resume_in_context(self) -> tuple[pd.DataFrame, Any]: async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: """ - Gather every pending sub-request over one shared - :class:`httpx.AsyncClient` and return the combined, finalized result. - - Pending sub-requests (:meth:`_pending`) fan out under - ``asyncio.gather`` with ``return_exceptions=True`` so completed - sub-requests survive a sibling's transient failure. On a - recognized transient (:class:`RateLimited`, :class:`ServiceUnavailable`, - or a bare ``httpx.HTTPError`` / ``httpx.InvalidURL``) a + Gather every pending sub-request; return the combined, finalized result. + + Pending sub-requests (:meth:`_pending`) fan out over one shared + :class:`httpx.AsyncClient` under ``asyncio.gather``, with + ``return_exceptions=True`` so completed sub-requests survive a + sibling's transient failure. On a recognized transient + (:class:`RateLimited`, :class:`ServiceUnavailable`, or a bare + ``httpx.HTTPError`` / ``httpx.InvalidURL``), a :class:`ChunkInterrupted` subclass is raised carrying ``self`` on - ``.call``; ``exc.call.resume()`` then re-issues only the unfinished + ``.call``. ``exc.call.resume()`` then re-issues only the unfinished indices through this same runner. The gather dispatches *every* pending sub-request at once, but an @@ -621,8 +619,8 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: The finalized aggregate — a raw :class:`httpx.Response` (canonical URL, headers from the response with the lowest reported remaining quota, and summed response elapsed durations) by default, - or whatever - :attr:`finalize` produces (e.g. ``BaseMetadata`` for OGC getters). + or whatever :attr:`finalize` produces (e.g. ``BaseMetadata`` for + OGC getters). Raises ------ @@ -715,7 +713,7 @@ def multi_value_chunked( ``async def fetch(args) -> (df, response)``, and drives it to completion via :meth:`ChunkedCall.resume`. The plan splits multi-value list params and the cql-text filter so each sub-request URL fits the - byte limit; an already-fitting request is a one-step plan, unless an + byte limit. An already-fitting request is a one-step plan, unless an active :func:`parallel_chunks` block asks the plan to fan out more finely. See the module docstring for the concurrency model. diff --git a/dataretrieval/ogc/dates.py b/dataretrieval/ogc/dates.py index 25af4860f..248e667c1 100644 --- a/dataretrieval/ogc/dates.py +++ b/dataretrieval/ogc/dates.py @@ -82,7 +82,7 @@ def _format_api_dates( """ Formats date or datetime input(s) for use with an API. - Handles single values or ranges, and converting to ISO 8601 or date-only + Handles single values or ranges, converting to ISO 8601 or date-only formats as needed. Parameters @@ -101,10 +101,10 @@ def _format_api_dates( ------- Union[str, None] - If input is a single value, returns the formatted date/datetime string - or None if parsing fails. + or None if parsing fails. - If input is a list of two values, returns a date/datetime range string - separated by "/" (e.g., "YYYY-MM-DD/YYYY-MM-DD" or - "YYYY-MM-DDTHH:MM:SSZ/YYYY-MM-DDTHH:MM:SSZ"). + separated by "/" (e.g., "YYYY-MM-DD/YYYY-MM-DD" or + "YYYY-MM-DDTHH:MM:SSZ/YYYY-MM-DDTHH:MM:SSZ"). - Returns None if input is empty, all NA, or cannot be parsed. Raises @@ -115,15 +115,15 @@ def _format_api_dates( Notes ----- - A single blank/NA value returns None. In a two-value range, a blank/NA - endpoint is rendered as ``".."`` to denote an open bound (e.g. - ``"2024-01-01/.."``); the range is only None when *every* element is - blank/NA or any non-NA element fails to parse. + endpoint is rendered as ``".."`` to denote an open bound (e.g. + ``"2024-01-01/.."``); the range is only None when *every* element is + blank/NA or any non-NA element fails to parse. - Supports ISO 8601 durations such as "P7D" and "PT36H" and pre-formatted - intervals containing ``"/"``; both are passed through unchanged. + intervals containing ``"/"``; both are passed through unchanged. - Converts datetimes to UTC and formats as ISO 8601 with 'Z' suffix when - `date` is False. Inputs with an explicit offset (``Z`` or ``+HH:MM``) are - converted from that offset to UTC; naive inputs are interpreted in the - local time zone for backwards compatibility. + `date` is False. Inputs with an explicit offset (``Z`` or ``+HH:MM``) are + converted from that offset to UTC; naive inputs are interpreted in the + local time zone for backwards compatibility. """ if datetime_input is None: return None @@ -156,7 +156,7 @@ def _format_api_dates( if _DURATION_RE.match(single) or "/" in single: return single - # element invalidates the range. + # Format each element; any element that fails to parse invalidates the range. formatted: list[str] = [] for dt in datetime_input: one = _format_one(dt, date=date) diff --git a/dataretrieval/ogc/engine.py b/dataretrieval/ogc/engine.py index 975259ede..d438bf18a 100644 --- a/dataretrieval/ogc/engine.py +++ b/dataretrieval/ogc/engine.py @@ -4,8 +4,8 @@ strategies and the chunked fetch entry point :func:`get_ogc_data`. Generic pagination and sync dispatch live in :mod:`dataretrieval.transport`; request construction lives in :mod:`~dataretrieval.ogc.requests`. The surrounding -concerns live in sibling modules it composes, each with its own reason to -change: :mod:`~dataretrieval.ogc.dates` (time-parameter marshalling), +concerns live in sibling modules this one composes, each with its own reason +to change: :mod:`~dataretrieval.ogc.dates` (time-parameter marshalling), :mod:`~dataretrieval.ogc.errors` (HTTP error mapping), and :mod:`~dataretrieval.ogc.shaping` (GeoJSON features to DataFrame and result finalization). It is deliberately free of any Water-Data-specific constants @@ -85,8 +85,7 @@ def _next_req_url( resp: httpx.Response, *, body: dict[str, Any] | None = None ) -> str | None: """ - Extracts the URL for the next page of results from an HTTP response from a - water data endpoint. + Extracts the next-page URL from a water data endpoint's HTTP response. Parameters ---------- @@ -106,7 +105,7 @@ def _next_req_url( ----- - Returns None when the response carries no features. - Expects the response JSON to contain a "links" list with objects having - "rel" and "href" keys. + "rel" and "href" keys. - Checks for the "next" relation in the "links" to determine the next URL. """ if body is None: @@ -209,8 +208,7 @@ async def _walk_pages( client: httpx.AsyncClient | None = None, ) -> tuple[pd.DataFrame, httpx.Response]: """ - Iterate paginated OGC API responses asynchronously and aggregate - them into one DataFrame. + Iterate paginated OGC API responses and aggregate them into one DataFrame. Thin wrapper that hands off to :func:`_paginate` with OGC-specific strategies: pages are parsed via :func:`_get_resp_data` @@ -269,12 +267,11 @@ def get_ogc_data( dialect: OgcDialect | None = None, ) -> tuple[pd.DataFrame, BaseMetadata]: """ - Retrieves OGC (Open Geospatial Consortium) data from a specified - endpoint and returns it as a pandas DataFrame with metadata. + Retrieves OGC (Open Geospatial Consortium) data as a DataFrame with metadata. - This function prepares request arguments, constructs API requests, - handles pagination, processes the results, and formats output - according to the specified parameters. + Prepares request arguments, constructs API requests, handles pagination, + processes the results, and formats output according to the specified + parameters. Parameters ---------- @@ -364,8 +361,7 @@ def get_ogc_data( async def _fetch_once( args: dict[str, Any], ) -> tuple[pd.DataFrame, httpx.Response]: - """Send one prepared-args OGC request asynchronously; return the - frame + response. + """Send one prepared-args OGC request asynchronously; return (frame, response). ``@chunking.multi_value_chunked`` models every multi-value list parameter and the cql-text filter as a chunkable axis, greedy-halves @@ -373,8 +369,8 @@ async def _fetch_once( and iterates the cartesian product. With no chunkable inputs the decorator passes args through unchanged. The decorator gathers every sub-request over one shared :class:`httpx.AsyncClient` (concurrency - bounded by a semaphore, sized from ``API_USGS_CONCURRENT``) - and returns a *synchronous* wrapper, so ``get_ogc_data`` keeps calling + bounded by a semaphore, sized from ``API_USGS_CONCURRENT``). It also + returns a *synchronous* wrapper, so ``get_ogc_data`` keeps calling ``_fetch_once(args, finalize=...)`` synchronously. The return shape is ``(frame, response)``. """ diff --git a/dataretrieval/ogc/errors.py b/dataretrieval/ogc/errors.py index 90b541dee..5b5f10fa0 100644 --- a/dataretrieval/ogc/errors.py +++ b/dataretrieval/ogc/errors.py @@ -90,9 +90,9 @@ def _raise_for_non_200(resp: httpx.Response) -> None: transient types (:class:`~dataretrieval.exceptions.TransientError`) are distinguished so ``ChunkedCall`` can wrap them as a resumable :class:`~dataretrieval.ogc.interruptions.QuotaExhausted` / - :class:`~dataretrieval.ogc.interruptions.ServiceInterrupted`; a fatal - :class:`~dataretrieval.exceptions.HTTPError` (not a ``TransientError``) - the chunker won't resume. + :class:`~dataretrieval.ogc.interruptions.ServiceInterrupted`. The + chunker won't resume a fatal + :class:`~dataretrieval.exceptions.HTTPError` (not a ``TransientError``). """ status = resp.status_code if status < 400: diff --git a/dataretrieval/ogc/filters.py b/dataretrieval/ogc/filters.py index 8ec55bf06..fdd08615b 100644 --- a/dataretrieval/ogc/filters.py +++ b/dataretrieval/ogc/filters.py @@ -123,8 +123,8 @@ def _check_numeric_filter_pitfall(filter_expr: str) -> None: ``hydrologic_unit_code``, ``channel_flow``). Any unquoted numeric comparison — ``value >= 1000``, ``parameter_code = 60``, ``parameter_code IN (60, 61)``, ``value BETWEEN 5 AND 10`` — either gets - rejected with HTTP 500 or silently produces lexicographic results; - zero-padded codes are the worst case (``parameter_code = '60'`` matches + rejected with HTTP 500 or silently produces lexicographic results. + Zero-padded codes are the worst case (``parameter_code = '60'`` matches nothing because the real codes are ``'00060'``-shaped). Quoted literals (``value >= '1000'``) are not flagged — the caller has diff --git a/dataretrieval/ogc/interruptions.py b/dataretrieval/ogc/interruptions.py index 8cb5723c3..a402fdc05 100644 --- a/dataretrieval/ogc/interruptions.py +++ b/dataretrieval/ogc/interruptions.py @@ -2,7 +2,7 @@ When a transparently-chunked request fails mid-stream (a 429, a 5xx, or a bare transport error), the work already completed is preserved and the call -is resumable: the raised exception carries a ``.call`` handle whose +is resumable. The raised exception carries a ``.call`` handle whose ``resume()`` re-issues only the still-pending sub-requests. These exception types are that contract, re-exported at the top level (``from dataretrieval import ChunkInterrupted``). The execution machinery @@ -24,8 +24,7 @@ class ChunkInterrupted(DataRetrievalError): """ - Base class for mid-stream chunk failures whose completed work is - preserved and resumable. + Base class for mid-stream chunk failures whose completed work is resumable. A ``ChunkInterrupted`` subclass means: a sub-request failed, but ``ChunkedCall`` still owns whatever completed successfully before @@ -139,20 +138,19 @@ def __getstate__(self) -> dict[str, Any]: # interruption can't cross a process boundary with ``.call`` attached. # The degraded ``call=None`` form keeps the counts, retry hint, and the # snapshotted partial frame / response — plain instance attributes the - # base ``__getstate__`` already pickles; only ``.resume()`` is lost - # (cross-process resume was never possible anyway). + # base ``__getstate__`` already pickles. Only ``.resume()`` is lost, and + # cross-process resume was never possible anyway. return {**super().__getstate__(), "call": None} class QuotaExhausted(ChunkInterrupted): """ - A sub-request returned HTTP 429 — the per-key rate-limit window - is exhausted. Subclass of :class:`ChunkInterrupted`. + A sub-request returned HTTP 429 — the per-key rate-limit window is exhausted. - The completed sub-requests are preserved on ``.call``; once the - rate-limit window resets, ``.call.resume()`` re-issues only the - still-pending work. ``partial_frame`` holds what completed - before the 429. + Subclass of :class:`ChunkInterrupted`. The completed sub-requests are + preserved on ``.call``; once the rate-limit window resets, + ``.call.resume()`` re-issues only the still-pending work. + ``partial_frame`` holds what completed before the 429. """ _MESSAGE_TEMPLATE = ( @@ -165,12 +163,11 @@ class QuotaExhausted(ChunkInterrupted): class ServiceInterrupted(ChunkInterrupted): """ - A sub-request returned HTTP 5xx — the upstream service failed - transiently. Subclass of :class:`ChunkInterrupted`. + A sub-request returned HTTP 5xx — the upstream service failed transiently. - The completed sub-requests are preserved on ``.call``; once the - upstream recovers, ``.call.resume()`` resumes only the - still-pending work. + Subclass of :class:`ChunkInterrupted`. The completed sub-requests are + preserved on ``.call``; once the upstream recovers, ``.call.resume()`` + resumes only the still-pending work. """ _MESSAGE_TEMPLATE = ( diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index 76796df41..03dc90736 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -98,8 +98,7 @@ def _safe_request_bytes( url_limit: int, ) -> int: """ - Size a candidate sub-request, treating ``httpx.InvalidURL`` as - "still too large". + Size a candidate sub-request, treating ``httpx.InvalidURL`` as "too large". ``httpx.URL`` enforces a hard 64 KB cap per URL component (``MAX_URL_LENGTH``) and raises ``httpx.InvalidURL`` for anything @@ -133,10 +132,10 @@ def _safe_request_bytes( @dataclass(frozen=True) class _Axis: """ - A single chunkable axis of one user-level request — a list of - atomic units and the separator that joins them in the URL. + A single chunkable axis of one user-level request. - Both multi-value list parameters (``sites=[...]``, joiner ``","``) + An axis is a list of atomic units plus the separator that joins them in + the URL. Both multi-value list parameters (``sites=[...]``, joiner ``","``) and the cql-text ``filter`` (split on top-level ``OR``, joiner ``" OR "``) fit this shape, so a single greedy halving loop in ``ChunkPlan._plan`` handles both — no need for two separate @@ -162,8 +161,7 @@ class _Axis: def chunk_bytes(self, chunk: list[str]) -> int: """ - Return the URL-encoded byte count this chunk contributes when - substituted into the request. + Return the URL-encoded byte count this chunk contributes to the request. ``quote_plus`` is faithful to what the real URL builder produces, so values containing characters that expand under URL @@ -211,7 +209,7 @@ def _extract_axes(args: dict[str, Any]) -> list[_Axis]: axis. The cql-text filter (when chunkable and split into more than one top-level OR-clause) becomes one too. Anything in ``_NEVER_CHUNK`` is excluded except ``filter`` itself, which is - handled separately so its atoms are clauses not characters. + handled separately so its atoms are clauses, not characters. Parameters ---------- @@ -259,10 +257,9 @@ def _split_at(chunks: list[list[str]], idx: int) -> None: class ChunkPlan: """ - Strategy for issuing one user-level request as a sequence of - sub-requests whose URLs each fit ``url_limit``. + Strategy for issuing one user-level request as URL-fitting sub-requests. - Constructing a plan *is* planning: + Every sub-request URL fits ``url_limit``. Constructing a plan *is* planning: ``ChunkPlan(args, build_request, url_limit)`` extracts the chunkable axes, runs greedy halving on the biggest chunk across all axes, and stores the result. @@ -289,12 +286,12 @@ class ChunkPlan: conservative plan, fewest sub-requests — so a fitting request is a passthrough. A cap of ``2`` or more fans the plan out to up to ``max_chunks`` sub-requests overall (the cartesian product across axes, - never fewer than the byte budget already forces) — capped as a whole, - not per axis, so several multi-value axes can't multiply past the cap. - The plan never exceeds the cap and may land below it when no whole - split lands on it exactly. ``max_chunks`` is a sub-request count, so a - value below ``1`` (``0`` or negative) is a caller error and raises - ``ValueError``. Set from the + never fewer than the byte budget already forces). The cap applies to + the plan as a whole, not per axis, so several multi-value axes can't + multiply past it. The plan never exceeds the cap and may land below it + when no whole split lands on it exactly. ``max_chunks`` is a + sub-request count, so a value below ``1`` (``0`` or negative) is a + caller error and raises ``ValueError``. Set from the :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see :meth:`_refine`. @@ -359,8 +356,8 @@ def __init__( # Over budget. A filter the chunker doesn't manage — cql-json — is # passed through unchanged (chunking applies only to cql-text); the # server, not us, judges it. Otherwise this is an in-domain shape we - # would normally chunk but can't (a single large CQL ``IN`` clause - # with no top-level ``OR``, or one oversized value), so raise an + # would normally chunk but can't: a single large CQL ``IN`` clause + # with no top-level ``OR``, or one oversized value. Raise an # actionable error instead of shipping it for an opaque HTTP 414. filter_expr = args.get("filter") if filter_expr is not None and not _is_chunkable( @@ -377,10 +374,10 @@ def __init__( # Constructing the initial request can itself trip # ``httpx.InvalidURL`` (URL > 64 KB) — that's the canonical # "needs chunking" signal, so swallow it and proceed to plan. - # When the unchunked URL does build, preserve it as - # ``canonical_url`` so ``BaseMetadata.url`` echoes the user's - # original query verbatim; only fall back to a worst-case - # sub-request URL when the URL itself can't be constructed. + # When the unchunked URL does build, preserve it as ``canonical_url`` + # so ``BaseMetadata.url`` echoes the user's original query verbatim. + # Only fall back to a worst-case sub-request URL when the URL itself + # can't be constructed. try: initial_request = build_request(**args) except httpx.InvalidURL: @@ -426,11 +423,12 @@ def _plan( url_limit: int, ) -> None: """ - Greedy-halve the biggest chunk across all axes until the - worst-case sub-request URL fits ``url_limit``. Mutates - ``self.chunks`` in place; treats list axes and the filter axis - uniformly — each is just a list of atoms joined by its axis's - separator. + Greedy-halve the biggest chunk across axes until every URL fits. + + Halving continues until the worst-case sub-request URL fits + ``url_limit``, mutating ``self.chunks`` in place. List axes and the + filter axis are treated uniformly — each is just a list of atoms + joined by its axis's separator. Raises ------ @@ -465,17 +463,18 @@ def _plan( def _refine(self, max_chunks: int) -> None: """ - Fan the plan out more finely than the byte budget alone requires — - the ``parallel_chunks`` dial (see + Fan the plan out more finely than the byte budget alone requires. + + This is the ``parallel_chunks`` dial: see :func:`~dataretrieval.ogc.chunking.parallel_chunks` for why a caller would want this, and :class:`ChunkPlan`'s ``max_chunks`` parameter for - the cap's contract: total-not-per-axis, a hard ceiling that may land + the cap's contract (total-not-per-axis, a hard ceiling that may land below the cap). Implementation. Each split multiplies the plan by ``(k+1)/k`` for the chosen axis (adding ``total // k`` sub-requests, not one), so a split - is taken only when it keeps :attr:`total` within the cap; when no - in-budget split remains the plan stops *below* the cap rather than + is taken only when it keeps :attr:`total` within the cap. When no + in-budget split remains, the plan stops *below* the cap rather than overshooting (two even axes can reach 4 but not 5, so a cap of 5 yields 4). Each split picks the single largest splittable chunk among the in-budget axes (ties broken by axis-extraction order, then lowest @@ -500,7 +499,7 @@ def _refine(self, max_chunks: int) -> None: # Largest splittable chunk among the axes whose split still fits the # cap. Splitting any chunk of an axis with ``k`` chunks turns that # ``k`` into ``k+1``, so it adds ``total // k`` sub-requests (the - # product of the other axes) regardless of which chunk — hence the + # product of the other axes) regardless of which chunk. Hence the # budget test is per axis, not per chunk. Skipping an over-budget # axis makes ``max_chunks`` a true ceiling. The ranking key is atom # count (``len``), not URL bytes like ``_plan`` — this pass balances @@ -527,9 +526,10 @@ def _refine(self, max_chunks: int) -> None: def _worst_case_args(self) -> dict[str, Any]: """ - Args dict representing the largest sub-request the current - ``self.chunks`` partition will issue — each axis's longest - (by URL-encoded bytes) chunk rendered back in. + Args for the largest sub-request the current partition will issue. + + Each axis contributes its longest chunk (by URL-encoded bytes), + rendered back into the args dict. """ out = dict(self.args) for axis in self.axes: @@ -552,11 +552,11 @@ def total(self) -> int: def iter_sub_args(self) -> Iterator[dict[str, Any]]: """ - Yield substituted args for each sub-request, in deterministic - order — cartesian product over axes in extraction order. + Yield substituted args for each sub-request, in deterministic order. - The same plan yields the same sub-args sequence on every - invocation, so resume is well-defined. + The order is the cartesian product over axes in extraction order. The + same plan yields the same sub-args sequence on every invocation, so + resume is well-defined. Yields ------ diff --git a/dataretrieval/ogc/shaping.py b/dataretrieval/ogc/shaping.py index 00cd3470f..16899c8f3 100644 --- a/dataretrieval/ogc/shaping.py +++ b/dataretrieval/ogc/shaping.py @@ -199,8 +199,7 @@ def _arrange_cols( extra_id_cols: frozenset[str] | set[str] = frozenset(), ) -> pd.DataFrame: """ - Rearranges and renames columns in a DataFrame based on provided - properties and the service output id. + Rearranges and renames DataFrame columns per ``properties`` and the output id. Parameters ---------- @@ -270,7 +269,6 @@ def _type_cols(df: pd.DataFrame, dialect: OgcDialect) -> pd.DataFrame: ------- pd.DataFrame The DataFrame with columns cast to appropriate types. - """ cols = set(df.columns) for col in cols.intersection(dialect.time_cols): @@ -303,7 +301,6 @@ def _sort_rows(df: pd.DataFrame, dialect: OgcDialect) -> pd.DataFrame: ------- pd.DataFrame The DataFrame with rows ordered per the dialect. - """ if not dialect.sort_cols or dialect.sort_cols[0] not in df.columns: return df @@ -365,7 +362,7 @@ def _finalize_ogc( ``max_rows`` is applied here (after dedup/sort, on the *combined* frame) rather than only per-sub-request, so a chunked call's total is bounded - to exactly ``max_rows`` and a resumed call honors the cap too — the + to exactly ``max_rows`` and a resumed call honors the cap too. The per-``_paginate`` ``_row_cap`` is only an early-stop download bound. """ if dialect is None: diff --git a/dataretrieval/progress.py b/dataretrieval/progress.py index 132ea9063..b8ac14e28 100644 --- a/dataretrieval/progress.py +++ b/dataretrieval/progress.py @@ -4,8 +4,7 @@ requests are split into URL-length-safe *chunks* (``chunking`` module), and each request follows ``next`` links across an unknown number of *pages* (``transport.pagination.paginate``). This module surfaces that work as one -line on stderr, -rewritten in place as data arrives:: +line on stderr, rewritten in place as data arrives:: Retrieving: daily · 6 pages · 2,881 rows · 995/1,000 requests remaining @@ -63,8 +62,7 @@ def _group_int(value: str) -> str: def _in_jupyter_kernel() -> bool: - """True when running inside a Jupyter/IPython *kernel* (notebook, lab, - qtconsole). + """True when running inside a Jupyter/IPython *kernel* (notebook, lab, qtconsole). A kernel's ``stderr`` isn't a TTY, but it honors carriage-return rewrites in the cell output area — the same mechanism ``tqdm`` rides on — so the line is diff --git a/dataretrieval/rdb.py b/dataretrieval/rdb.py index 2b52656bd..2a5a6c24b 100644 --- a/dataretrieval/rdb.py +++ b/dataretrieval/rdb.py @@ -32,10 +32,9 @@ def read_rdb(text: str, dtypes: dict[str, type] | None = None) -> pd.DataFrame: text : str The RDB text response from a USGS web service. dtypes : dict[str, type] or None, optional - Optional column-name to dtype hints, forwarded to - ``pandas.read_csv``. Unknown column names are silently ignored, so - callers may safely pass a dict of all columns they might be - interested in. + Column-name to dtype hints, forwarded to ``pandas.read_csv``. Unknown + column names are silently ignored, so callers can pass a dict of every + column they might be interested in. Returns ------- diff --git a/dataretrieval/streamstats.py b/dataretrieval/streamstats.py index 6681191f6..af0763b09 100644 --- a/dataretrieval/streamstats.py +++ b/dataretrieval/streamstats.py @@ -1,5 +1,4 @@ -""" -This module is a wrapper for the StreamStats API (`streamstats documentation`_). +"""Wrapper for the StreamStats API (`streamstats documentation`_). .. _streamstats documentation: https://streamstats.usgs.gov/streamstatsservices/#/ @@ -19,16 +18,16 @@ def download_workspace(workspaceID: str, format: str = "") -> httpx.Response: - """Function to download a StreamStats workspace. + """Download a StreamStats workspace. Parameters ---------- workspaceID: string - Service workspace received from watershed result + Service workspace received from a watershed result. format: string - Download return format. Default will return ESRI geodatabase zipfile. - 'SHAPE' will return a zip file containing shape format. + Format of the download. The default returns an ESRI geodatabase + zipfile; 'SHAPE' returns a zip file containing shape format. Returns ------- @@ -51,11 +50,10 @@ def download_workspace(workspaceID: str, format: str = "") -> httpx.Response: def get_sample_watershed() -> Watershed: - """Sample function to get a watershed object for a location in NY. + """Get a watershed object for a sample location in NY. - Makes the function call :obj:`dataretrieval.streamstats.get_watershed` - with the parameters 'NY', -74.524, 43.939, and returns the watershed - object. + Calls :obj:`dataretrieval.streamstats.get_watershed` with the parameters + 'NY', -74.524, and 43.939, and returns the resulting watershed object. Returns ------- @@ -81,13 +79,13 @@ def get_watershed( simplify: bool = True, format: str = "geojson", ) -> httpx.Response | Watershed: - """Get watershed object based on location + """Get a watershed object for a location. **StreamStats documentation:** Returns a watershed object. The request configuration will determine the - overall request response. However all returns will return a watershed + overall request response. However, all returns will return a watershed object with at least the workspaceid. The workspace id is the id to the - service workspace where files are stored and can be used for further + service workspace where files are stored, and can be used for further processing such as for downloads and flow statistic computations. See: https://streamstats.usgs.gov/streamstatsservices/#/ for more @@ -103,17 +101,16 @@ def get_watershed( ylocation: float Y location of the most downstream point of desired study area. crs: integer, string, optional - EPSG spatial reference code, default is 4326 + EPSG spatial reference code. Default is 4326. includeparameters: bool, optional - Boolean flag to include parameters in response. + Whether to include parameters in the response. includeflowtypes: bool, string, optional - Not yet implemented. Would be a comma separated list of region flow - types to compute with the default being True + Comma-separated list of region flow types to compute, with the default + being True. Not yet implemented. includefeatures: list, optional - Comma separated list of features to include in response. + Comma-separated list of features to include in the response. simplify: bool, optional - Boolean flag controlling whether or not to simplify the returned - result. + Whether to simplify the returned result. format: string, optional Controls the return type, default is 'geojson'. 'geojson' returns the raw ``httpx.Response``; 'object' parses the response into a @@ -187,8 +184,10 @@ class Watershed: """ def __init__(self, rcode: str, xlocation: float, ylocation: float) -> None: - """Delineate the watershed at ``(xlocation, ylocation)`` and - parse the response onto this instance.""" + """Delineate the watershed at ``(xlocation, ylocation)``. + + Parses the response onto this instance. + """ response = cast( httpx.Response, get_watershed(rcode, xlocation, ylocation, format="geojson"), @@ -197,21 +196,19 @@ def __init__(self, rcode: str, xlocation: float, ylocation: float) -> None: @classmethod def from_streamstats_json(cls, streamstats_json: dict[str, Any]) -> Watershed: - """Create a :class:`Watershed` from an already-parsed StreamStats - JSON payload, without issuing a new request. + """Create a :class:`Watershed` from a parsed StreamStats JSON payload. - Builds a fresh instance (via ``__new__``, so the - network-fetching ``__init__`` is bypassed) and populates it; each - call returns an independent object rather than mutating shared - class state. + No new request is issued. Builds a fresh instance (via ``__new__``, so + the network-fetching ``__init__`` is bypassed) and populates it; each + call returns an independent object rather than mutating shared class + state. """ self = cls.__new__(cls) self._populate(streamstats_json) return self def _populate(self, streamstats_json: dict[str, Any]) -> None: - """Extract watershed fields from a StreamStats JSON payload onto - this instance.""" + """Extract watershed fields from ``streamstats_json`` onto this instance.""" self.watershed_point = streamstats_json["featurecollection"][0]["feature"] self.watershed_polygon = streamstats_json["featurecollection"][1]["feature"] self.parameters = streamstats_json["parameters"] diff --git a/dataretrieval/utils.py b/dataretrieval/utils.py index bab5929ed..66f46d492 100644 --- a/dataretrieval/utils.py +++ b/dataretrieval/utils.py @@ -1,6 +1,4 @@ -""" -Useful utilities for data munging. -""" +"""Useful utilities for data munging.""" from __future__ import annotations @@ -108,21 +106,21 @@ def _require_positive_int( def to_str(listlike: object, delimiter: str = ",") -> str | None: - """Translates list-like objects into strings. + """Translate a list-like object into a delimited string. Parameters ---------- listlike: list-like object - An object that is a list, or list-like - (e.g., ``pandas.core.series.Series``) + A list, or a list-like object + (e.g. ``pandas.core.series.Series``). delimiter: string, optional - The delimiter that is placed between entries in listlike when it is - turned into a string. Default value is a comma. + String placed between entries of ``listlike`` when it is turned into a + string. Default value is a comma. Returns ------- listlike: string - The listlike object as string separated by the delimiter + The listlike object as a string separated by the delimiter. Examples -------- @@ -147,8 +145,7 @@ def to_str(listlike: object, delimiter: str = ",") -> str | None: def format_datetime( df: pd.DataFrame, date_field: str, time_field: str, tz_field: str ) -> pd.DataFrame: - """Creates a datetime field from separate date, time, and - time zone fields. + """Create a datetime field from separate date, time, and time zone fields. Assumes ISO 8601. @@ -157,16 +154,16 @@ def format_datetime( df: ``pandas.DataFrame`` A data frame containing date, time, and timezone fields. date_field: string - Name of date column in df. + Name of the date column in ``df``. time_field: string - Name of time column in df. + Name of the time column in ``df``. tz_field: string - Name of time zone column in df. + Name of the time zone column in ``df``. Returns ------- df: ``pandas.DataFrame`` - The data frame with a formatted 'datetime' column + The data frame with a formatted 'datetime' column. """ # create a datetime index from the columns in qwdata response @@ -224,8 +221,7 @@ def _build_utc_datetime( def _attach_datetime_columns(df: pd.DataFrame) -> pd.DataFrame: - """Add ``DateTime`` UTC columns for any Date/Time/TimeZone triplets - and sort the frame by the activity-start datetime. + """Append a UTC ``DateTime`` column per Date/Time/TimeZone triplet. Detects two naming patterns that appear in USGS Samples and Water Quality Portal CSV responses: @@ -300,16 +296,16 @@ class BaseMetadata: Attributes ---------- url : str - Response url + Response url. query_time: datetime.timedelta - Response elapsed time + Response elapsed time. header: httpx.Headers - Response headers + Response headers. """ def __init__(self, response: httpx.Response) -> None: - """Generates a standard set of metadata informed by the response. + """Generate a standard set of metadata informed by the response. Parameters ---------- @@ -367,11 +363,10 @@ def _raise_for_status( *, detail_from: Callable[[httpx.Response], str | None] | None = None, ) -> None: - """Raise the typed :class:`DataRetrievalError` for an HTTP error response; - return ``None`` on success. + """Raise the typed :class:`DataRetrievalError` for an HTTP error response. - Shared by the legacy :func:`query` path (and ``streamstats`` / - ``wateruse``). Delegates the status-to-type mapping to + A success status returns ``None``. Shared by the legacy :func:`query` path + (and ``streamstats`` / ``wateruse``). Delegates the status-to-type mapping to :func:`dataretrieval.exceptions.error_for_status`, except a too-long-URL status (413 / 414): that gets the same actionable "split your query" remediation as the client-side over-long-URL case below, rather than a bare @@ -443,20 +438,19 @@ def _query_impl( ) -> httpx.Response: """Send a query. - Wrapper for httpx.get that handles errors, converts listed - query parameters to comma separated strings, and returns response. + Wrapper for ``httpx.get`` that handles errors, converts listed query + parameters to comma-separated strings, and returns the response. Parameters ---------- url: string - URL to query + URL to query. payload: dict - query parameters passed to ``httpx.get`` + Query parameters passed to ``httpx.get``. delimiter: string - delimiter to use with lists + Delimiter to use with lists. ssl_check: bool - If True, check SSL certificates, if False, do not check SSL, - default is True + Whether to check SSL certificates. Default is True. Returns ------- diff --git a/dataretrieval/waterdata/__init__.py b/dataretrieval/waterdata/__init__.py index eb231469a..48c4d9fb9 100644 --- a/dataretrieval/waterdata/__init__.py +++ b/dataretrieval/waterdata/__init__.py @@ -1,10 +1,9 @@ -""" -Water Data API module for accessing USGS water data services. +"""Water Data API module for accessing USGS water data services. This module provides functions for downloading data from the Water Data APIs, including the USGS Aquarius Samples database. -See https://api.waterdata.usgs.gov/ for API reference. +See https://api.waterdata.usgs.gov/ for the API reference. """ from __future__ import annotations diff --git a/dataretrieval/waterdata/measurements.py b/dataretrieval/waterdata/measurements.py index 18640b0c9..cfd615638 100644 --- a/dataretrieval/waterdata/measurements.py +++ b/dataretrieval/waterdata/measurements.py @@ -46,32 +46,29 @@ def get_field_measurements( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get discrete measurements collected in person during a site visit. - Field measurements are physically measured values collected during a - visit to the monitoring location. Field measurements consist of measurements - of gage height and discharge, and readings of groundwater levels, and are - primarily used as calibration readings for the automated sensors collecting - continuous data. They are collected at a low frequency, and delivery of the - data in WDFN may be delayed due to data processing time. + Field measurements consist of measurements of gage height and discharge, and + readings of groundwater levels. They are used primarily as calibration + readings for the automated sensors that collect continuous data. Field + measurements are collected at a low frequency, and their delivery in WDFN + may be delayed by data processing time. Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of the - agency responsible for the monitoring location (e.g. USGS) with the ID - number of the monitoring location (e.g. 02238500), separated by a hyphen - (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter codes - and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. observing_procedure_code : string or iterable of strings, optional A short code corresponding to the observing procedure for the field measurement. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. See the + The columns to return from the query. See the field-measurements schema in the OpenAPI reference for the available columns (e.g. geometry, id, monitoring_location_id, parameter_code, value, unit_of_measure, approval_status, qualifier, last_modified): @@ -80,34 +77,33 @@ def get_field_measurements( A universally unique identifier (UUID) for the field visit. Multiple measurements may be made during a single field visit. approval_status : string or iterable of strings, optional - Some of the data that you have obtained from this U.S. Geological Survey - database may not have received Director's approval. Any such data values - are qualified as provisional and are subject to revision. Provisional - data are released on the condition that neither the USGS nor the United - States Government may be held liable for any damages resulting from its - use. This field reflects the approval status of each record, and is either - "Approved", meaning processing review has been completed and the data is - approved for publication, or "Provisional" and subject to revision. For - more information about provisional data, go to: + The approval status of each record: either "Approved", meaning + processing review has been completed and the data are approved for + publication, or "Provisional", meaning the data are subject to revision. + Some of the data you obtain from this U.S. Geological Survey database + may not have received Director's approval. Any such data values are + qualified as provisional and are subject to revision. Provisional data + are released on the condition that neither the USGS nor the United + States Government may be held liable for any damages resulting from + their use. For more information about provisional data, see https://waterdata.usgs.gov/provisional-data-statement/. unit_of_measure : string or iterable of strings, optional A human-readable description of the units of measurement associated with an observation. qualifier : string or iterable of strings, optional - This field indicates any qualifiers associated with an observation, for - instance if a sensor may have been impacted by ice or if values were - estimated. + Any qualifiers associated with an observation, for instance whether a + sensor may have been impacted by ice or whether values were estimated. value : string or iterable of strings, optional The value of the observation. Values are transmitted as strings in - the JSON response format in order to preserve precision. + the JSON response format to preserve precision. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a last_modified that - intersects the value of datetime are selected. + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -125,18 +121,17 @@ def get_field_measurements( measuring_agency : string or iterable of strings, optional The agency performing the measurement. skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. time : string, optional - The date an observation represents. You can query this field using date-times - or intervals, adhering to RFC 3339, or using ISO 8601 duration objects. - Intervals may be bounded or half-bounded (double-dots at start or end). - Only features that have a time that intersects the value of datetime are - selected. If a feature has multiple temporal properties, it is the - decision of the server whether only a single temporal property is used - to determine the extent or all relevant temporal properties. + The date an observation represents. You can query this field using + date-times or intervals, adhering to RFC 3339, or using ISO 8601 + duration objects. Intervals may be bounded or half-bounded (double-dots + at start or end). Only features whose time intersects the requested + value are selected. If a feature has multiple temporal properties, the + server decides whether to use a single property or all relevant ones to + determine the extent. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -147,22 +142,20 @@ def get_field_measurements( "PT36H" for the last 36 hours bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See @@ -410,32 +403,27 @@ def get_channel( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get channel-geometry measurements recorded during streamflow field visits. - Channel measurements taken as part of streamflow field measurements. - Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of - the agency responsible for the monitoring location (e.g. USGS) with - the ID number of the monitoring location (e.g. 02238500), separated - by a hyphen (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). field_visit_id : string or iterable of strings, optional - A universally unique identifier (UUID) for the field visit. - Multiple measurements - may be made during a single field visit. + A universally unique identifier (UUID) for the field visit. Multiple + measurements may be made during a single field visit. measurement_number : string or iterable of strings, optional Measurement number. time : string or iterable of strings, optional The date an observation represents. You can query this field using date-times or intervals, adhering to RFC 3339, or using ISO 8601 duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a time that intersects the - value of datetime are selected. If a feature has multiple temporal - properties, it is the decision of the server whether only a single - temporal property is used to determine the extent or all relevant - temporal properties. + at start or end). Only features whose time intersects the requested + value are selected. If a feature has multiple temporal properties, the + server decides whether to use a single property or all relevant ones to + determine the extent. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -482,13 +470,13 @@ def get_channel( measurement_type : string or iterable of strings, optional The type of channel measurement. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a last_modified that - intersects the value of datetime are selected. + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -499,14 +487,13 @@ def get_channel( "PT36H" for the last 36 hours skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. channel_measurement_type : string or iterable of strings, optional The channel measurement type. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. Available + The columns to return from the query. Available options are: geometry, channel_measurements_id, monitoring_location_id, field_visit_id, measurement_number, time, channel_name, channel_flow, channel_flow_unit, channel_width, channel_width_unit, channel_area, @@ -515,24 +502,22 @@ def get_channel( channel_material, channel_evenness, horizontal_velocity_description, vertical_velocity_description, longitudinal_velocity_description, measurement_type, last_modified, channel_measurement_type. The default - (None) will return all columns of the data. + (None) returns all columns. bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See diff --git a/dataretrieval/waterdata/metadata.py b/dataretrieval/waterdata/metadata.py index 0f10f2ae7..97230167f 100644 --- a/dataretrieval/waterdata/metadata.py +++ b/dataretrieval/waterdata/metadata.py @@ -76,31 +76,28 @@ def get_monitoring_locations( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get the catalog of monitoring locations and their attributes. - Location information is basic information about the monitoring location - including the name, identifier, agency responsible for data collection, and - the date the location was established. It also includes information about + Location information includes the name, identifier, agency responsible for + data collection, and the date the location was established. It also includes the type of location, such as stream, lake, or groundwater, and geographic - information about the location, such as state, county, latitude and - longitude, and hydrologic unit code (HUC). + information such as state, county, latitude and longitude, and hydrologic + unit code (HUC). Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of - the agency responsible for the monitoring location (e.g. USGS) with - the ID number of the monitoring location (e.g. 02238500), separated - by a hyphen (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). agency_code : string or iterable of strings, optional The agency that is reporting the data. Agency codes are fixed values assigned by the National Water Information System (NWIS). agency_name : string or iterable of strings, optional The name of the agency that is reporting the data. monitoring_location_number : string or iterable of strings, optional - Each monitoring location in the USGS data base has a unique 8- to - 15-digit identification number. Monitoring location numbers are - assigned based on this logic: + A unique 8- to 15-digit identification number. Every monitoring location + in the USGS database has one, assigned according to this logic: https://help.waterdata.usgs.gov/faq/sites/do-station-numbers-have-any-particular-meaning. monitoring_location_name : string or iterable of strings, optional This is the official name of the monitoring location in the database. @@ -109,10 +106,10 @@ def get_monitoring_locations( The Water Science Centers (WSCs) across the United States use the FIPS state code as the district code. In some cases, monitoring locations and samples may be managed by a water science center that is adjacent to the - state in which the monitoring location actually resides. For example a - monitoring location may have a district code of 30 which translates to - Montana, but the state code could be 56 for Wyoming because that is where - the monitoring location actually is located. + state in which the monitoring location actually resides. For example, a + monitoring location may have a district code of 30, which translates to + Montana, but a state code of 56 for Wyoming, because that is where the + monitoring location is actually located. country_code : string or iterable of strings, optional The code for the country in which the monitoring location is located. country_name : string or iterable of strings, optional @@ -150,14 +147,13 @@ def get_monitoring_locations( site_type : string or iterable of strings, optional A description of the hydrologic setting of the monitoring location. hydrologic_unit_code : string or iterable of strings, optional - The United States is divided and sub-divided into successively smaller - hydrologic units which are classified into four levels: regions, - sub-regions, accounting units, and cataloging units. The hydrologic - units are arranged within each other, from the smallest (cataloging - units) to the largest (regions). Each hydrologic unit is identified by a - unique hydrologic unit code (HUC) consisting of two to eight digits - based on the four levels of classification in the hydrologic unit - system. + A unique hydrologic unit code (HUC) of two to eight digits, based on the + four levels of classification in the hydrologic unit system. The United + States is divided and sub-divided into successively smaller hydrologic + units, classified into four levels: regions, sub-regions, accounting + units, and cataloging units. The hydrologic units are arranged within + each other, from the smallest (cataloging units) to the largest + (regions). basin_code : string or iterable of strings, optional The Basin Code or "drainage basin code" is a two-digit code that further subdivides the 8-digit hydrologic-unit code. The drainage basin code is @@ -208,12 +204,12 @@ def get_monitoring_locations( situation can occur when part of the drainage area consists of very porous soil or depressions that either allow all runoff to enter the groundwater or trap the water in ponds so that rainfall does not - contribute to runoff. A transbasin diversion can also affect the total + contribute to runoff. A transbasin diversion can also affect the total drainage area. time_zone_abbreviation : string or iterable of strings, optional A short code describing the time zone used by a monitoring location. uses_daylight_savings : string or iterable of strings, optional - A flag indicating whether or not a monitoring location uses daylight savings. + A flag indicating whether a monitoring location uses daylight savings. construction_date : string or iterable of strings, optional Date the well was completed. aquifer_code : string or iterable of strings, optional @@ -252,7 +248,7 @@ def get_monitoring_locations( codes `_ is available. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. Available + The columns to return from the query. Available options are: geometry, id, agency_code, agency_name, monitoring_location_number, monitoring_location_name, district_code, country_code, country_name, state_code, state_name, county_code, @@ -268,27 +264,24 @@ def get_monitoring_locations( national_aquifer_code, aquifer_type_code, well_constructed_depth, hole_constructed_depth, depth_source_code. bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See @@ -383,32 +376,29 @@ def get_time_series_metadata( """Get metadata describing the time series available at a location. Use this to discover what a location measures before requesting the - observations themselves. Daily data and continuous measurements are - grouped into time series, - which represent a collection of observations of a single parameter, - potentially aggregated using a standard statistic, at a single monitoring - location. This endpoint provides metadata about those time series, - including their operational thresholds, units of measurement, and when - the earliest and most recent observations in a time series occurred. + observations themselves. Daily data and continuous measurements are grouped + into time series, which represent a collection of observations of a single + parameter, potentially aggregated using a standard statistic, at a single + monitoring location. This endpoint provides metadata about those time + series, including their operational thresholds, units of measurement, and + when the earliest and most recent observations in a time series occurred. Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of - the agency responsible for the monitoring location (e.g. USGS) with - the ID number of the monitoring location (e.g. 02238500), separated - by a hyphen (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter - codes and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. parameter_name : string or iterable of strings, optional A human-understandable name corresponding to parameter_code. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. + The columns to return from the query. Available options are: begin, begin_utc, computation_identifier, computation_period_identifier, end, end_utc, geometry, hydrologic_unit_code, id, last_modified, monitoring_location_id, @@ -421,13 +411,13 @@ def get_time_series_metadata( A complete list of codes and their descriptions can be found at https://help.waterdata.usgs.gov/code/stat_cd_nm_query?stat_nm_cd=%25&fmt=html. hydrologic_unit_code : string or iterable of strings, optional - The United States is divided and sub-divided into successively smaller - hydrologic units which are classified into four levels: regions, - sub-regions, accounting units, and cataloging units. The hydrologic - units are arranged within each other, from the smallest (cataloging units) - to the largest (regions). Each hydrologic unit is identified by a unique - hydrologic unit code (HUC) consisting of two to eight digits based on the - four levels of classification in the hydrologic unit system. + A unique hydrologic unit code (HUC) of two to eight digits, based on the + four levels of classification in the hydrologic unit system. The United + States is divided and sub-divided into successively smaller hydrologic + units, classified into four levels: regions, sub-regions, accounting + units, and cataloging units. The hydrologic units are arranged within + each other, from the smallest (cataloging units) to the largest + (regions). state : string or iterable of strings, optional State/territory filter (the recommended parameter). Accepts a full name (``"Wisconsin"``), a two-letter postal code (``"WI"``), or a two-digit @@ -436,13 +426,13 @@ def get_time_series_metadata( The name of the state or state equivalent in which the monitoring location is located. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a last_modified that - intersects the value of datetime are selected. + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -519,33 +509,30 @@ def get_time_series_metadata( primary : string or iterable of strings, optional parent_time_series_id : string or iterable of strings, optional time_series_id : string or iterable of strings, optional - A unique identifier representing a single time series. This - corresponds to the id field in the time-series-metadata endpoint. + A unique identifier representing a single time series, corresponding to + the id field in the time-series-metadata endpoint. web_description : string or iterable of strings, optional A description of what this time series represents, as used by WDFN and other USGS data dissemination products. skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See diff --git a/dataretrieval/waterdata/nearest.py b/dataretrieval/waterdata/nearest.py index 3ee0c063d..24b72c4d2 100644 --- a/dataretrieval/waterdata/nearest.py +++ b/dataretrieval/waterdata/nearest.py @@ -29,7 +29,7 @@ def get_nearest_continuous( on_tie: OnTie = "first", **kwargs: Any, ) -> tuple[pd.DataFrame, BaseMetadata]: - """For each target timestamp, return the nearest continuous observation. + """Return the nearest continuous observation to each target timestamp. Builds one bracketed ``(time >= t-window AND time <= t+window)`` clause per target, joins them as a top-level CQL ``OR`` filter, and lets diff --git a/dataretrieval/waterdata/reference.py b/dataretrieval/waterdata/reference.py index b19b31bc2..5d43ac357 100644 --- a/dataretrieval/waterdata/reference.py +++ b/dataretrieval/waterdata/reference.py @@ -43,14 +43,12 @@ def get_reference_table( "parameter-codes", "reliability-codes", "site-types", "states", "statistic-codes", "topographic-codes", "time-zone-codes" limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. query: dictionary, optional - The optional query parameter can be used to pass a dictionary of - query parameters to the collection API call. + A dictionary of extra query parameters to pass to the collection API + call. max_rows : int, optional Cap the total number of rows returned, stopping pagination early instead of downloading the whole table. Useful for cheaply @@ -121,9 +119,9 @@ def get_queryables(collection: str) -> tuple[pd.DataFrame, BaseMetadata]: Every OGC collection (``daily``, ``continuous``, ``monitoring-locations``, ...) advertises the set of properties that can be filtered on -- exposed as the typed keyword arguments of the matching ``get_*`` function, and usable - directly in a CQL2 ``filter``. This returns that set, so the available - filters can be discovered programmatically and monitored for upstream - additions. + directly in a CQL2 ``filter``. This function returns that set, so you can + discover the available filters programmatically and monitor them for + upstream additions. Parameters ---------- diff --git a/dataretrieval/waterdata/samples.py b/dataretrieval/waterdata/samples.py index bda8d363e..864bc962b 100644 --- a/dataretrieval/waterdata/samples.py +++ b/dataretrieval/waterdata/samples.py @@ -178,32 +178,32 @@ def get_samples( down the database's ability to assemble a result before it times out, so filtering narrowly is faster than filtering exhaustively. - The web GUI for the Samples database can be found here: + The web GUI for the Samples database is at https://waterdata.usgs.gov/download-samples/#dataProfile=site - If you would like more details on feasible query parameters (complete with - examples), please visit the Samples database swagger docs, here: + For more details on feasible query parameters, complete with examples, see + the Samples database swagger docs at https://api.waterdata.usgs.gov/samples-data/docs#/ Parameters ---------- ssl_check : bool, optional - Check the SSL certificate. + Verify the server's SSL certificate. service : string One of the available Samples services: "results", "locations", "activities", "projects", or "organizations". Defaults to "results". profile : string - One of the available profiles associated with a service. Options for each - service are: - results - "fullphyschem", "basicphyschem", - "fullbio", "basicbio", "narrow", - "resultdetectionquantitationlimit", - "labsampleprep", "count" - locations - "site", "count" - activities - "sampact", "actmetric", - "actgroup", "count" - projects - "project", "projectmonitoringlocationweight" - organizations - "organization", "count" + One of the available profiles associated with a service. Options for + each service are: + + * results - "fullphyschem", "basicphyschem", "fullbio", "basicbio", + "narrow", "resultdetectionquantitationlimit", "labsampleprep", + "count" + * locations - "site", "count" + * activities - "sampact", "actmetric", "actgroup", "count" + * projects - "project", "projectmonitoringlocationweight" + * organizations - "organization", "count" + activity_media_name : string or iterable of strings, optional Name or code indicating environmental medium in which sample was taken. Call ``get_codes("samplemedia")`` for the valid inputs. @@ -400,7 +400,7 @@ def get_samples_summary( are accepted by the service but return an empty result, so a prefix is effectively required. (Samples API: ``monitoringLocationIdentifier``) ssl_check : bool, optional - Check the SSL certificate. Default is True. + Verify the server's SSL certificate. Default is True. Returns ------- diff --git a/dataretrieval/waterdata/stats.py b/dataretrieval/waterdata/stats.py index 0a3523654..5666fb305 100644 --- a/dataretrieval/waterdata/stats.py +++ b/dataretrieval/waterdata/stats.py @@ -53,9 +53,9 @@ def _handle_nesting( body: dict[str, Any], geopd: bool = False, ) -> pd.DataFrame: - """ - Takes nested json from stats service and flattens into a dataframe with - one row per monitoring location, parameter, and statistic. + """Flatten nested JSON from the stats service into a dataframe. + + The result has one row per monitoring location, parameter, and statistic. Parameters ---------- @@ -140,13 +140,12 @@ def _handle_nesting( def _expand_percentiles(df: pd.DataFrame) -> pd.DataFrame: - """ - Takes percentile value and thresholds columns containing lists - of values and turns each list element into its own row in the - original dataframe. Exploded ``'nan'`` values are dropped. If - no percentile data exist, it adds a percentile column and - populates it with the percentile assigned to min, max, and - median. + """Explode percentile value and threshold lists into one row per element. + + The "values" and "percentiles" columns hold lists; each element becomes its + own row in the original dataframe, and exploded ``'nan'`` values are + dropped. If no percentile data exist, a percentile column is added and + populated with the percentile assigned to min, max, and median. Parameters ---------- @@ -215,13 +214,12 @@ def get_data( expand_percentiles: bool, client: httpx.AsyncClient | None = None, ) -> tuple[pd.DataFrame, BaseMetadata]: - """ - Retrieves statistical data from a specified endpoint and returns it - as a pandas DataFrame with metadata. + """Retrieve statistical data from a statistics endpoint. - This function prepares request arguments, constructs API requests, - handles pagination, processes results, and formats output according - to the specified parameters. + Returns the data as a pandas DataFrame with metadata. This function + prepares request arguments, constructs API requests, handles pagination, + processes results, and formats output according to the specified + parameters. The stats path doesn't go through ``multi_value_chunked`` (its query shape has no chunkable list axes), so it drives transport pagination @@ -237,10 +235,10 @@ def get_data( The statistics service type (for example, "observationNormals" or "observationIntervals"). expand_percentiles : bool - Determines whether the percentiles column is expanded so that - each percentile gets its own row in the returned dataframe. If - True and the user requests a computation_type other than - percentiles, a percentile column is still returned. + Whether to expand the percentiles column so that each percentile gets + its own row in the returned dataframe. If True and the caller requests a + computation_type other than percentiles, a percentile column is still + returned. client : httpx.AsyncClient, optional Caller-borrowed async client. ``None`` (default) opens a temporary one inside the portal. Primarily a test seam. Deliberately does *not* fall @@ -254,7 +252,8 @@ def get_data( pd.DataFrame A DataFrame containing the retrieved and processed statistical data. BaseMetadata - A metadata object containing request information including URL and query time. + A metadata object with request information, including the URL and + query time. Raises ------ diff --git a/dataretrieval/waterdata/time_series.py b/dataretrieval/waterdata/time_series.py index 819e73a67..32657d3bc 100644 --- a/dataretrieval/waterdata/time_series.py +++ b/dataretrieval/waterdata/time_series.py @@ -50,9 +50,6 @@ def get_daily( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get daily values: one value per monitoring location, parameter, and day. - Daily data provide one data value to represent water conditions for the - day. - Throughout much of the history of the USGS, the primary water data available was daily data collected manually at the monitoring location once each day. With improved availability of computer storage and automated transmission of @@ -66,66 +63,64 @@ def get_daily( Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of - the agency responsible for the monitoring location (e.g. USGS) with - the ID number of the monitoring location (e.g. 02238500), separated - by a hyphen (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter - codes and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. statistic_id : string or iterable of strings, optional A code corresponding to the statistic an observation represents. Example codes include 00001 (max), 00002 (min), and 00003 (mean). A complete list of codes and their descriptions can be found at https://help.waterdata.usgs.gov/code/stat_cd_nm_query?stat_nm_cd=%25&fmt=html. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. + The columns to return from the query. Available options are: geometry, id, time_series_id, monitoring_location_id, parameter_code, statistic_id, time, value, unit_of_measure, approval_status, qualifier, last_modified time_series_id : string or iterable of strings, optional - A unique identifier representing a single time series. This - corresponds to the id field in the time-series-metadata endpoint. + A unique identifier representing a single time series, corresponding to + the id field in the time-series-metadata endpoint. daily_id : string or iterable of strings, optional A universally unique identifier (UUID) representing a single version of - a record. It is not stable over time. Every time the record is refreshed - in our database (which may happen as part of normal operations and does - not imply any change to the data itself) a new ID will be generated. To - uniquely identify a single observation over time, compare the time and - time_series_id fields; each time series will only have a single + a record. The UUID is not stable over time: every time the record is + refreshed in our database, a new ID is generated. A refresh may happen + as part of normal operations and does not imply any change to the data + itself. To uniquely identify a single observation over time, compare the + time and time_series_id fields; each time series has only a single observation at a given time. approval_status : string or iterable of strings, optional - Some of the data that you have obtained from this U.S. Geological Survey - database may not have received Director's approval. Any such data values - are qualified as provisional and are subject to revision. Provisional - data are released on the condition that neither the USGS nor the United - States Government may be held liable for any damages resulting from its - use. This field reflects the approval status of each record, and is either - "Approved", meaning processing review has been completed and the data is - approved for publication, or "Provisional" and subject to revision. For - more information about provisional data, go to: + The approval status of each record: either "Approved", meaning + processing review has been completed and the data are approved for + publication, or "Provisional", meaning the data are subject to revision. + Some of the data you obtain from this U.S. Geological Survey database + may not have received Director's approval. Any such data values are + qualified as provisional and are subject to revision. Provisional data + are released on the condition that neither the USGS nor the United + States Government may be held liable for any damages resulting from + their use. For more information about provisional data, see https://waterdata.usgs.gov/provisional-data-statement/. unit_of_measure : string or iterable of strings, optional A human-readable description of the units of measurement associated with an observation. qualifier : string or iterable of strings, optional - This field indicates any qualifiers associated with an observation, for - instance if a sensor may have been impacted by ice or if values were - estimated. + Any qualifiers associated with an observation, for instance whether a + sensor may have been impacted by ice or whether values were estimated. value : string or iterable of strings, optional The value of the observation. Values are transmitted as strings in - the JSON response format in order to preserve precision. + the JSON response format to preserve precision. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -135,22 +130,18 @@ def get_daily( * Duration objects: "P1M" for data from the past month or "PT36H" for the last 36 hours - Only features that have a last_modified that intersects the value of - datetime are selected. skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. time : string, optional The date an observation represents. You can query this field using date-times or intervals, adhering to RFC 3339, or using ISO 8601 duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a time that intersects the - value of datetime are selected. If a feature has multiple temporal - properties, it is the decision of the server whether only a single - temporal property is used to determine the extent or all relevant - temporal properties. + at start or end). Only features whose time intersects the requested + value are selected. If a feature has multiple temporal properties, the + server decides whether to use a single property or all relevant ones to + determine the extent. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -161,22 +152,20 @@ def get_daily( "PT36H" for the last 36 hours bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See @@ -287,37 +276,32 @@ def get_continuous( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get continuous sensor observations, typically at a 15-minute interval. - Continuous data provide instantaneous water conditions. - This is an early version of the continuous endpoint that is feature-complete - and is being made available for limited use. Geometries are not included + and is being made available for limited use. Geometries are not included with the continuous endpoint. If the "time" input is left blank, the service - will return the most recent year of measurements. Users may request no more - than three years of data with each function call. + returns the most recent year of measurements. Users may request no more than + three years of data with each function call. Continuous data are collected at a high frequency, typically 15-minute - intervals. Depending on the specific monitoring location, the data may be - transmitted automatically via telemetry and be available on WDFN within - minutes of collection, while other times the delivery of data may be delayed - if the monitoring location does not have the capacity to automatically - transmit data. Continuous data are described by parameter name and - parameter code (pcode). These data might also be referred to as + intervals. Depending on the monitoring location, the data may be transmitted + automatically via telemetry and be available on WDFN within minutes of + collection. Delivery may be delayed where the monitoring location cannot + transmit data automatically. Continuous data are described by parameter name + and parameter code (pcode). These data might also be referred to as "instantaneous values" or "IV". Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of - the agency responsible for the monitoring location (e.g. USGS) with - the ID number of the monitoring location (e.g. 02238500), separated - by a hyphen (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter - codes and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. statistic_id : string or iterable of strings, optional A code corresponding to the statistic an observation represents. Continuous data are nearly always associated with statistic id @@ -326,49 +310,49 @@ def get_continuous( descriptions can be found at https://help.waterdata.usgs.gov/code/stat_cd_nm_query?stat_nm_cd=%25&fmt=html. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. + The columns to return from the query. Available options are: geometry, id, time_series_id, monitoring_location_id, parameter_code, statistic_id, time, value, unit_of_measure, approval_status, qualifier, last_modified time_series_id : string or iterable of strings, optional - A unique identifier representing a single time series. This - corresponds to the id field in the time-series-metadata endpoint. + A unique identifier representing a single time series, corresponding to + the id field in the time-series-metadata endpoint. continuous_id : string or iterable of strings, optional A universally unique identifier (UUID) representing a single version of - a record. It is not stable over time. Every time the record is refreshed - in our database (which may happen as part of normal operations and does - not imply any change to the data itself) a new ID will be generated. To - uniquely identify a single observation over time, compare the time and - time_series_id fields; each time series will only have a single + a record. The UUID is not stable over time: every time the record is + refreshed in our database, a new ID is generated. A refresh may happen + as part of normal operations and does not imply any change to the data + itself. To uniquely identify a single observation over time, compare the + time and time_series_id fields; each time series has only a single observation at a given time. approval_status : string or iterable of strings, optional - Some of the data that you have obtained from this U.S. Geological Survey - database may not have received Director's approval. Any such data values - are qualified as provisional and are subject to revision. Provisional - data are released on the condition that neither the USGS nor the United - States Government may be held liable for any damages resulting from its - use. This field reflects the approval status of each record, and is either - "Approved", meaning processing review has been completed and the data is - approved for publication, or "Provisional" and subject to revision. For - more information about provisional data, go to: + The approval status of each record: either "Approved", meaning + processing review has been completed and the data are approved for + publication, or "Provisional", meaning the data are subject to revision. + Some of the data you obtain from this U.S. Geological Survey database + may not have received Director's approval. Any such data values are + qualified as provisional and are subject to revision. Provisional data + are released on the condition that neither the USGS nor the United + States Government may be held liable for any damages resulting from + their use. For more information about provisional data, see https://waterdata.usgs.gov/provisional-data-statement/. unit_of_measure : string or iterable of strings, optional A human-readable description of the units of measurement associated with an observation. qualifier : string or iterable of strings, optional - This field indicates any qualifiers associated with an observation, for - instance if a sensor may have been impacted by ice or if values were - estimated. + Any qualifiers associated with an observation, for instance whether a + sensor may have been impacted by ice or whether values were estimated. value : string or iterable of strings, optional The value of the observation. Values are transmitted as strings in - the JSON response format in order to preserve precision. + the JSON response format to preserve precision. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -378,17 +362,14 @@ def get_continuous( * Duration objects: "P1M" for data from the past month or "PT36H" for the last 36 hours - Only features that have a last_modified that intersects the value of - datetime are selected. time : string, optional The date an observation represents. You can query this field using date-times or intervals, adhering to RFC 3339, or using ISO 8601 duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a time that intersects the - value of datetime are selected. If a feature has multiple temporal - properties, it is the decision of the server whether only a single - temporal property is used to determine the extent or all relevant - temporal properties. + at start or end). Only features whose time intersects the requested + value are selected. If a feature has multiple temporal properties, the + server decides whether to use a single property or all relevant ones to + determine the extent. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -399,14 +380,12 @@ def get_continuous( "PT36H" for the last 36 hours limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 10000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 10000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See @@ -498,80 +477,78 @@ def get_latest_continuous( """Get only the most recent observation of each continuous time series. Use this for a current-conditions view; use :func:`get_continuous` for a - history. Continuous data are collected via automated sensors - installed at a monitoring location. They are collected at a high frequency - and often at a fixed 15-minute interval. Depending on the specific monitoring - location, the data may be transmitted automatically via telemetry and be - available on WDFN within minutes of collection, while other times the delivery - of data may be delayed if the monitoring location does not have the capacity to - automatically transmit data. Continuous data are described by parameter name + history. + + Continuous data are collected via automated sensors installed at a + monitoring location, at a high frequency and often at a fixed 15-minute + interval. Depending on the monitoring location, the data may be transmitted + automatically via telemetry and be available on WDFN within minutes of + collection. Delivery may be delayed where the monitoring location cannot + transmit data automatically. Continuous data are described by parameter name and parameter code. These data might also be referred to as "instantaneous values" or "IV". Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of the - agency responsible for the monitoring location (e.g. USGS) with the ID - number of the monitoring location (e.g. 02238500), separated by a hyphen - (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter codes - and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. statistic_id : string or iterable of strings, optional A code corresponding to the statistic an observation represents. Example codes include 00001 (max), 00002 (min), and 00003 (mean). A complete list of codes and their descriptions can be found at https://help.waterdata.usgs.gov/code/stat_cd_nm_query?stat_nm_cd=%25&fmt=html. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. Available + The columns to return from the query. Available options are: geometry, id, time_series_id, monitoring_location_id, parameter_code, statistic_id, time, value, unit_of_measure, approval_status, qualifier, last_modified time_series_id : string or iterable of strings, optional - A unique identifier representing a single time series. This - corresponds to the id field in the time-series-metadata endpoint. + A unique identifier representing a single time series, corresponding to + the id field in the time-series-metadata endpoint. latest_continuous_id : string or iterable of strings, optional A universally unique identifier (UUID) representing a single version of - a record. It is not stable over time. Every time the record is refreshed - in our database (which may happen as part of normal operations and does - not imply any change to the data itself) a new ID will be generated. To - uniquely identify a single observation over time, compare the time and - time_series_id fields; each time series will only have a single + a record. The UUID is not stable over time: every time the record is + refreshed in our database, a new ID is generated. A refresh may happen + as part of normal operations and does not imply any change to the data + itself. To uniquely identify a single observation over time, compare the + time and time_series_id fields; each time series has only a single observation at a given time. approval_status : string or iterable of strings, optional - Some of the data that you have obtained from this U.S. Geological Survey - database may not have received Director's approval. Any such data values - are qualified as provisional and are subject to revision. Provisional - data are released on the condition that neither the USGS nor the United - States Government may be held liable for any damages resulting from its - use. This field reflects the approval status of each record, and is either - "Approved", meaning processing review has been completed and the data is - approved for publication, or "Provisional" and subject to revision. For - more information about provisional data, go to: + The approval status of each record: either "Approved", meaning + processing review has been completed and the data are approved for + publication, or "Provisional", meaning the data are subject to revision. + Some of the data you obtain from this U.S. Geological Survey database + may not have received Director's approval. Any such data values are + qualified as provisional and are subject to revision. Provisional data + are released on the condition that neither the USGS nor the United + States Government may be held liable for any damages resulting from + their use. For more information about provisional data, see https://waterdata.usgs.gov/provisional-data-statement/. unit_of_measure : string or iterable of strings, optional A human-readable description of the units of measurement associated with an observation. qualifier : string or iterable of strings, optional - This field indicates any qualifiers associated with an observation, for - instance if a sensor may have been impacted by ice or if values were - estimated. + Any qualifiers associated with an observation, for instance whether a + sensor may have been impacted by ice or whether values were estimated. value : string or iterable of strings, optional The value of the observation. Values are transmitted as strings in - the JSON response format in order to preserve precision. + the JSON response format to preserve precision. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a last_modified that - intersects the value of datetime are selected. + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -582,19 +559,17 @@ def get_latest_continuous( "PT36H" for the last 36 hours skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. time : string, optional The date an observation represents. You can query this field using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a time that intersects the - value of datetime are selected. If a feature has multiple temporal - properties, it is the decision of the server whether only a single - temporal property is used to determine the extent or all relevant - temporal properties. + duration objects. Intervals may be bounded or half-bounded (double-dots + at start or end). Only features whose time intersects the requested + value are selected. If a feature has multiple temporal properties, the + server decides whether to use a single property or all relevant ones to + determine the extent. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -605,22 +580,20 @@ def get_latest_continuous( "PT36H" for the last 36 hours bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See @@ -715,9 +688,9 @@ def get_latest_daily( """Get only the most recent daily value of each time series. Use this for a current-conditions view; use :func:`get_daily` for a - history. Daily data provide one data value to represent water conditions - for the day. + history. + Daily data provide one data value to represent water conditions for the day. Throughout much of the history of the USGS, the primary water data available was daily data collected manually at the monitoring location once each day. With improved availability of computer storage and automated transmission of @@ -731,67 +704,64 @@ def get_latest_daily( Parameters ---------- monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of the - agency responsible for the monitoring location (e.g. USGS) with the ID - number of the monitoring location (e.g. 02238500), separated by a hyphen - (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter codes - and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. statistic_id : string or iterable of strings, optional A code corresponding to the statistic an observation represents. Example codes include 00001 (max), 00002 (min), and 00003 (mean). A complete list of codes and their descriptions can be found at https://help.waterdata.usgs.gov/code/stat_cd_nm_query?stat_nm_cd=%25&fmt=html. properties : string or iterable of strings, optional - A list of requested columns to be returned from the query. Available + The columns to return from the query. Available options are: geometry, id, time_series_id, monitoring_location_id, parameter_code, statistic_id, time, value, unit_of_measure, approval_status, qualifier, last_modified time_series_id : string or iterable of strings, optional - A unique identifier representing a single time series. This - corresponds to the id field in the time-series-metadata endpoint. + A unique identifier representing a single time series, corresponding to + the id field in the time-series-metadata endpoint. latest_daily_id : string or iterable of strings, optional A universally unique identifier (UUID) representing a single version of - a record. It is not stable over time. Every time the record is refreshed - in our database (which may happen as part of normal operations and does - not imply any change to the data itself) a new ID will be generated. To - uniquely identify a single observation over time, compare the time and - time_series_id fields; each time series will only have a single + a record. The UUID is not stable over time: every time the record is + refreshed in our database, a new ID is generated. A refresh may happen + as part of normal operations and does not imply any change to the data + itself. To uniquely identify a single observation over time, compare the + time and time_series_id fields; each time series has only a single observation at a given time. approval_status : string or iterable of strings, optional - Some of the data that you have obtained from this U.S. Geological Survey - database may not have received Director's approval. Any such data values - are qualified as provisional and are subject to revision. Provisional - data are released on the condition that neither the USGS nor the United - States Government may be held liable for any damages resulting from its - use. This field reflects the approval status of each record, and is either - "Approved", meaning processing review has been completed and the data is - approved for publication, or "Provisional" and subject to revision. For - more information about provisional data, go to: + The approval status of each record: either "Approved", meaning + processing review has been completed and the data are approved for + publication, or "Provisional", meaning the data are subject to revision. + Some of the data you obtain from this U.S. Geological Survey database + may not have received Director's approval. Any such data values are + qualified as provisional and are subject to revision. Provisional data + are released on the condition that neither the USGS nor the United + States Government may be held liable for any damages resulting from + their use. For more information about provisional data, see https://waterdata.usgs.gov/provisional-data-statement/. unit_of_measure : string or iterable of strings, optional A human-readable description of the units of measurement associated with an observation. qualifier : string or iterable of strings, optional - This field indicates any qualifiers associated with an observation, for - instance if a sensor may have been impacted by ice or if values were - estimated. + Any qualifiers associated with an observation, for instance whether a + sensor may have been impacted by ice or whether values were estimated. value : string or iterable of strings, optional The value of the observation. Values are transmitted as strings in - the JSON response format in order to preserve precision. + the JSON response format to preserve precision. last_modified : string, optional - The last time a record was refreshed in our database. This may happen - due to regular operational processes and does not necessarily indicate - that anything about the measurement has changed. You can query this field - using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a last_modified that - intersects the value of datetime are selected. + The last time a record was refreshed in our database. A refresh may + happen due to regular operational processes and does not necessarily + indicate that anything about the measurement has changed. You can query + this field using date-times or intervals, adhering to RFC 3339, or using + ISO 8601 duration objects. Intervals may be bounded or half-bounded + (double-dots at start or end). Only features whose last_modified + intersects the requested value are selected. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -802,19 +772,17 @@ def get_latest_daily( "PT36H" for the last 36 hours skip_geometry : boolean, optional - This option can be used to skip response geometries for each feature. - The returning object will be a data frame with no spatial information. - Note that the USGS Water Data APIs use camelCase "skipGeometry" in - CQL2 queries. + If True, the response omits the geometry of each feature and the + returned object is a data frame with no spatial information. The USGS + Water Data APIs use camelCase "skipGeometry" in CQL2 queries. time : string, optional The date an observation represents. You can query this field using date-times or intervals, adhering to RFC 3339, or using ISO 8601 - duration objects. Intervals may be bounded or half-bounded (double-dots - at start or end). Only features that have a time that intersects the - value of datetime are selected. If a feature has multiple temporal - properties, it is the decision of the server whether only a single - temporal property is used to determine the extent or all relevant - temporal properties. + duration objects. Intervals may be bounded or half-bounded (double-dots + at start or end). Only features whose time intersects the requested + value are selected. If a feature has multiple temporal properties, the + server decides whether to use a single property or all relevant ones to + determine the extent. Examples: * A date-time: "2018-02-12T23:20:50Z" @@ -825,22 +793,20 @@ def get_latest_daily( "PT36H" for the last 36 hours bbox : list of numbers, optional - Only features that have a geometry that intersects the bounding box are - selected. The bounding box is provided as four or six numbers, - depending on whether the coordinate reference system includes a vertical - axis (height or depth). Coordinates are assumed to be in crs 4326. The - expected format is ``[xmin, ymin, xmax, ymax]``, i.e. - ``[Western-most longitude, Southern-most latitude, Eastern-most - longitude, Northern-most latitude]``. + Only features whose geometry intersects the bounding box are selected. + The bounding box is provided as four or six numbers, depending on + whether the coordinate reference system includes a vertical axis (height + or depth). Coordinates are assumed to be in crs 4326. The expected + format is ``[xmin, ymin, xmax, ymax]``, i.e. ``[Western-most longitude, + Southern-most latitude, Eastern-most longitude, Northern-most + latitude]``. limit : int, optional - The optional limit parameter is used to control the subset of the - selected features that should be returned in each page. The maximum - allowable limit is 50000. It may be beneficial to set this number lower - if your internet connection is spotty. The default (None) will set the - limit to the maximum allowable limit for the service. - This is a per-page size, not a cap on the total result: a query - matching more rows than ``limit`` still returns every matching row - across multiple pages. Use ``max_rows`` to cap the total instead. + The number of features returned in each page. The maximum allowable + limit is 50000; the default (None) requests that maximum. Set a lower + number if your internet connection is spotty. This is a per-page size, + not a cap on the total result: a query matching more rows than ``limit`` + still returns every matching row across multiple pages. Use ``max_rows`` + to cap the total instead. filter, filter_lang : optional Server-side CQL filter passed through as the OGC ``filter`` / ``filter-lang`` query parameters. See @@ -965,52 +931,49 @@ def get_stats_por( end_date: string or datetime, optional End day for the query in the month-day format (MM-DD). monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of the - agency responsible for the monitoring location (e.g. USGS) with the ID - number of the monitoring location (e.g. 02238500), separated by a hyphen - (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). page_size : int, optional The number of results to return per page, where one result represents a monitoring location. The default is 1000. parent_time_series_id: string, optional - The parent_time_series_id returns statistics tied to a - particular database entry. + Returns statistics tied to a particular database entry. site_type_code: string, optional - Site type code query parameter. - A list of valid site type codes is available at: + Site type code query parameter. A list of valid site type codes is + available at https://api.waterdata.usgs.gov/ogcapi/v0/collections/site-types/items. Example: "GW" (Groundwater site) site_type_name: string, optional Site type name query parameter. parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter codes - and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. normal_type : string, optional Filter the returned normals to a single period. If unspecified (default), all matching data are returned. Available values: "DOY" (day-of-year) and "MOY" (month-of-year). expand_percentiles : boolean - Percentile data for a given day of year or month of year by default - are returned from the service as lists of string values and percentile - thresholds in the "values" and "percentiles" columns, respectively. - When `expand_percentiles` is set to True (default), each value and - percentile threshold specific to a computation id are returned as - individual rows in the dataframe, with the value reported in the - "value" column and the corresponding percentile reported in a - "percentile" column (and the "values" and "percentiles" columns - are removed). Missing percentile values expressed as 'nan' in the - list of string values are removed from the dataframe to save space. - Setting `expand_percentiles` to False retains the "values" and - "percentiles" columns produced by the service. Including - both 'percentiles' and one or more other statistics ('median', - 'minimum', 'maximum', or 'arithmetic_mean') in the `computation_type` - argument will return both the "values" column, containing the list - of percentile threshold values, and a "value" column, containing - the singular summary value for the other statistics. + Whether to expand percentile lists into one row per percentile. + By default, the service returns percentile data for a given day of year + or month of year as lists of string values and percentile thresholds, in + the "values" and "percentiles" columns respectively. When + `expand_percentiles` is True (default), each value and percentile + threshold specific to a computation id becomes its own row in the + dataframe: the value is reported in a "value" column and the + corresponding percentile in a "percentile" column, and the "values" and + "percentiles" columns are removed. Missing percentile values, expressed + as 'nan' in the list of string values, are removed from the dataframe to + save space. Setting `expand_percentiles` to False retains the "values" + and "percentiles" columns produced by the service. Including both + 'percentiles' and one or more other statistics ('median', 'minimum', + 'maximum', or 'arithmetic_mean') in the `computation_type` argument + returns both the "values" column, containing the list of percentile + threshold values, and a "value" column, containing the singular summary + value for the other statistics. Returns ------- @@ -1110,55 +1073,52 @@ def get_stats_date_range( End date for the query in the year-month-day format (YYYY-MM-DD). monitoring_location_id : string or iterable of strings, optional - A unique identifier representing a single monitoring location. This - corresponds to the id field in the monitoring-locations endpoint. - Monitoring location IDs are created by combining the agency code of the - agency responsible for the monitoring location (e.g. USGS) with the ID - number of the monitoring location (e.g. 02238500), separated by a hyphen - (e.g. USGS-02238500). + A unique identifier representing a single monitoring location, + corresponding to the id field in the monitoring-locations endpoint. IDs + combine the agency code of the agency responsible for the monitoring + location (e.g. USGS) with the location's ID number (e.g. 02238500), + separated by a hyphen (e.g. USGS-02238500). page_size : int, optional The number of results to return per page, where one result represents a monitoring location. The default is 1000. parent_time_series_id: string, optional - The parent_time_series_id returns statistics tied to a - particular database entry. + Returns statistics tied to a particular database entry. site_type_code: string, optional - Site type code query parameter. - You can see a list of valid site type codes here: + Site type code query parameter. A list of valid site type codes is + available at https://api.waterdata.usgs.gov/ogcapi/v0/collections/site-types/items. Example: "GW" (Groundwater site) site_type_name: string, optional - Site type name query parameter. - You can see a list of valid site type names here: + Site type name query parameter. A list of valid site type names is + available at https://api.waterdata.usgs.gov/ogcapi/v0/collections/site-types/items. Example: "Well" parameter_code : string or iterable of strings, optional - Parameter codes are 5-digit codes used to identify the constituent - measured and the units of measure. A complete list of parameter codes - and associated groupings can be found at - https://help.waterdata.usgs.gov/codes-and-parameters/parameters. + A 5-digit code identifying the constituent measured and the units of + measure. A complete list of parameter codes and associated groupings is + available at https://help.waterdata.usgs.gov/codes-and-parameters/parameters. interval_type : string or iterable of strings, optional Filter the returned intervals to one or more periods. If unspecified (default), all matching data are returned. Available values: "M" (month), "CY" (calendar year), and "WY" (water year). expand_percentiles : boolean - Percentile data for a given day of year or month of year by default - are returned from the service as lists of string values and percentile - thresholds in the "values" and "percentiles" columns, respectively. - When `expand_percentiles` is set to True (default), each value and - percentile threshold specific to a computation id are returned as - individual rows in the dataframe, with the value reported in the - "value" column and the corresponding percentile reported in a - "percentile" column (and the "values" and "percentiles" columns - are removed). Missing percentile values expressed as 'nan' in the - list of string values are removed from the dataframe to save space. - Setting `expand_percentiles` to False retains the "values" and - "percentiles" columns produced by the service. Including - both 'percentiles' and one or more other statistics ('median', - 'minimum', 'maximum', or 'arithmetic_mean') in the `computation_type` - argument will return both the "values" column, containing the list - of percentile threshold values, and a "value" column, containing - the singular summary value for the other statistics. + Whether to expand percentile lists into one row per percentile. + By default, the service returns percentile data for a given day of year + or month of year as lists of string values and percentile thresholds, in + the "values" and "percentiles" columns respectively. When + `expand_percentiles` is True (default), each value and percentile + threshold specific to a computation id becomes its own row in the + dataframe: the value is reported in a "value" column and the + corresponding percentile in a "percentile" column, and the "values" and + "percentiles" columns are removed. Missing percentile values, expressed + as 'nan' in the list of string values, are removed from the dataframe to + save space. Setting `expand_percentiles` to False retains the "values" + and "percentiles" columns produced by the service. Including both + 'percentiles' and one or more other statistics ('median', 'minimum', + 'maximum', or 'arithmetic_mean') in the `computation_type` argument + returns both the "values" column, containing the list of percentile + threshold values, and a "value" column, containing the singular summary + value for the other statistics. Returns ------- diff --git a/dataretrieval/waterdata/utils.py b/dataretrieval/waterdata/utils.py index aff35a7a7..18bb8129b 100644 --- a/dataretrieval/waterdata/utils.py +++ b/dataretrieval/waterdata/utils.py @@ -128,8 +128,9 @@ def _flatten_queryables(local_vars: dict[str, Any]) -> dict[str, Any]: - """Merge a getter's ``**queryables`` passthrough kwargs -- collected by - ``locals()`` under the ``queryables`` key -- up into ``local_vars`` as + """Merge a getter's ``**queryables`` passthrough kwargs into ``local_vars``. + + ``locals()`` collects them under the ``queryables`` key; this lifts them to top-level entries, so an extra server-side filter such as ``state_name="Wisconsin"`` is normalized, mutual-exclusion-checked, and sent exactly like a named param. See @@ -159,16 +160,15 @@ def _get_args( def _with_state(local_vars: dict[str, Any], *, to: str, into: str) -> dict[str, Any]: - """Resolve the unified ``state`` argument into an endpoint's native state - queryable, returning the (mutated) args mapping. - - ``state`` is the canonical, format-flexible parameter (full name / postal / - FIPS); it is normalized via :func:`~dataretrieval.codes.states.to_state` to - the ``to`` representation and stored under ``into`` (the queryable this - endpoint actually filters on). It is additive sugar over the native - ``state_code`` / ``state_name`` parameters, which still accept the API's - raw values (e.g. non-US FIPS); passing ``state`` together with either - raises ``ValueError``. + """Resolve the unified ``state`` argument into an endpoint's state queryable. + + Returns the (mutated) args mapping. ``state`` is the canonical, + format-flexible parameter (full name / postal / FIPS); it is normalized via + :func:`~dataretrieval.codes.states.to_state` to the ``to`` representation + and stored under ``into`` (the queryable this endpoint actually filters on). + It is additive sugar over the native ``state_code`` / ``state_name`` + parameters, which still accept the API's raw values (e.g. non-US FIPS); + passing ``state`` together with either raises ``ValueError``. """ # Flatten ``**queryables`` first so a native state param arriving that way # (e.g. ``get_time_series_metadata``'s ``state_code``, which isn't an @@ -215,7 +215,8 @@ def get_ogc_data( pd.DataFrame or gpd.GeoDataFrame A DataFrame containing the retrieved and processed OGC data. BaseMetadata - A metadata object containing request information including URL and query time. + A metadata object with request information, including the URL and + query time. """ if output_id is None: output_id = _OUTPUT_ID_BY_SERVICE[service] @@ -296,8 +297,10 @@ def _check_profiles( def _accept_legacy_kwargs( mapping: Mapping[str, str], ) -> Callable[[Callable[..., _R]], Callable[..., _R]]: - """Decorator: accept deprecated keyword-argument names, translating them - to their modern equivalents and emitting a :class:`DeprecationWarning`. + """Accept deprecated keyword-argument names on the decorated function. + + Translates them to their modern equivalents and emits a + :class:`DeprecationWarning`. ``mapping`` maps each deprecated keyword name to the new keyword name the wrapped function expects (e.g. ``{"stateFips": "state_code"}``). When a diff --git a/dataretrieval/wateruse.py b/dataretrieval/wateruse.py index b78bdd517..5e09e3a17 100644 --- a/dataretrieval/wateruse.py +++ b/dataretrieval/wateruse.py @@ -1,20 +1,20 @@ -"""Retrieve USGS water-use data from the National Water Availability -Assessment Data Companion (NWDC). +"""Retrieve USGS water-use data from the NWDC web service. -The NWDC web services provide national-scale, USGS-modeled water-use data that -underlie the `USGS National Water Availability Assessment -`_. Estimates are served on a HUC12 -(12-digit hydrologic unit) spatial grid and can be queried for any county, -state, or hydrologic unit. This is the modern replacement for the defunct -legacy NWIS water-use service (``nwis.get_water_use``). +The National Water Availability Assessment Data Companion (NWDC) web services +provide national-scale, USGS-modeled water-use data that underlie the `USGS +National Water Availability Assessment `_. +Estimates are served on a HUC12 (12-digit hydrologic unit) spatial grid and can +be queried for any county, state, or hydrologic unit. This is the modern +replacement for the defunct legacy NWIS water-use service +(``nwis.get_water_use``). Unlike the main Water Data getters (:mod:`dataretrieval.waterdata`) and NGWMN (:mod:`dataretrieval.ngwmn`), the NWDC is a plain CSV REST service rather than an OGC API Features collection. This module supplies the NWDC-specific bits — request building, CSV parsing, the ``Link``-header cursor, and the ``{detail}`` -error envelope — and uses the service-neutral transport layer for cursor pagination, -response aggregation, client lifecycle, and sync-from-async dispatch. It follows -the same conventions: host-scoped request headers, the typed +error envelope. The service-neutral transport layer supplies cursor pagination, +response aggregation, client lifecycle, and sync-from-async dispatch. The module +follows the same conventions: host-scoped request headers, the typed :class:`~dataretrieval.exceptions.DataRetrievalError` taxonomy, and a ``(DataFrame, BaseMetadata)`` return. @@ -122,8 +122,8 @@ def get_wateruse( ``state``, ``county``, or ``huc``; results are always returned on a HUC12 grid, in a long (tidy) frame with one row per HUC12 and time step. Large areas (e.g. a whole region or a populous state) are served across multiple - pages, which this function follows transparently and concatenates into one - frame. + pages; this function follows those pages transparently and concatenates + them into one frame. Each selector also accepts a list of values. The NWDC queries one area per request, so a list is fanned out into one request per value — up to @@ -307,9 +307,12 @@ def _resolve_locations( def _as_list(value: object) -> list[Any]: - """A scalar becomes a one-element list; any non-string iterable (list, - tuple, Series, ndarray, generator) is materialized to a list. A string is - treated as a scalar so it isn't exploded into characters.""" + """Normalize a value to a list. + + A scalar becomes a one-element list; any non-string iterable (list, tuple, + Series, ndarray, generator) is materialized to a list. A string is treated + as a scalar so it isn't exploded into characters. + """ if isinstance(value, Iterable) and not isinstance(value, str): return list(value) return [value] @@ -343,12 +346,11 @@ async def _fan_out( """Fetch every request (each paginated) concurrently over one shared client. Each request is paginated by :func:`dataretrieval.transport.pagination.paginate` - with NWDC strategies: parse a CSV - page and read its ``Link`` header cursor (``parse``), follow that cursor - (``follow``), and raise the typed error carrying the NWDC ``detail`` - (``raise_for_status``). Concurrency is bounded by a semaphore at - :data:`MAX_CONCURRENT_REQUESTS`, and ``asyncio.gather`` preserves input - order, so the concatenation is deterministic. The shared + with NWDC strategies: parse a CSV page and read its ``Link`` header cursor + (``parse``), follow that cursor (``follow``), and raise the typed error + carrying the NWDC ``detail`` (``raise_for_status``). Concurrency is bounded + by a semaphore at :data:`MAX_CONCURRENT_REQUESTS`, and ``asyncio.gather`` + preserves input order, so the concatenation is deterministic. The shared :class:`httpx.AsyncClient` keeps connections alive across pages and requests. """ @@ -442,7 +444,7 @@ def _next_page_url(response: httpx.Response) -> str | None: Reads the standard ``Link: <...>; rel="next"`` header (parsed by httpx into ``response.links``). The cursor is normalized before it is trusted, because - the service spells it inconsistently: a relative reference is resolved + the service spells it inconsistently. A relative reference is resolved against the page it came from, and the bare ``water.usgs.gov`` host is rewritten to the public ``api.water.usgs.gov`` gateway (over https, whatever scheme the link used) so the follow-up request reaches the API. Only a diff --git a/dataretrieval/wqp.py b/dataretrieval/wqp.py index a64d4dbf5..b77348ea2 100644 --- a/dataretrieval/wqp.py +++ b/dataretrieval/wqp.py @@ -1,7 +1,6 @@ -""" -Tool for downloading data from the Water Quality Portal (https://waterqualitydata.us) +"""Download data from the Water Quality Portal (https://waterqualitydata.us). -See https://waterqualitydata.us/webservices_documentation for API reference +See https://waterqualitydata.us/webservices_documentation for the API reference. .. todo:: @@ -58,9 +57,11 @@ def _is_code_column(name: str) -> bool: - """True if a WQP column name denotes a code/identifier whose leading zeros - are significant and must be preserved as ``str`` (HUCs, parameter codes, - FIPS codes): the name ends with "code" or contains "identifier"/"huc"/"fips". + """Report whether a WQP column name denotes a code or identifier. + + Such columns (HUCs, parameter codes, FIPS codes) have leading zeros that + are significant and must be preserved as ``str``. A name qualifies if it + ends with "code" or contains "identifier", "huc", or "fips". """ lname = name.lower() return lname.endswith("code") or any( @@ -101,17 +102,17 @@ def get_results( Parameters ---------- ssl_check : bool, optional - Check the SSL certificate. + Whether to check the SSL certificate. Default is True. legacy : bool, optional Return the legacy WQX data profile. Default is True. dataProfile : string, optional - Specifies the data fields returned by the query. + Data fields returned by the query. WQX3.0 profiles include 'fullPhysChem', 'narrow', and 'basicPhysChem'. Legacy profiles include 'resultPhysChem', 'biological', and 'narrowResult'. For WQX3.0 queries (``legacy=False``), defaults to 'fullPhysChem'; legacy queries have no default profile. siteid : string - Monitoring location identified by agency code, a hyphen, and + Monitoring location identifier: an agency code, a hyphen, and an identification number (Example: "USGS-05586100"). statecode : string US state FIPS code (Example: Illinois is "US:17"). @@ -120,7 +121,7 @@ def get_results( huc : string Eight-digit hydrologic unit (HUC), delimited by semicolons. bBox : string - Search bounding box (Example: bBox=-92.8,44.2,-88.9,46.0) + Search bounding box (Example: bBox=-92.8,44.2,-88.9,46.0). lat : string Radial-search central latitude in WGS84 decimal degrees. long : string @@ -131,11 +132,11 @@ def get_results( Five-digit USGS parameter code, delimited by semicolons. NWIS only. startDateLo : string - Date of earliest desired data-collection activity, - expressed as 'MM-DD-YYYY' + Date of the earliest desired data-collection activity, + expressed as 'MM-DD-YYYY'. startDateHi : string - Date of last desired data-collection activity, - expressed as 'MM-DD-YYYY' + Date of the last desired data-collection activity, + expressed as 'MM-DD-YYYY'. characteristicName : string One or more case-sensitive characteristic names, separated by semicolons (https://www.waterqualitydata.us/public_srsnames/). @@ -213,7 +214,7 @@ def _what( ``service`` is the WQP service name (e.g. ``"Station"``). Services with a WQX3.0 equivalent (those in :data:`services_wqx3`) use :func:`wqx3_url` - when ``legacy=False`` and :func:`wqp_url` otherwise; legacy-only services + when ``legacy=False`` and :func:`wqp_url` otherwise. Legacy-only services route through :func:`_legacy_only_url`, which warns and falls back to the legacy profile. The CSV response is parsed via :func:`_read_wqp_csv`. """ @@ -249,11 +250,11 @@ def what_sites( Parameters ---------- ssl_check : bool, optional - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool, optional - If True, returns the legacy WQX data profile and warns the user of - the issues associated with it. If False, returns the new WQX3.0 - profile, if available. Defaults to True. + If True, return the legacy WQX data profile and warn the user about + the issues associated with it. If False, return the new WQX3.0 + profile when one is available. Defaults to True. **kwargs : optional Accepts the same parameters as :obj:`dataretrieval.wqp.get_results` @@ -296,7 +297,7 @@ def what_organizations( Parameters ---------- ssl_check : bool, optional - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool, optional Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -339,7 +340,7 @@ def what_projects( Parameters ---------- ssl_check : bool, optional - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool, optional Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -382,7 +383,7 @@ def what_activities( Parameters ---------- ssl_check : bool, optional - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool, optional Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -425,8 +426,7 @@ def what_detection_limits( legacy: bool = True, **kwargs: Any, ) -> tuple[DataFrame, WQP_Metadata]: - """Search WQP for result detection limits within a region with specific - data. + """Search WQP for result detection limits within a region with specific data. Any WQP API parameter can be passed as a keyword argument to this function. More information about the API can be found at: @@ -439,7 +439,7 @@ def what_detection_limits( Parameters ---------- ssl_check : bool - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -493,7 +493,7 @@ def what_habitat_metrics( Parameters ---------- ssl_check : bool - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -536,7 +536,7 @@ def what_project_weights( Parameters ---------- ssl_check : bool - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -589,7 +589,7 @@ def what_activity_metrics( Parameters ---------- ssl_check : bool - Check the SSL certificate. Default is True. + Whether to check the SSL certificate. Default is True. legacy : bool Return the legacy WQX data profile. Default is True. **kwargs : optional @@ -665,8 +665,7 @@ class WQP_Metadata(BaseMetadata): """ def __init__(self, response: httpx.Response, **parameters: Any) -> None: - """Generates a standard set of metadata informed by the response with specific - metadata for WQP data. + """Generate the standard metadata set, plus WQP-specific metadata. Parameters ---------- @@ -674,7 +673,7 @@ def __init__(self, response: httpx.Response, **parameters: Any) -> None: Response object from the ``httpx`` module. parameters : dict - Unpacked dictionary of the parameters supplied in the request + Unpacked dictionary of the parameters supplied in the request. """ @@ -704,7 +703,7 @@ def site_info(self) -> tuple[DataFrame, WQP_Metadata] | None: def _check_kwargs(kwargs: dict[str, Any]) -> dict[str, Any]: - """Private function to check kwargs for unsupported parameters.""" + """Check kwargs for unsupported parameters.""" mimetype = kwargs.get("mimeType") if mimetype == "geojson": raise NotImplementedError("GeoJSON not yet supported. Set 'mimeType=csv'.") @@ -748,10 +747,10 @@ def _warn_wqx3_unavailable() -> None: def _legacy_only_url(service: str, legacy: bool) -> str: """URL builder for WQP services that have no WQX3.0 equivalent. - When ``legacy=False`` is passed to one of these helpers we emit a - ``UserWarning`` explaining the fallback and *also* suppress the legacy - ``DeprecationWarning`` that ``wqp_url`` would otherwise raise — its - message claims setting ``legacy=False`` removes the warning, which is + Passing ``legacy=False`` to one of these helpers emits a ``UserWarning`` + explaining the fallback and *also* suppresses the legacy + ``DeprecationWarning`` that ``wqp_url`` would otherwise raise. That + warning's message claims setting ``legacy=False`` removes it, which is a lie for endpoints that have no WQX3.0 alternative. """ with warnings.catch_warnings(): diff --git a/docs/source/architecture/decisions/0006-service-neutral-transport.rst b/docs/source/architecture/decisions/0006-service-neutral-transport.rst index f6d9fd526..440ba5889 100644 --- a/docs/source/architecture/decisions/0006-service-neutral-transport.rst +++ b/docs/source/architecture/decisions/0006-service-neutral-transport.rst @@ -57,10 +57,10 @@ error details. OGC retains its protocol concerns: dialects, CQL2, request construction, feature shaping, URL-byte chunk planning, resumable ``ChunkedCall`` state, and typed interruption handles. Thin imports at previous private OGC and utility paths -preserve compatibility where a consumer still uses them; a path no consumer +preserve compatibility where a consumer still uses them. A path no consumer imports is deleted rather than kept as a module that exists to satisfy its own -test. Tunables are never re-exported by value: a copy taken at import time is -one a caller can patch without reaching the policy that reads it, so +test. Tunables are never re-exported by value: a caller can patch a copy taken +at import time without reaching the policy that reads it, so ``transport.retry`` is the single place they are read from. Automatic retry is enabled only on active, idempotent request paths, and only @@ -87,7 +87,7 @@ Consequences - Service-specific request and result contracts remain explicit instead of being forced into a universal adapter abstraction. - Retry can increase latency and quota consumption, so attempt counts, waits, - and total silent time remain bounded and cancellation signals are never + and total silent time remain bounded, and cancellation signals are never wrapped. - Guidance the progress reporter prints is gated on the host it applies to, so a service that cannot use an API key is not told to obtain one. diff --git a/docs/source/architecture/decisions/index.rst b/docs/source/architecture/decisions/index.rst index 94f034768..450f29669 100644 --- a/docs/source/architecture/decisions/index.rst +++ b/docs/source/architecture/decisions/index.rst @@ -6,10 +6,10 @@ cross-cutting choice was made, the trade-offs it accepts, and how compliance is checked. They complement code and API documentation rather than repeating implementation details. -Statuses are ``Proposed``, ``Accepted``, ``Superseded``, or ``Rejected``. An -accepted decision is not edited to reverse its meaning; a later ADR supersedes -it and links back to the old record. Keep records concise and commit them with -the change that makes the decision effective. +Statuses are ``Proposed``, ``Accepted``, ``Superseded``, or ``Rejected``. Do not +edit an accepted decision to reverse its meaning; a later ADR supersedes it and +links back to the old record. Keep records concise and commit them with the +change that makes the decision effective. Use :doc:`template` when proposing a decision. Number accepted and proposed records sequentially. diff --git a/docs/source/architecture/index.rst b/docs/source/architecture/index.rst index 9760caa34..3f4e30f76 100644 --- a/docs/source/architecture/index.rst +++ b/docs/source/architecture/index.rst @@ -118,18 +118,19 @@ Shared components response aggregation, progress, and sync-over-async dispatch. Internally, ``liveness`` is a stdlib-only leaf recording when data last arrived, so the page loop that observes progress and the retry loop that acts on it both - depend on it rather than on each other. It imports no service adapter or OGC - protocol module, and it is not exposed as a public framework API. + depend on ``liveness`` rather than on each other. Transport imports no + service adapter or OGC protocol module, and it is not exposed as a public + framework API. ``dataretrieval.exceptions`` - Stable error-policy leaf. It has no runtime third-party dependency and may - be imported by every service without creating an infrastructure cycle. + Stable error-policy leaf. It has no runtime third-party dependency, and + every service can import it without creating an infrastructure cycle. ``dataretrieval.utils`` Shared metadata, data-shaping helpers, ambient context support, legacy request composition, and compatibility imports for transport names that - historically lived here. New service-specific behavior should not be added - there by default. + historically lived here. By default, do not add new service-specific + behavior there. ``dataretrieval.codes`` and ``dataretrieval.rdb`` State/time-zone code conversion and RDB parsing leaves. @@ -157,7 +158,7 @@ inspect ``status_code``, ``retry_after``, and ``retryable`` without knowing the concrete subtype. OGC calls may raise ``ChunkInterrupted`` subclasses carrying a resumable call handle and completed partial state. -The public surface is defined by package/module exports and documentation. +Package/module exports and documentation define the public surface. Underscore-prefixed symbols are implementation details even where existing internal adapters currently import them; those imports are known variances, not new extension points. @@ -242,21 +243,22 @@ Resource and configuration view Seconds a call may go without receiving any data before retrying stops and the failure surfaces; defaults to 60, and ``0`` disables the bound. It complements ``API_USGS_RETRIES``, which caps attempts rather than elapsed - time: without it, four retries of a request that times out after a minute is - four silent minutes. Progress restarts the budget — a page received, or a - queued sub-request acquiring its concurrency slot — so neither a slow but - productive download nor the tail of a wide fan-out is cut short, and an - attempt already in flight is never interrupted. The first retry is never - withheld by this bound, so one slow attempt cannot disable retry by itself; - the budget decides whether to continue after that. A dead connection - therefore costs about two read timeouts rather than five attempts' worth. + time: without this bound, four retries of a request that times out after a + minute add up to four silent minutes. Progress restarts the budget — a page + received, or a queued sub-request acquiring its concurrency slot. Neither a + slow but productive download nor the tail of a wide fan-out is cut short, + and an attempt already in flight is never interrupted. This bound never + withholds the first retry, so one slow attempt cannot disable retry by + itself; after that, the budget decides whether to continue. A dead + connection therefore costs about two read timeouts rather than five + attempts' worth. ``API_USGS_PROGRESS`` Controls best-effort progress display. Reporting failures must never change retrieval results. -HTTP timeout, redirect, and authentication policy is centralized in -``dataretrieval.transport``. OGC subrequest fan-out and Water Use location +``dataretrieval.transport`` centralizes HTTP timeout, redirect, and +authentication policy. OGC subrequest fan-out and Water Use location fan-out retain separate explicit concurrency caps because their upstream costs and request shapes differ. diff --git a/docs/source/index.rst b/docs/source/index.rst index deb6c00ce..b33bf1546 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -2,11 +2,10 @@ Welcome ======= Welcome to the documentation for the Python ``dataretrieval`` package. -``dataretrieval`` is a Python alternative to the `USGS R dataRetrieval package`_ -and is used to obtain USGS and EPA water quality data, streamflow data, and -metadata directly from webservices (see the -:doc:`data portals documentation ` for additional -details about specific data sources). +``dataretrieval`` is a Python alternative to the `USGS R dataRetrieval package`_. +The package obtains USGS and EPA water quality data, streamflow data, and +metadata directly from webservices. For additional details about specific data +sources, see the :doc:`data portals documentation `. .. _USGS R dataRetrieval package: https://github.com/DOI-USGS/dataRetrieval diff --git a/docs/source/meta/contributing.rst b/docs/source/meta/contributing.rst index a0a41b411..d868a689b 100644 --- a/docs/source/meta/contributing.rst +++ b/docs/source/meta/contributing.rst @@ -2,10 +2,9 @@ Contributing ============ Contributions to ``dataretrieval`` are welcome. The repository's contributor -requirements and development commands are maintained in `CONTRIBUTING.md`_. -That file is the single source of truth for reporting issues, proposing -changes, preparing pull requests, coding standards, testing, documentation, -and release guidance. +requirements and development commands live in `CONTRIBUTING.md`_. That file is +the single source of truth for issue reports, change proposals, pull requests, +coding standards, testing, documentation, and releases. For the design constraints that apply to architecturally significant changes, see :doc:`../architecture/index` and its architecture decision records. diff --git a/docs/source/meta/installing.rst b/docs/source/meta/installing.rst index 1461fe1d1..03ccf3cd7 100644 --- a/docs/source/meta/installing.rst +++ b/docs/source/meta/installing.rst @@ -1,11 +1,11 @@ Installation Guide ================== -Whether you are a user or developer we recommend installing ``dataretrieval`` -in a virtual environment. This can be done using something like ``virtualenv`` -or ``conda``. Package dependencies are declared in ``pyproject.toml``: the core -runtime dependencies under ``[project.dependencies]``, and optional extras -(``test``, ``doc``, ``nldi``) under ``[project.optional-dependencies]``. +Whether you are a user or a developer, we recommend installing ``dataretrieval`` +in a virtual environment, using a tool such as ``virtualenv`` or ``conda``. +Package dependencies are declared in ``pyproject.toml``: the core runtime +dependencies under ``[project.dependencies]``, and optional extras (``test``, +``doc``, ``nldi``) under ``[project.optional-dependencies]``. User Installation @@ -14,7 +14,7 @@ User Installation Via ``pip``: ^^^^^^^^^^^^ To install the latest stable release of ``dataretrieval`` from `PyPI`_, run the -following commands: +following command: .. code-block:: bash @@ -26,7 +26,7 @@ following commands: Via ``conda``: ^^^^^^^^^^^^^^ To install the latest stable release of ``dataretrieval`` from the -`conda-forge channel`_, run the following commands: +`conda-forge channel`_, run the following command: .. code-block:: bash @@ -39,19 +39,18 @@ Developer Installation ---------------------- To install ``dataretrieval`` for development, we recommend first forking -the repository on GitHub. This will allow you to develop on your own -feature branch, and propose changes as pull requests to the main branch of +the repository on GitHub. Forking lets you develop on your own feature +branch and propose changes as pull requests to the main branch of the repository. -The first step is to clone your fork of the repository: +First, clone your fork of the repository: .. code-block:: bash $ git clone https://github.com/DOI-USGS/dataretrieval-python.git -Then, set the cloned repository as your current working directory in your -terminal and run the following commands to get an "editable" installation of -the package for development: +Then, make the cloned repository your working directory and run the following +command to get an "editable" installation of the package for development: .. code-block:: bash @@ -61,14 +60,14 @@ This installs ``dataretrieval`` in editable mode along with the development extras: ``test`` (test runner), ``doc`` (documentation build), and ``nldi`` (``geopandas``, required by the NLDI module). -To check your installation you can run the tests with the following commands: +To check your installation, run the tests with the following commands: .. code-block:: bash $ cd tests $ pytest -In order to fetch the latest version of ``dataretrieval``, we recommend +To fetch the latest version of ``dataretrieval``, we recommend defining the main repository as a remote `upstream` repository: .. code-block:: bash @@ -82,7 +81,7 @@ You can also build the documentation locally by running the following commands: $ cd docs $ make docs -This both tests the documentation (runs code blocks and checks links), and also -locally *builds* the documentation, placing the HTML files within the -``docs/build/html`` directory. You can then open the ``index.html`` file in -your browser to view the documentation. +These commands both test the documentation (running code blocks and checking +links) and *build* it locally, placing the HTML files within the +``docs/build/html`` directory. Open the ``index.html`` file in your browser to +view the documentation. diff --git a/docs/source/meta/license.rst b/docs/source/meta/license.rst index a24ec1d97..3e1b41277 100644 --- a/docs/source/meta/license.rst +++ b/docs/source/meta/license.rst @@ -4,8 +4,8 @@ License and Disclaimer Unless otherwise noted, this project is in the public domain in the United States because it contains materials that originally came from the United States Geological Survey, an agency of the United States Department of -Interior. For more information, see the `LICENSE.md`_ file. See the -`Disclaimer.md`_ file for more information about the disclaimer. +Interior. For more information, see the `LICENSE.md`_ file. The +`Disclaimer.md`_ file explains the disclaimer. .. _LICENSE.md: https://github.com/DOI-USGS/dataretrieval-python/blob/main/LICENSE.md diff --git a/docs/source/reference/exceptions.rst b/docs/source/reference/exceptions.rst index 1a963187f..447da2263 100644 --- a/docs/source/reference/exceptions.rst +++ b/docs/source/reference/exceptions.rst @@ -10,10 +10,10 @@ dataretrieval.exceptions Resumable chunk interruptions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -These are raised when a transparently-chunked request is interrupted -mid-stream; the completed work is preserved and ``exc.call.resume()`` continues +These exceptions are raised when a transparently-chunked request is interrupted +mid-stream. The completed work is preserved, and ``exc.call.resume()`` continues it. They are defined in ``dataretrieval.ogc.interruptions`` (they carry -pandas/httpx state) but are importable from the top level, e.g. +pandas/httpx state), but you can import them from the top level, e.g. ``from dataretrieval import ChunkInterrupted``. .. autoclass:: dataretrieval.ChunkInterrupted diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst index 13c449634..959d26750 100644 --- a/docs/source/reference/index.rst +++ b/docs/source/reference/index.rst @@ -1,4 +1,4 @@ -.. api: +.. _api: ============= API reference diff --git a/docs/source/userguide/dataportals.rst b/docs/source/userguide/dataportals.rst index 82474a167..adfa127b2 100644 --- a/docs/source/userguide/dataportals.rst +++ b/docs/source/userguide/dataportals.rst @@ -4,9 +4,8 @@ Data Portals ============ -``dataretrieval`` provides a number of functions to retrieve data from several -data portals, a table listing the portals and corresponding web addresses is -provided below. +``dataretrieval`` provides functions to retrieve data from several data +portals. The table below lists those portals and their web addresses. +-----------------------------------+---------------------------------------------------------------+ | Data Portal | Uniform Resource Locator (URL) | diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index 28da515f7..c8ed723c6 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -51,9 +51,9 @@ read-anywhere fields, so you rarely need to import the specific subclasses: Retry transient failures with backoff ===================================== -``.retryable`` and ``.retry_after`` make a backoff loop type-agnostic -- it +``.retryable`` and ``.retry_after`` make a backoff loop type-agnostic: one loop covers rate limits (429), server errors (5xx), and connection failures alike, -honoring the server's ``Retry-After`` hint when present: +and honors the server's ``Retry-After`` hint when present: .. code-block:: python @@ -75,9 +75,9 @@ Resume a large Water Data request ================================= The Water Data getters transparently split an over-large request into chunks. -When a transient failure interrupts one mid-stream, the work already completed -is preserved: catch ``ChunkInterrupted`` and call ``exc.call.resume()`` once the -condition clears -- only the unfinished sub-requests are re-issued. +When a transient failure interrupts a chunk mid-stream, the work already +completed is preserved: catch ``ChunkInterrupted`` and call ``exc.call.resume()`` +once the condition clears -- only the unfinished sub-requests are re-issued. .. code-block:: python @@ -102,15 +102,15 @@ Chunk a large request more finely By default the getters split an over-large request only as much as the server's ~8 KB URL limit forces -- the fewest sub-requests. Because each sub-request paginates, splitting a large result further costs little or no -extra quota *as long as each sub-request still spans many pages* (ten states +extra quota *as long as each sub-request still spans many pages*. (Ten states pulled as one request then page nearly as many times as ten per-state requests would; a split that leaves each sub-request only a page or two adds its partial -final page). So if you *know* your pull is large you can ask for a finer split -with ``parallel_chunks(n)`` -- trading roughly the same pages for more, smaller +final page.) So if you *know* your pull is large, ask for a finer split with +``parallel_chunks(n)``: you trade roughly the same pages for more, smaller sub-requests, which gives smoother progress, more even concurrency, and a -smaller unit of retry/resume. It is a scoped ``with`` -block, so an aggressive setting can't leak into unrelated calls and -accidentally spend quota: +smaller unit of retry/resume. ``parallel_chunks`` is a scoped ``with`` block, so +an aggressive setting can't leak into unrelated calls and accidentally spend +quota: .. code-block:: python @@ -123,15 +123,15 @@ accidentally spend quota: ``n`` is a positive integer (e.g. ``2``, ``8``, ``32``) -- the number of sub-requests to fan the call out into; a non-integer or non-positive value -raises ``ValueError`` at the ``with``. It caps the *total* sub-request count +raises ``ValueError`` at the ``with``. ``n`` caps the *total* sub-request count across every multi-value argument combined (not per argument), bounded below by what the byte limit already forces and above by how many values there are to -split, so several multi-value arguments can't multiply past it and ``n=1`` asks -for no extra fan-out. Each sub-request costs a request against your hourly rate -limit, and because how many run *at once* is capped separately by -``API_USGS_CONCURRENT`` (default 32) an ``n`` beyond that adds quota without +split. Several multi-value arguments therefore can't multiply past it, and +``n=1`` asks for no extra fan-out. Each sub-request costs a request against your +hourly rate limit. How many run *at once* is capped separately by +``API_USGS_CONCURRENT`` (default 32), so an ``n`` beyond that adds quota without adding parallelism -- the useful range is roughly ``2`` up to -``API_USGS_CONCURRENT``. There is no "off" level: simply don't enter the block +``API_USGS_CONCURRENT``. There is no "off" level: don't enter the block unless you already expect a large, multi-page result -- on a query that would have fit in a single page, extra chunks only burn quota. diff --git a/docs/source/userguide/index.rst b/docs/source/userguide/index.rst index 3cc4748a7..96ca88fcc 100644 --- a/docs/source/userguide/index.rst +++ b/docs/source/userguide/index.rst @@ -4,7 +4,7 @@ User Guide ========== -Topic guides to provide additional information about various aspects of +These topic guides provide additional information about various aspects of ``dataretrieval``. Contents diff --git a/docs/source/userguide/timeconventions.rst b/docs/source/userguide/timeconventions.rst index 03b4d8900..cfbcefea2 100644 --- a/docs/source/userguide/timeconventions.rst +++ b/docs/source/userguide/timeconventions.rst @@ -1,11 +1,11 @@ -.. timeconventions: +.. _timeconventions: Datetime Information -------------------- ``dataretrieval`` normalizes time data to UTC when converting Water Data API -responses into data frames. Timestamps are returned in the ``time`` column (the -dataframe itself uses a default integer index). For sub-daily data — such as +responses into data frames. The ``time`` column holds the timestamps; the +dataframe itself uses a default integer index. For sub-daily data — such as continuous (instantaneous) values — ``time`` is a timezone-aware ``datetime64[us, UTC]`` column. Daily values represent a whole calendar day, so their ``time`` column is timezone-naive (dates only). @@ -34,7 +34,7 @@ For continuous data, the ``time`` column holds UTC-localized pandas timestamps. Each timestamp has the format ``YYYY-MM-DD HH:MM:SS+HH:MM``. Because the values are localized to UTC, the offset (``+HH:MM``) is ``+00:00``. You can convert -them to a local timezone of your choosing with the pandas ``.dt`` accessor. +them to any local timezone with the pandas ``.dt`` accessor. .. code:: python @@ -49,9 +49,9 @@ them to a local timezone of your choosing with the pandas ``.dt`` accessor. After conversion the timestamps carry New York's offset — ``-05:00`` during standard time, or ``-04:00`` during daylight saving time, since New York is 4 -or 5 hours behind UTC depending on the time of year. Note that the first -midnight-UTC reading rolls back to the previous calendar day (``2024-02-29``) -once shifted into New York time. +or 5 hours behind UTC depending on the time of year. The first midnight-UTC +reading rolls back to the previous calendar day (``2024-02-29``) once shifted +into New York time. Daily values