perf: fast feature-frame builds and response-body release in the chunked OGC path

NEWS.md

@@ -1,3 +1,5 @@
+**08/22/2026:** The OGC feature-frame builders take vectorized fast paths, halving the client-side CPU of a chunked call. Flat feature properties — every Water Data / NGWMN collection — build through a plain `DataFrame` constructor instead of `pd.json_normalize` (~2.6x), and all-point geometry pages build their `GeoDataFrame` through one vectorized `geopandas.points_from_xy` call instead of the per-feature Python walk in `GeoDataFrame.from_features` (~4.8x). Results are identical — a page with nested properties, or any non-point or malformed geometry, takes the previous path — and this CPU runs inside the fan-out's event loop, so trimming it also tightens chunk overlap (measured: a 93k-row, 12-chunk `get_daily` re-pull fell from 1.33 s to 0.95 s median wall; spatial frame construction alone is 4.7–4.9x faster at 10k–500k rows). The gain is larger under pandas 2, where every string column is object dtype: the nested-value check that decides the fast path is gated on `infer_dtype`, so it walks only a genuinely mixed column rather than all of them. Aggregated responses also no longer retain a page body: the response behind a getter's metadata and each per-chunk aggregate held for resume each carried one full JSON page's bytes, which on a many-chunk fan-out kept roughly the whole download resident until the call finished (~19% lower peak Python-heap in a 12-chunk replay, growing with chunk count and page size). **Behavior change:** the aggregated `httpx.Response` a completed call or `FanOutInterrupted.partial_response` carries now has an empty body — it was previously one arbitrary page's bytes, not the query's data; status, headers, URL, and elapsed are unchanged, and per-page responses are untouched. **Deprecation:** `utils.format_datetime` — orphaned since the qw services it shaped responses for were retired — now emits a `DeprecationWarning`; it will be removed on or after 2027-08-22. Combine the columns with `pandas.to_datetime` directly.
+
 **08/13/2026:** Warning categories now say what they mean. Two advisories about *upstream data* were emitted as `DeprecationWarning` and as an uncategorized `warnings.warn` respectively; both are now `DataCurrencyWarning` (a `UserWarning` subclass, exported as `dataretrieval.DataCurrencyWarning`). **Behavior change:** WQP's legacy-WQX notice moves from `DeprecationWarning` to `DataCurrencyWarning`. Because `legacy=True` is the default on every WQP getter and the notice is unconditional, a downstream project running `-W error::DeprecationWarning` previously could not call any WQP getter with default arguments; it now can. The flip side is visibility — `DeprecationWarning` is silent by default outside `__main__`, so this notice will now print to stderr for library and notebook callers who never saw it. Silence it with `warnings.filterwarnings("ignore", category=dataretrieval.DataCurrencyWarning)`, or set `legacy=False` where a WQX3.0 profile exists. The NWIS qw-endpoint retirement notice gains the same category (it previously had none, arriving as a bare `UserWarning` that deprecation filters ignored). `DeprecationWarning` now means only that a name in this package is going away, always with a replacement and, where published, a removal date; those horizons are declared once in `dataretrieval._deprecation.REMOVALS` rather than spelled at each call site.
 
 **08/11/2026:** Settings resolve through a layered chain instead of the environment alone, and a *configuration profile* is a named set of settings for **one adapter**. The new `dataretrieval.configuration` module resolves every setting in one order, highest first: a configuration passed to an active `dataretrieval.configure(...)` block, a profile that block selected, the setting's `API_USGS_*` environment variable, the adapter's `[<adapter>]` table in `~/.dataretrieval/config.toml` (or `DATARETRIEVAL_CONFIG`), the file's top-level keys, the adapter's own built-in preference, then the package default. Precedence applies **per setting**, so a file that sets only `concurrency` leaves an environment `API_USGS_PAT` in effect, and a `[ngwmn]` table still inherits every top-level key it does not name. `configure()` takes configuration objects positionally, at most one per adapter and nothing else: `configure(Configuration(api_key=vault.read("usgs/pat")), WaterdataConfiguration.load("bulk"), NgwmnConfiguration(concurrency=4))`. The adapter a configuration targets is a property of its class, so a caller never restates it, and each adapter owns its class in the module that *reads* those settings (`waterdata.WaterdataConfiguration`, `ngwmn.NgwmnConfiguration`, `nwdc.NwdcConfiguration`, `wqp.WqpConfiguration`, `nldi.NldiConfiguration`, `streamstats.StreamstatsConfiguration`) — an adapter accepts only the settings it reads, so `[streamstats] parallel_chunks = 8` is an error rather than a line that quietly does nothing. The block is delivered through a `ContextVar`, so a credential set inside it cannot leak across threads or asyncio tasks, which is what makes it safe for a server or notebook handling several users' keys and is the thing assigning to `os.environ` could never do (issue #352). The file gains named profiles beside each adapter's default profile: `[waterdata]` is always in effect, `[waterdata.bulk]` only when a caller selects it with `WaterdataConfiguration.load("bulk")`, and a selected profile still inherits the default profile and the package-wide keys per setting. A profile named in code outranks the setting's environment variable — the one place the ladder inverts the environment-above-file rule, because losing a deliberate selection to a stale shell export is what a caller would file a bug about. An adapter's configuration may also carry a `base_url`, which redirects that adapter's requests for the duration of the block — a staging instance, a mirror, a recording proxy — and no other adapter's; for Water Data one value moves the OGC collections, the Samples database, the statistics service and the STAC catalog together. It is settable in a `configure()` block only: a `base_url` key in the file and an exported `API_USGS_BASE_URL` each raise rather than being read, since a redirect a config file or a shell profile can set is one no reader of the script can see. The API key does not follow a redirect — it is scoped to the single host that honors it — and is deliberately not per-adapter: it authenticates to the gateway fronting a host, and Water Data and NGWMN share that host, one key, and one hourly quota. `dataretrieval.show_configuration()` reports each setting's effective value and where it came from, naming the profile behind each value (`configure() block [waterdata.bulk]` rather than a bare block), listing the profiles a file defines whether or not this run selected any, and naming any adapter this process has not imported rather than omitting it — without ever printing the key. One parser per setting owns its grammar, so a value means the same thing whichever source wrote it. **Breaking change:** `RetryPolicy.from_env()` is now `RetryPolicy.from_configuration()` and resolves through the whole chain rather than the environment alone. **Behavior change:** a credential-shaped keyword passed to a getter's `**kwargs` query passthrough — Water Data's `**queryables` and every WQP getter's search filters — now raises `TypeError` naming `configure(Configuration(api_key=...))` instead of putting a secret in a URL that clients, proxies and logs retain. The names refused are `api_key=`, `token=`, `x_api_key=`, `password=`, `auth=`, `pat=` and similar spellings; a filter the server actually defines is unaffected. **Bug fix:** `API_USGS_STALL_TIMEOUT` was read straight from `os.environ`, so it could not be set by a `configure()` block or the config file and never appeared in `show_configuration()`; it now resolves through the chain like every other setting. Rationale in ADRs 0009, 0010 and 0011; terms in `CONTEXT.md`.

dataretrieval/_deprecation.py

@@ -27,6 +27,7 @@
     "nwis": "2027-05-06",
     "waterdata.get_cql(service=)": "2027-08-09",
     "wateruse": "2027-08-11",
+    "utils.format_datetime": "2027-08-22",
 }
 
 

dataretrieval/combining.py

@@ -11,6 +11,13 @@
 A top-level leaf rather than part of :mod:`dataretrieval.transport`: these are
 pandas transforms over already-fetched results, with no HTTP or event-loop
 concern, consumed by chunk planning and service fan-out as well as by pagination.
+
+The module's second half adjusts already-fetched :class:`httpx.Response`
+objects in place of issuing any -- url, elapsed, headers, body release. Those
+live here rather than in transport because both the merge and the page walk
+need one spelling of each, and transport sits *above* this leaf in the layer
+stack (see the import-linter contracts), so a shared helper can only live
+below it.
 """
 
 from __future__ import annotations
@@ -72,6 +79,15 @@ def _set_response_url(response: httpx.Response, url: str | httpx.URL) -> None:
     response.request = httpx.Request(method=old.method, url=target, headers=old.headers)
 
 
+def _drop_body(response: httpx.Response) -> None:
+    """Free a response's fetched body, keeping status/headers/URL readable.
+
+    ``_content`` is the slot httpx caches a read body in; ``b""`` releases
+    the bytes while leaving the ``.content`` accessors valid.
+    """
+    response._content = b""
+
+
 def _lowest_remaining(responses: list[httpx.Response]) -> httpx.Response:
     """The response reporting the lowest ``x-ratelimit-remaining``.
 
@@ -104,13 +120,16 @@ def _merge_response(
 
     The copy's ``.headers`` are rebuilt as a fresh ``httpx.Headers`` from
     ``headers_from``, ``.elapsed`` is set to ``elapsed``, and ``.url`` is
-    overridden when ``url`` is given.  ``base`` and ``headers_from`` are never
+    overridden when ``url`` is given.  The copy's body is emptied — an
+    aggregate's content would be one arbitrary page's bytes, not the
+    combined query's data.  ``base`` and ``headers_from`` are never
     mutated, and the fresh ``httpx.Headers`` means downstream mutations don't
     back-propagate into any underlying response — so callers may re-fold
     idempotently.  This is the one low-level merge behind both pagination
     (:func:`~dataretrieval.transport.pagination.paginate`) and the chunked /
     fan-out aggregation (:func:`_combine_chunk_responses`)."""
     merged = copy.copy(base)
+    _drop_body(merged)
     merged.headers = httpx.Headers(headers_from.headers)
     merged.elapsed = elapsed
     if url is not None:

dataretrieval/ogc/shaping.py

@@ -16,6 +16,7 @@
 
 import httpx
 import pandas as pd
+from pandas.api.types import infer_dtype
 
 from dataretrieval._response_metadata import BaseMetadata
 from dataretrieval.ogc.policy import DEFAULT_DIALECT, OgcDialect
@@ -35,6 +36,10 @@
 # (EPSG:4269).
 _CRS = "EPSG:4326"
 
+# What ``infer_dtype`` reports for an object column that holds a dict among
+# its values; a homogeneous column reports its own type and cannot hide one.
+_MIXED_DTYPES = frozenset({"mixed", "mixed-integer"})
+
 # Whether geopandas is present is a static, environment-level fact, so warn
 # once here at import time rather than per query/chunk.
 if not GEOPANDAS:
@@ -99,23 +104,110 @@ def _geo_feature_frame(features: list[dict[str, Any]]) -> pd.DataFrame:
     )
 
 
+def _properties_frame(features: list[dict[str, Any]]) -> pd.DataFrame:
+    """Build the frame of feature properties, normalizing only when values nest.
+
+    Flat properties — every Water Data / NGWMN collection — build with the
+    plain ``DataFrame`` constructor; a nested value in *any* feature routes
+    the whole page through ``json_normalize`` so no row keeps a raw dict.
+    Dicts can only land in object-dtype columns, and only in one whose
+    values ``infer_dtype`` calls mixed, so the Python scan runs on those
+    alone — under pandas 2, where every string column is object dtype, that
+    is the difference between scanning one column and scanning all of them.
+    """
+    properties = [feature.get("properties") or {} for feature in features]
+    frame = pd.DataFrame(properties)
+    if any(_holds_nested_value(column) for _, column in frame.items()):
+        return pd.json_normalize(properties, sep="_")
+    return frame
+
+
+def _holds_nested_value(column: pd.Series) -> bool:
+    """Whether a built column carries a raw dict that needs flattening."""
+    if column.dtype != object:
+        return False
+    values = column.to_numpy()
+    if infer_dtype(values, skipna=True) not in _MIXED_DTYPES:
+        return False
+    return any(isinstance(value, dict) for value in values)
+
+
 def _plain_feature_frame(
     features: list[dict[str, Any]], *, include_geometry: bool
 ) -> pd.DataFrame:
     """Build a plain DataFrame from GeoJSON features."""
-    properties = [feature.get("properties") or {} for feature in features]
-    df = pd.json_normalize(properties, sep="_")
+    df = _properties_frame(features)
     df["id"] = [feature.get("id") for feature in features]
     if include_geometry:
         _attach_coordinates(df, features)
     return df
 
 
+def _point_geometries(features: list[dict[str, Any]]) -> Any:
+    """Build the geometry array for an all-2D-point page in one vectorized
+    :func:`geopandas.points_from_xy` call.
+
+    Returns ``None`` when any feature carries a non-point (or malformed)
+    geometry, so the caller falls back to ``GeoDataFrame.from_features``.
+    A feature with no geometry stays ``None`` in the result, matching the
+    fallback. Two flat x/y lists rather than coordinate pairs: the paired
+    form is ~1.6x slower.
+    """
+    nan = float("nan")
+    xs: list[Any] = []
+    ys: list[Any] = []
+    missing: list[int] = []
+    for index, feature in enumerate(features):
+        geometry = feature.get("geometry") or {}
+        xy: Any = geometry.get("coordinates")
+        if not geometry:
+            missing.append(index)
+            xy = (nan, nan)
+        elif geometry.get("type") != "Point" or not _is_pair(xy):
+            return None
+        xs.append(xy[0])
+        ys.append(xy[1])
+    try:
+        points = gpd.points_from_xy(xs, ys)
+    except (TypeError, ValueError):
+        return None
+    points[missing] = None
+    return points
+
+
+def _is_pair(value: Any) -> bool:
+    """Whether ``value`` is a two-element coordinate sequence."""
+    return isinstance(value, (list, tuple)) and len(value) == 2
+
+
+def _point_feature_frame(features: list[dict[str, Any]]) -> pd.DataFrame | None:
+    """Fast-path GeoDataFrame for an all-2D-point page, or ``None`` to fall
+    back to :func:`_geo_feature_frame`.
+
+    A ``geometry`` *property* would collide with the geometry column this
+    constructor names; no known collection has one, so that page takes the
+    slow path rather than guessing a resolution here.
+    """
+    points = _point_geometries(features)
+    if points is None:
+        return None
+    frame = _properties_frame(features)
+    if "geometry" in frame.columns:
+        return None
+    return gpd.GeoDataFrame(frame, geometry=points, crs=_CRS)
+
+
 def _spatial_feature_frame(features: list[dict[str, Any]]) -> pd.DataFrame:
     """Build a GeoDataFrame from GeoJSON features with ``id`` first."""
-    df = _geo_feature_frame(features)
+    df = _point_feature_frame(features)
+    if df is None:
+        df = _geo_feature_frame(features)
     df["id"] = [f.get("id") for f in features]
-    return df[["id"] + [col for col in df.columns if col != "id"]]
+    # Pin both names: the fast path appends ``geometry`` last and
+    # ``from_features`` emits it first, so only naming them keeps the two
+    # paths' column order identical across a chunked concat.
+    ordered = ["id", "geometry"]
+    return df[ordered + [col for col in df.columns if col not in ordered]]
 
 
 def _get_resp_data(

dataretrieval/transport/pagination.py

@@ -18,6 +18,7 @@
 from dataretrieval import progress as _progress
 from dataretrieval.combining import (
     _QUOTA_HEADER,
+    _drop_body,
     _merge_response,
     _safe_elapsed,
 )
@@ -117,6 +118,9 @@ def report_page(page: httpx.Response, frame: pd.DataFrame) -> None:
         nrows = len(frame)
         seen: set[Any] = set()
         report_page(response, frame)
+        # Parsed and never read again; free the body now rather than pinning
+        # one page of JSON for the whole walk.
+        _drop_body(initial_response)
 
         while (
             cursor is not None

dataretrieval/utils.py

@@ -17,6 +17,7 @@
 
 import pandas as pd
 
+import dataretrieval._deprecation as _deprecation
 import dataretrieval._querying as _querying
 import dataretrieval.transport.http as _transport_http
 from dataretrieval._ambient import Ambient  # noqa: F401 - compatibility re-export
@@ -59,8 +60,16 @@ def format_datetime(
     df: ``pandas.DataFrame``
         The data frame with a formatted 'datetime' column.
 
+    Deprecated: the qw services this shaped responses for are retired and
+    nothing in the package calls it. Combine the columns with
+    :func:`pandas.to_datetime` directly. See
+    :data:`dataretrieval._deprecation.REMOVALS` for the removal horizon.
     """
-    # create a datetime index from the columns in qwdata response
+    _deprecation.warn_deprecated(
+        "`utils.format_datetime`",
+        replacement="a direct `pandas.to_datetime` over the combined columns",
+        removal=_deprecation.REMOVALS["utils.format_datetime"],
+    )
     df[tz_field] = df[tz_field].map(tz)
 
     df["datetime"] = pd.to_datetime(

tests/utils_test.py

@@ -478,3 +478,12 @@ def test_retrying_get_maps_invalid_url(monkeypatch):
 
     with pytest.raises(exceptions.URLTooLong):
         _querying._get_with_retry("https://example.invalid")
+
+
+def test_format_datetime_is_deprecated_with_horizon():
+    """The orphaned qw-era helper warns with the published removal date."""
+    from dataretrieval._deprecation import REMOVALS
+
+    df = pd.DataFrame({"d": ["2020-01-01"], "t": ["12:00"], "z": ["EST"]})
+    with pytest.warns(DeprecationWarning, match=REMOVALS["utils.format_datetime"]):
+        utils.format_datetime(df, "d", "t", "z")