Future-enhancement (deferred): async-iterator helpers for paginated CSAPI list methods — out-of-scope until upstream broadens scope

[Sam-Bolling]

⏸️ Status: DEFERRED — Filed-but-not-actionable

This issue is intentionally non-actionable at the time of filing. It documents a known future-enhancement opportunity surfaced during Phase 8 triage of #167 so that the design context, supporting research, and decision criteria are preserved if/when the time comes to implement.

Do not pick up this issue under current governance. Implementation is gated on upstream camptocamp/ogc-client adopting (or signalling intent to adopt) auto-pagination patterns. Until then, adding this to our fork would be a scope-broadening contribution that diverges from upstream's convention — exactly the class of change our governance model says to defer.

When this issue does become actionable, the body below contains everything needed to start: motivation, audit results, prior art, design sketches, API options, scope boundaries, acceptance criteria, and a links-to-context section. No re-research should be required.

---

Why this issue exists

While triaging #167 ("List methods do not document the pagination contract"), it became clear that the genuine ergonomic solution to the underlying user-experience pain point — consumers reading partial results because they don't follow next HATEOAS links — is not better documentation but a first-class auto-pagination helper that walks next links on the consumer's behalf.

#167 intentionally restricts its scope to JSDoc-only changes that surface the pagination contract to consumers reading our API. That's the right Phase 8 fix because it's:

• Spec-conformant (OGC 23-001 §7.6 delegates default page size to the server).
• Aligned with upstream's existing convention (no auto-pagination helpers anywhere in src/).
• Zero behavior change, low review-risk for PR #136.

This issue captures the next-step enhancement that would close the loop on the user-experience pain point — but only when it becomes appropriate to introduce it.

Upstream audit — confirming this is genuinely scope-broadening

Audit performed 2026-04-28 against src/ on branch phase-7 (which contains all upstream code we've inherited):

Searched src/**/*.ts for: async\s*\*, AsyncIterable, AsyncIterator, iterate, paginate, nextLink, followLink, fetchNext, allPages, getAllItems, fetchAll

Result: Zero matches in any upstream module — src/ogc-api/endpoint.ts, src/stac/endpoint.ts, src/wfs/, src/wms/, src/wmts/, src/tms/. Upstream's OgcApiEndpoint.getCollectionItems() returns a single Promise<OgcApiCollectionItem[]> representing one page; pagination link handling is documented in JSDoc but left to the consumer. STAC endpoint mentions rel="next" in JSDoc for one method but provides no walking helper. WFS/WMS/WMTS/TMS have no pagination-iterator patterns at all.

Conclusion: Adding async iterators to CSAPIQueryBuilder (or anywhere else) would establish a new pattern category in this codebase. That's the kind of architectural choice that should be made — or at minimum endorsed — by upstream maintainers, not unilaterally introduced by a contributor adding a new module.

Why we should NOT implement this now

1. No upstream precedent. See audit above. Introducing a pattern category that doesn't exist anywhere else in the codebase risks reviewer pushback (similar to #122-class concerns about opinionated CSAPI surface).
2. Spec-conformant alternative exists. Documentation alone (per #167) is sufficient for a careful consumer to implement correct pagination themselves. The library does not strictly need this helper to be usable.
3. Adds public API surface to a draft PR. PR #136 is in active review with the maintainer trimming opinionated surface area. Adding 30+ public iterator methods (one per list endpoint) is the wrong direction for that conversation.
4. Design space is large; choosing wrong is costly. Async iterators, AsyncIterable wrappers, page-callback callbacks, "fetch all" Promise<T[]>, and combinations thereof all have trade-offs (memory, abort-ability, error-handling semantics, type ergonomics with TypeScript generics). A choice made unilaterally and merged into upstream is one we live with for years.
5. Per our governance model (docs/governance/AI_OPERATIONAL_CONSTRAINTS.md): scope expansion requires a deliberate decision; "we noticed an ergonomic gap during testing" is not by itself sufficient warrant.

When this issue becomes actionable

Any one of the following triggers should re-open the conversation:

• ✅ Upstream camptocamp/ogc-client adds an auto-pagination helper to any existing module (OGC-API, STAC, etc.) — establishes the precedent we'd be aligning to.
• ✅ Upstream maintainers explicitly endorse this direction in PR #136 review or a related conversation.
• ✅ A user-facing consumer (e.g., ogc-csapi-explorer, OSHConnect-Python via WASM, or a future TypeScript-first CSAPI consumer) demonstrates that the documentation-only fix from #167 is materially insufficient for real production use, and the surfaced pain justifies a fork-specific extension.
• ✅ A future Phase decides to broaden CSAPI surface beyond strict OGC-API parity (for example, to add SensorML-aware convenience methods or domain-specific aggregations) — at which point auto-pagination becomes a sibling concern.

If none of these trigger within a reasonable horizon (e.g., 18-24 months), this issue should be closed as wontfix with a one-line rationale.

---

Design sketch — preserved for future implementation

The remainder of this issue captures research-grade design exploration so that whoever picks this up later does not have to re-derive it. Nothing below this line is binding. It is a starting point for a real design conversation, not a finished spec.

Motivating user story

A consumer integrating with a CSAPI server wants to retrieve every system whose name matches a filter:

// Today (after #167 docs land), the consumer must hand-walk the contract:
let url = builder.getSystems({ q: 'temperature' });
const all: System[] = [];
while (url) {
  const res = await fetch(url).then((r) => r.json());
  all.push(...parseSystems(res));
  const next = res.links?.find((l) => l.rel === 'next');
  url = next?.href ?? null;
}

// With this enhancement, the consumer writes:
for await (const system of builder.iterateSystems({ q: 'temperature' })) {
  // ... process each system as it streams in
}

Affected method families

All 39 public list methods on CSAPIQueryBuilder would gain an iterate* sibling. Examples:

URL-builder method	Proposed iterator sibling
getSystems(opts) → string	iterateSystems(opts) → AsyncIterable
getDataStreams(opts) → string	iterateDataStreams(opts) → AsyncIterable
getObservations(opts) → string	iterateObservations(opts) → AsyncIterable
getDeployments(opts) → string	iterateDeployments(opts) → AsyncIterable
getProcedures(opts), getProperties(opts), getControlStreams(opts), getCommands(opts), all get*History(), get*Subsystems(), get*Observations()	corresponding iterate*

Architectural question — where do iterators live?

CSAPIQueryBuilder today returns URL strings only. It does not perform HTTP requests, does not parse responses, and does not depend on fetch semantics. Adding iterate* methods to it would fundamentally change its responsibility model — from "URL builder" to "URL builder + HTTP client."

Three architecture options to evaluate when this issue activates:

Option α — Methods on CSAPIQueryBuilder directly.

• Pros: Discoverable; one-stop API surface.
• Cons: Conflates URL building with HTTP I/O; complicates testing (URL builder tests today are sync and pure); breaks symmetry with upstream OgcApiEndpoint whose URL builders are also pure.

Option β — A separate CSAPIClient (or CSAPIPager) wrapper class that takes a builder + a fetcher and exposes the iterator methods.

• Pros: Separation of concerns preserved; testable with mocked fetchers; lets builders stay pure; matches the layered pattern in upstream where high-level convenience composes lower-level URL builders.
• Cons: Slightly more verbose for consumers (new CSAPIClient(builder).iterateSystems(...)); potentially more code.

Option γ — Free functions that take a builder + endpoint URL + options.

• Pros: Smallest surface; tree-shakable; no class hierarchy to maintain.
• Cons: Less discoverable; harder to hang shared state on (e.g., per-session HTTP defaults, AbortSignal).

Recommendation when activated: start with Option β unless upstream signals otherwise.

API shape options

Shape 1 — Async generator (AsyncIterable<T>):

async function* iterateSystems(
  builder: CSAPIQueryBuilder,
  options?: QueryOptions,
  signal?: AbortSignal
): AsyncIterable<System> {
  let url = builder.getSystems(options);
  while (url) {
    const page = await fetchAndParse(url, signal);
    for (const system of page.items) yield system;
    url = page.links.find((l) => l.rel === 'next')?.href ?? null;
  }
}

Idiomatic for-await-of consumption; small runtime overhead; AbortSignal-friendly.

Shape 2 — Page-by-page iterator (AsyncIterable<Page<T>>):

Yields whole pages instead of items. Useful when consumer wants explicit per-page control (e.g., progress callbacks, batched DB writes).

Shape 3 — Eager fetchAll() returning Promise<T[]>:

Easiest mental model but accumulates everything in memory. Should NOT be the only option (footgun on large datasets) but might be a thin convenience layer over Shape 1.

Recommendation when activated: provide both Shape 1 (primary) and Shape 3 (convenience), document that Shape 3 has no memory safety net.

Cross-cutting concerns to design through

• AbortSignal: Iterators must be cancellable. Plumb AbortSignal from constructor or per-call.
• Error semantics: Mid-iteration HTTP failure → throw from the iterator? Resume from last good page? Document explicitly.
• Maximum-page guard: Pathological servers can return circular next links. Iterators should support a maxPages safety cap (default off; documented).
• Rate limiting / backoff: Out of scope initially; consumers wrap the iterator.
• TypeScript generics: Today our list methods are 39 distinct named methods, each returning a different type. Iterators must preserve that type information. Probably feasible without generics gymnastics if we keep one method per resource (rather than a single generic method).
• Link-resolution helper #110: When #110 (DEFERRED — @link resolution utilities) eventually lands, it will likely share infrastructure with this enhancement (both walk OGC HATEOAS links). Coordinate the two if both reactivate.
• Coordination with @link fallback work (#166): Iterators consume parsed objects — those objects must already correctly extract IDs from both @id and @link forms. #166 must land first.

Files this would likely touch (if implemented)

File	Action	Est. Lines
src/ogc-api/csapi/iterator.ts (new)	Implement iterator helpers	~200-400
src/ogc-api/csapi/iterator.spec.ts (new)	Unit tests with mocked fetchers	~300-500
src/ogc-api/csapi/integration/iteration.spec.ts (new)	Integration tests against fixture servers	~200
src/ogc-api/csapi/index.ts	Export new helpers	~5
src/ogc-api/csapi/model.ts	Add iterator-related types if any	~10-30
Module-level docs	Update pagination-contract section (#167) to point at iterator	~10

Scope — What to NOT touch when this activates

• ❌ Do not modify CSAPIQueryBuilder.buildQueryString() behavior. Iterators consume URLs from the builder; they do not modify it.
• ❌ Do not modify validateLimit() or any other helper.
• ❌ Do not change behavior of any existing get*() URL-building method.
• ❌ Do not modify the upstream-owned README.md.
• ❌ Do not introduce iterators on upstream modules (OGC-API, STAC, WFS, etc.) unless upstream explicitly sanctions it. This issue is scoped to CSAPI only.
• ❌ Do not implement rate-limiting, retry, or backoff logic — that's the consumer's concern.

Acceptance criteria (when this activates)

• One trigger from the "When this issue becomes actionable" section has fired and is documented in the PR description.
• Architecture option (α / β / γ) is chosen with documented rationale.
• API shape (1 / 2 / 3 or combination) is chosen with documented rationale.
• One iterator helper exists for every public list method on CSAPIQueryBuilder (current count: 39).
• Each iterator supports AbortSignal cancellation.
• Each iterator has unit tests (mocked fetcher) and at least one integration test (fixture server).
• next-link circular-reference safety mechanism is in place (e.g., maxPages guard or visited-URL tracking).
• JSDoc on each iterator clearly states pagination semantics, error behavior, and abort behavior.
• #167 JSDoc updated to point consumers at the iterator as the recommended pattern (was: "follow next link manually").
• No regression in existing CSAPI tests; no behavior change to existing methods.
• All modified files pass npx prettier --check. No lint or typecheck regressions.

---

Dependencies

Blocked by (when activating):

• A trigger from "When this issue becomes actionable" must have fired.
• #166 (@link fallback in Part 2 parsers) should land first — iterators yield parsed objects whose IDs must be correctly extracted.
• #167 (pagination JSDoc) should land first — establishes the documentation contract this enhancement implements.

Blocks: Nothing.

Related:

• #110 (DEFERRED — @link resolution utilities; shares HATEOAS-link-walking infrastructure)
• #122 (calibration on opinionated CSAPI surface concerns)
• #166, #167 (companion findings from the same Go-server interop testing effort)

---

Operational Constraints

⚠️ MANDATORY when activating: Before starting work, review docs/governance/AI_OPERATIONAL_CONSTRAINTS.md.

Key constraints:

• Precedence: OGC specifications → AI Collaboration Agreement → This issue description → Existing code → Conversational context.
• Scope expansion gate: This issue itself documents a scope-broadening enhancement. Re-confirm an activation trigger has fired before any implementation work begins. Do not pre-emptively activate.
• Minimal viable surface: When implementing, prefer the smallest API surface that satisfies the user story. Defer follow-on features (rate limiting, server-side filter caching, etc.) to separate issues.
• Upstream alignment: Once activated, audit upstream again at that time — patterns may have shifted since this issue was filed.
• No unilateral decisions on architecture: The α/β/γ and 1/2/3 choices in the design sketch above must be re-deliberated at activation time; they are not pre-decided.

Context preservation — what we knew at filing time

#	Source	What it provides
1	#167	Companion finding that motivated this enhancement; established the documentation-only Phase 8 fix this enhancement would later supplement.
2	#166	Companion finding from same Go-server interop effort; must land first so iterators yield correctly-parsed objects.
3	#110	Sibling DEFERRED enhancement on @link resolution; potential infrastructure overlap.
4	src/ogc-api/csapi/url_builder.ts	Current CSAPIQueryBuilder — pure URL-building class with no HTTP I/O. Architectural baseline iterators would extend or wrap.
5	src/ogc-api/csapi/integration/observation.spec.ts L253-267	Hand-written next-link walking pattern in our own test suite; reference implementation for the iterator's loop body.
6	src/ogc-api/csapi/integration/navigation.spec.ts L360-394	Multi-page navigation with end-of-pagination handling (nextLink === undefined); reference for termination logic.
7	src/ogc-api/endpoint.ts L501-525 (upstream)	Upstream getCollectionItems() — confirms upstream returns single-page promises; no auto-pagination.
8	src/stac/endpoint.ts L404 (upstream)	Upstream STAC mentions rel="next" in JSDoc but provides no walking helper.
9	OGC 23-001 §7.6	limit is optional with server-defined default; consumer-side next-link walking is the spec-defined pagination contract.
10	docs/governance/AI_OPERATIONAL_CONSTRAINTS.md	Governance model that requires deferring scope-broadening contributions absent upstream signal.
11	PR #136	Active upstream review where opinionated CSAPI surface is being trimmed — adds risk to introducing iterator surface unilaterally.
12	Audit (2026-04-28)	Searched src/**/*.ts for async\s*\*, AsyncIterable, iterate, getAllItems, fetchAll, paginate — zero matches in upstream modules. Recorded in #167 institutional-learning comment.

---

Filing rationale: Filing this issue now, while the context is fresh, costs ~5 minutes; reconstructing this context from scratch in 18 months when the enhancement might activate would cost much more. The issue is harmless while it sits — it's labelled deferred, the status banner makes the non-actionability obvious to any future contributor, and our governance model forbids picking it up without a documented trigger. This is a "preservation of institutional memory" filing, not a work request.