[improve][pip] PIP-484: Expose Window interface for WindowFunction processing

[Dream95]

Motivation

Pulsar Window Functions currently invoke WindowFunction.process(Collection<Record<X>>, ...) with all messages in the window on every trigger. Internally, WindowManager already classifies events into full, newly added, and expired lists on each activation, but WindowFunctionExecutor drops getNew() and getExpired() before calling the user function.
This makes incremental computation inefficient for sliding windows and forces users to manually diff full collections.

Modifications

This PR adds PIP-484, which proposes:

• Promote the existing internal Window<T> interface to the public API (get(), getNew(), getExpired(), timestamps).
• Remove the internal org.apache.pulsar.functions.windowing.Window type; instance code imports org.apache.pulsar.functions.api.Window directly.
• Add a new default process(Window<Record<X>>, WindowContext) overload on the existing WindowFunction<X, T> interface. The default delegates to process(window.get(), context), so lambdas and existing class implementations remain unchanged.
• Update WindowFunctionExecutor to pass the full Window object to WindowFunction.process(window, context) instead of only inputWindow.get().

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[dao-jun]

overall LGTM, just 2 points:

1. The FunctionCommon.getFunctionClassParent() change is more critical than the PIP describes — it's not about "keeping type inference correct," it's that without it, deployment will NPE (assuming doJavaChecks is fixed first). The call chain is getFunctionTypes → getFunctionClassParent → resolveInterfaceTypeArguments.
   When the parent interface is misidentified as java.util.function.Function, resolveInterfaceTypeArguments traverses the interface hierarchy and never finds Function (since IncrementalWindowFunction doesn't extend it), returns null, and typeArgsList.get(0) throws NPE. This should be documented as a hard dependency, not a
   correctness improvement.
2. Window.get() / getNew() / getExpired() return mutable List. While each trigger creates fresh lists so mutations can't corrupt internal state, once this becomes a public API the Javadoc should explicitly state whether callers are allowed to modify the returned lists.

[david-streamlio]

Review of PIP-484. Overall this is a clear, well-scoped proposal — good background, an honest backward-compat analysis, a concrete example, and a diagram; it passes the "can a reader understand it without hours of code reading" health check. The inline comments below are (1) a few required template sections that are missing and (2) some public-API design points to nail down. I've left out the two points already raised on the PR (the getFunctionClassParent NPE dependency and list mutability).

[david-streamlio]

The template requires three sections that are currently missing — could you add them, even if brief?

• Security Considerations — this is a pure API addition with no new endpoints, so a sentence confirming "no new REST/protocol surface, no new auth or multi-tenancy implications" is enough.
• Monitoring / Metrics — please state explicitly "no new metrics; runtime behavior is unchanged."
• Alternatives — the most important one. Why a new interface rather than (a) default methods on WindowFunction, (b) an overloaded process(Window, ...), or (c) a config flag? Documenting why these were rejected will pre-empt the obvious review questions. It's also the right place to defend the name IncrementalWindowFunction, since it exposes expired events too, not just increments.

[Dream95]

Option A seems better than the current approach of adding new interfaces. Let me think about whether there are any compatibility issues.

[david-streamlio]

Promoting an internal type to public API is exactly the surface the PIP process exists to scrutinize, so this shouldn't be left as an either/or ("replaced by a reference … or removed entirely"). Please commit to one approach and spell out what happens to any existing references to the old org.apache.pulsar.functions.windowing.Window (even though it's an internal package today).

[david-streamlio]

In addition to the list-mutability question already raised on this PR, please document the lifetime of the Window reference: is it valid only during the process() call, or may a user retain it across triggers? Lifetime/ownership contracts matter once this interface is public.

[david-streamlio]

The public interface is declared IncrementalWindowFunction<X, T> (X=input, T=output), but this executor field is <T, X>. This matches the internal WindowFunction<T,X> convention, so it's defensible — but the doc shows both orderings without comment, which will trip readers. A one-line note clarifying the convention would help.

[david-streamlio]

The dispatch order here is Function → IncrementalWindowFunction → WindowFunction. A user class implementing both IncrementalWindowFunction and WindowFunction (or both Function and IncrementalWindowFunction) resolves by this precedence. Since that becomes an observable public-API contract, please state the ordering explicitly and confirm it's intentional.

[david-streamlio]

Not required by the template, but reviewers usually ask: a sentence on intended test coverage (executor dispatch for each interface type, and deployment-validation acceptance of the new interface) would strengthen the proposal.

[david-streamlio]

getStartTimestamp() documents its null behavior, but getEndTimestamp()'s description implies it is never null. Please confirm and capture this in the Javadoc, since both methods are now public.

[david-streamlio]

A few PIP-process items (separate from the proposal content):

• PR title is [improve][fn], but the PIP README specifies [improve][pip] PIP-xxx: {title} for PIP PRs. Worth aligning (the semantic-PR CI may flag it).
• No PIP label is applied yet — it's needed for the PIP index/tracking query.
• Please confirm PIP-484 is still the next free number — numbers are assigned by open-PR order and can race with other in-flight PIPs.
• The Links section still has TBD for the [DISCUSS]/[VOTE] mailing-list threads — expected at this stage, just a reminder to fill them in before the vote concludes.

[Dream95]

@dao-jun @david-streamlio Thanks for the review.
I've revised PIP-484: instead of adding IncrementalWindowFunction, we now expose the public Window interface and extend WindowFunction with a default process(Window, WindowContext) overload. Existing WindowFunction users (including lambdas) remain backward compatible, and no deployment-validation changes are required.

[david-streamlio]

Thanks for the substantial revision — the pivot from a new IncrementalWindowFunction interface to a default process(Window, WindowContext) overload on the existing WindowFunction is a cleaner direction, and it resolves most of the prior round's concerns as a side effect (no new submit-time validation needed, no multi-interface dispatch precedence to document, type-param ordering now consistent). The template is now complete (Security / Monitoring / Alternatives all present) and the backward-compat + classloader analysis is thorough. I verified the executor/interface claims against current master and they hold.

Posting as COMMENTED — the remaining items are example quality plus one design tradeoff I think the mailing-list vote should explicitly ratify, not correctness defects in the proposal. Details inline.

[david-streamlio]

This example surfaces a footgun worth calling out in the prose. The abstract process(Collection, …) is what every user must implement, but once a class also overrides the process(Window, …) default, the executor only ever calls the Window overload — so the Collection override becomes dead code at runtime. Here the two overloads even return divergent results (stream().sum() vs. the counter-based running sum), which reads as if both paths are live.

Suggest either (a) having the Collection method share logic with the Window method (e.g. the Window override calls process(window.get(), context) for the base case), or (b) dropping the second override from the example — and adding one sentence: "once you override process(Window, …), the runtime no longer invokes any process(Collection, …) override."

[david-streamlio]

Minor / optional: using durable counter state (incrCounter/getCounter) to hold a window-scoped aggregate is fragile as a reference example — the counter and the window have different lifetimes, so an instance restart drifts the result (counter state persists while the window is rebuilt, or vice-versa). For teaching incremental aggregation, a local accumulator field (with the per-instance caveat noted) or returning the net delta against a maintained value would be clearer and wouldn't imply that durable state is the intended pattern.

[david-streamlio]

This is the central tradeoff of the chosen design and I'd make it more prominent (it's currently only implied here and in Alternative 2): lambda implementations cannot access the incremental views at all, since they bind to the abstract method and can't override a default. Given that lambdas are a very common window-function authoring style, this is exactly the point I'd want the mailing-list discussion/vote to ratify explicitly — Alternative 2 (WindowContext.getWindow()) would cover lambdas too, at the cost of weaker typing. No change needed to the design; just recommend stating the lambda limitation as an accepted, voted-on tradeoff rather than a footnote.

[david-streamlio]

Please confirm this wording matches how the runtime actually populates these. WindowImpl just stores constructor-injected startTimestamp/endTimestamp; the "start = end − window length" computation lives in the caller (WindowManager). Since these are now public-API contracts, worth double-checking the sliding-window case in particular (window length vs. sliding interval) so the Javadoc doesn't over-promise.

[david-streamlio]

@Dream95 The proposal reads well now and the design discussion has mostly converged here — at this point the main thing gating progress is the mailing-list [DISCUSS] thread, which is a required step in the PIP process before a vote can be called.

Could you kick that off? Roughly:

1. Send a [DISCUSS] PIP-484: Expose Window interface for WindowFunction processing email to dev@pulsar.apache.org (link back to this PR).
2. Once it's on the list, replace Mailing List discussion thread: TBD in the PIP with the lists.apache.org permalink.
3. Let it run ~a week / until comments settle, fold in any feedback, then start the [VOTE] thread and fill in that link too.

That moves the design conversation to where the binding +1s happen. The review comments I left are all polish and can be folded in during the discussion — they're not blockers. Thanks for the iterations on this!