Regenerate derived columns when transforms change

[Vamsi-klu]

Fixes #9989

What changed

This PR persists derived-column transform function metadata into segment column metadata and uses that metadata during default-column reload decisions.

The implementation adds a transformFunction column metadata property and exposes it through ColumnMetadata. Segment generation now records the transform function associated with a generated column, and default-column creation records the transform function for auto-generated derived columns as well.

During reload, BaseDefaultColumnHandler now compares the transform function stored in segment metadata with the transform function currently configured for the column. If the value changed, or if a legacy auto-generated derived column has no stored transform metadata but the table config now declares a transform, Pinot regenerates the column instead of leaving stale derived values in the segment.

The default-value fallback path is covered as well: if a derived column's transform argument is missing from the segment and Pinot has to materialize the default value instead of evaluated values, the transform function is still persisted in metadata. That prevents the segment from repeatedly looking like it has no transform metadata on subsequent reloads.

Why it matters

Derived columns are produced from transform expressions. If a table config changes the expression, reloaded segments need to reflect the new expression for auto-generated derived columns. Without tracking the expression in segment metadata, Pinot can know that the column exists, but not whether it was created from the current transform function.

That creates a correctness risk: a segment can continue serving values computed by an older transform even after the table config has been updated. Persisting and comparing the transform function gives reload a stable, segment-local way to decide whether an auto-generated derived column must be rebuilt.

The legacy metadata handling matters because existing segments created before this metadata existed will naturally have no transform function recorded. Those segments should not be permanently stuck with stale values once they are reloaded under a table config that defines a transform for the auto-generated column.

Impact

Auto-generated derived columns are regenerated when their transform function changes.

Legacy auto-generated derived columns that do not yet have stored transform metadata are brought forward on reload and receive the current transform metadata.

Segment-generation metadata now includes transform-function provenance for generated columns, which improves observability and future reload decisions.

The metadata API remains backward compatible: ColumnMetadata#getTransformFunction() defaults to null, and older segment metadata without the property still loads successfully.

Testing

• ./mvnw -Dmaven.repo.local=/tmp/m2-pinot-9989 -pl pinot-segment-spi,pinot-segment-local -am -Dtest=ColumnMetadataImplTest,DefaultColumnHandlerTest -Dsurefire.failIfNoSpecifiedTests=false test
• ./mvnw -Dmaven.repo.local=/tmp/m2-pinot-9989 -pl pinot-segment-spi,pinot-segment-local spotless:apply checkstyle:check license:check
• git diff --check

The tests cover metadata round-trip, missing metadata backward compatibility, transform-function action-map decisions, regeneration when a legacy auto-generated derived column lacks stored transform metadata, and persistence of transform metadata when Pinot falls back to default values because a transform argument is absent.

---

Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before marking ready for review

[Vamsi-klu]

Review request: @Jackie-Jiang

GitHub did not allow this account/integration to add requested reviewers through the reviewer section, so I am tagging the relevant reviewer here for visibility.

This PR fixes #9989 by persisting derived-column transform-function metadata into segment column metadata and using that metadata during reload to regenerate auto-generated derived columns when the configured transform changes. Review would be especially useful because the implementation follows the issue guidance to persist the transform function and compare it against the table config during reload.

---

Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before posting