[FLINK-40204][runtime] Introduce statement-level codegen for YAML Transform expressions by haruki-830 · Pull Request #4482 · apache/flink-cdc

haruki-830 · 2026-07-22T03:48:17Z

Summary

This commit introduces statement-level code generation for Flink CDC YAML Transform expressions. Transform expressions are now represented by a GeneratedExpression intermediate structure and compiled through Janino ScriptEvaluator, allowing nullable AND / OR short-circuit logic to be generated as explicit if / else statements instead of per-record Supplier wrappers.

Key Changes

Introduce Statement-Level Transform Codegen

Added GeneratedExpression to carry generated statements, result term, and result Java type.
Added TransformExpressionEvaluator as a small wrapper around Janino ScriptEvaluator.
Updated projection and filter processors to compile and evaluate generated scripts.

Support Statement-Level Nullable AND / OR

Added translateSqlNodeToGeneratedExpression(...) in JaninoCompiler.
Generates explicit short-circuit if / else blocks for nullable AND and OR.
Ensures the right operand is evaluated only when required.

Improve Generated Script Diagnostics

Updated compile/evaluation errors and debug output to print the generated script instead of only the result term such as result$0.
Added validation at the script compilation entry to reject empty result terms before passing generated scripts to Janino.

Test Coverage

Added coverage for nullable AND / OR SQL three-valued logic.
Added short-circuit tests for:
- false AND (1 / 0 > 0)
- true OR (1 / 0 > 0)
Added projection and filter coverage.
Added assertions that generated code no longer contains new Supplier.

JIRA Reference

https://issues.apache.org/jira/browse/FLINK-40204

…nsform expressions

haruki-830 · 2026-07-22T03:51:12Z

@lvyanquan PTAL when you have time, Thanks!

Copilot

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR introduces statement-level Java code generation for Flink CDC YAML Transform expressions, switching evaluation to Janino ScriptEvaluator to enable explicit short-circuiting (if/else) for nullable AND/OR and improve diagnostics by surfacing full generated scripts.

Changes:

Add GeneratedExpression as an intermediate representation (statements + result term + result type) and generate statement-level code for transforms.
Switch expression compilation/evaluation from Janino ExpressionEvaluator to ScriptEvaluator via a small TransformExpressionEvaluator wrapper.
Update projection/filter processors and tests to assert short-circuit code generation and improved error/debug output.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java	Adds translation entry point for filter expressions to `GeneratedExpression` and wires it into projection generation.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java	Introduces statement-level codegen (`GeneratedExpressionGenerator`) and adds nullable `AND`/`OR` support without `Supplier`.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/GeneratedExpression.java	Adds the new intermediate type representing generated statements and a returnable script.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformExpressionKey.java	Stores `GeneratedExpression` and exposes compiled/full scripts; validates non-empty result term for full script.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformExpressionCompiler.java	Switches compilation to `ScriptEvaluator` and caches `TransformExpressionEvaluator`.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformExpressionEvaluator.java	Adds a wrapper around Janino `ScriptEvaluator` to evaluate compiled scripts.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilterProcessor.java	Generates/uses `GeneratedExpression` for filters and updates error output to show compiled scripts.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/ProjectionColumnProcessor.java	Uses compiled scripts for projection evaluation error reporting and `GeneratedExpression` for compilation keys.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/ProjectionColumn.java	Stores `GeneratedExpression` for projections and updates string output to include compiled scripts.
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilter.java	Updates class-level documentation to remove obsolete `scriptExpression` notes.
flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/parser/TransformParserTest.java	Updates expectations for compiled scripts and adds assertions for statement-level codegen characteristics.
flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/parser/JaninoCompilerTest.java	Adds tests compiling/evaluating generated scripts and validates empty result-term rejection.
flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperatorTest.java	Adds an end-to-end test ensuring nullable logical transforms work with statement-level codegen; updates error text expectation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

haruki-830 · 2026-07-23T09:52:54Z

+        private void translateSqlNodeToGeneratedAtoms(
+                SqlNode sqlNode, List<GeneratedExpression> atoms) {
+            if (sqlNode instanceof SqlNodeList) {
+                for (SqlNode node : (SqlNodeList) sqlNode) {
+                    translateSqlNodeToGeneratedAtoms(node, atoms);
+                }
+            } else if (sqlNode instanceof SqlIdentifier
+                    || sqlNode instanceof SqlLiteral
+                    || sqlNode instanceof SqlBasicCall
+                    || sqlNode instanceof SqlCase) {
+                atoms.add(translate(sqlNode, deduceGeneratedExpressionClass(context, sqlNode)));
+            }
+        }


Addressed in the latest commit.

lvyanquan · 2026-07-27T03:51:41Z

This change offers very limited benefits but has a relatively large impact surface. I think we need to evaluate it carefully.

CC @yuxiqian

yuxiqian

Thanks Haruki for the contribution!

I also doubt the necessity of such changes. Deducing types from SqlNode and manipulating generated code strings looks fragile. Is there any alternative solution to implement "short-circuit call"?

yuxiqian · 2026-07-27T03:57:58Z

+    public String asScript() {
+        StringBuilder script = new StringBuilder();
+        appendCode(script, code);
+        script.append("return ").append(resultTerm).append(";");
+        return script.toString();
+    }


Better avoid string manipulations on dumped Java codes. Is Java.ReturnStatement useful here?

yuxiqian · 2026-07-27T03:58:49Z

+    private static void appendCode(StringBuilder builder, String code) {
+        if (code.isEmpty()) {
+            return;
+        }
+        builder.append(code);
+        if (code.charAt(code.length() - 1) != '\n') {
+            builder.append('\n');
+        }
+    }


Ditto, avoid string ops

yuxiqian · 2026-07-27T04:03:04Z

+            if (rvalue == null) {
+                throw new ParseException("Unrecognized expression: " + sqlNode);
+            }
+            return GeneratedExpression.fromExpression(rvalue.toString(), resultClass);


rvalue.toString() is for debugging purposes only and may not generate Java code with correct parentheses.

For example, SQL statement (a - b) / (c + 1) will be converted to a - b / c + 1.

yuxiqian · 2026-07-27T05:46:56Z

+        return Object.class;
+    }
+
+    private static boolean isBooleanResultCall(SqlBasicCall sqlBasicCall) {


One may write a UDF returning BOOLEAN?

haruki-830 · 2026-07-27T06:37:15Z

Thanks @lvyanquan and @yuxiqian for the feedback.

I ran a JMH benchmark comparing the Supplier path against an equivalent if/else script and found no statistically significant difference — all configurations fall in the 330k-372k ops/ms range with overlapping confidence intervals, that means the JIT escape analysis reliably eliminates the anonymous Supplier allocation, so the per-record allocation concern is not a real bottleneck in practice.

Given the lack of measurable benefit and the correctness risks @yuxiqian identified, I agree this change is not warranted. I'll go ahead and close this PR later.

yuxiqian · 2026-07-27T11:30:18Z

Thanks @lvyanquan and @yuxiqian for the feedback.

I ran a JMH benchmark comparing the Supplier path against an equivalent if/else script and found no statistically significant difference — all configurations fall in the 330k-372k ops/ms range with overlapping confidence intervals, that means the JIT escape analysis reliably eliminates the anonymous Supplier allocation, so the per-record allocation concern is not a real bottleneck in practice.

Given the lack of measurable benefit and the correctness risks @yuxiqian identified, I agree this change is not warranted. I'll go ahead and close this PR later.

Thanks for the detailed benchmark report.

rvalue.toString() is for debugging purposes only and may not generate Java code with correct parentheses.

For example, SQL statement (a - b) / (c + 1) will be converted to a - b / c + 1.

Seems this is an existing bug, not introduced by changes in this PR. Could you please investigate this later?

haruki-830 · 2026-07-27T12:00:26Z

rvalue.toString() is for debugging purposes only and may not generate Java code with correct parentheses.
For example, SQL statement (a - b) / (c + 1) will be converted to a - b / c + 1.

Seems this is an existing bug, not introduced by changes in this PR. Could you please investigate this later?

Thanks for pointing out this bug! I confirmed that BinaryOperation.toString() does lose parentheses when a sub-expression has lower precedence than its parent operator. This affects the existing translateSqlNodeToJaninoExpression path as well.

I'll open a separate PR to fix it.

[FLINK-40204][runtime] Introduce statement-level codegen for YAML Tra…

1f9f8f6

…nsform expressions

github-actions Bot added the runtime label Jul 22, 2026

haruki-830 marked this pull request as ready for review July 22, 2026 03:51

haruki-830 marked this pull request as draft July 22, 2026 04:09

fix

1d48d38

haruki-830 force-pushed the FLINK-40204 branch from 71d3b81 to 1d48d38 Compare July 22, 2026 04:20

haruki-830 marked this pull request as ready for review July 22, 2026 04:21

fix empty generated expression script

f2281e5

lvyanquan self-assigned this Jul 22, 2026

lvyanquan requested a review from Copilot July 22, 2026 09:11

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java Outdated

Comment thread flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java

address review comments

a0aade1

haruki-830 force-pushed the FLINK-40204 branch from f95616b to a0aade1 Compare July 22, 2026 09:41

Copilot started reviewing on behalf of lvyanquan July 22, 2026 09:46 View session

lvyanquan requested a review from Copilot July 23, 2026 06:48

Copilot started reviewing on behalf of lvyanquan July 23, 2026 06:49 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

github-actions Bot added the composer label Jul 23, 2026

Fix statement-level expression codegen regressions

171ffcf

haruki-830 force-pushed the FLINK-40204 branch from 583ebd8 to 171ffcf Compare July 23, 2026 09:40

yuxiqian reviewed Jul 27, 2026

View reviewed changes

Uh oh!

Conversation

haruki-830 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

JIRA Reference

Uh oh!

haruki-830 commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

haruki-830 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

lvyanquan commented Jul 27, 2026

Uh oh!

yuxiqian left a comment

Choose a reason for hiding this comment

Uh oh!

yuxiqian Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

yuxiqian Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

yuxiqian Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

yuxiqian Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

haruki-830 commented Jul 27, 2026

Uh oh!

yuxiqian commented Jul 27, 2026

Uh oh!

haruki-830 commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

haruki-830 commented Jul 22, 2026 •

edited

Loading