[Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes#4441
[Feature][Flink] Define FlinkTableJob and FlinkStreamTableJob Java base classes#4441ocb3916 wants to merge 4 commits into
Conversation
- Add Java lifecycle base classes for Table API and Streaming+Table API jobs, following the same ready/handle/destroy lifecycle established by FlinkStreamingJob (sub-issue 0.1). - Use composition (getTableEnv()/getEnv()) instead of extending/mimicking TableEnvironment / StreamTableEnvironment, avoiding ~40-50 delegate methods per class that the legacy Scala traits needed. - Add JUnit 5 smoke tests covering lifecycle order, SQL bridging, and the markConvertedToDataStream() flag. Part of apache#4408 (Flink Scala-to-Java migration), Phase 0.2.
- Add Java lifecycle base classes for Table API and Streaming+Table API jobs, following the same ready/handle/destroy lifecycle established by FlinkStreamingJob (sub-issue 0.1). - Use composition (getTableEnv()/getEnv()) instead of extending/mimicking TableEnvironment / StreamTableEnvironment, avoiding ~40-50 delegate methods per class that the legacy Scala traits needed. - Add JUnit 5 smoke tests covering lifecycle order, SQL bridging, and the markConvertedToDataStream() flag. Part of apache#4408 (Flink Scala-to-Java migration), Phase 0.2.
|
|
Thank you for your valuable contribution. Currently, we plan to proceed with Stage 1 — Flink Core: 4445 , 4446 , 4447 , 4448 , migrating modules incrementally according to the proposed execution plan(Principle: no logic changes; migrate modules to Java with readable, idiomatic code). We kindly suggest following the steps outlined in the issue: flink core Thanks again for your effort and contribution. We look forward to continuing the collaboration on this migration. |



What is the purpose of the change
Implements Phase 0.2 of the Scala removal plan (#4408).
Defines
FlinkTableJobandFlinkStreamTableJobas the Java abstract base classes that willeventually replace the Scala
FlinkTableTraitandFlinkStreamTableTraittraits — for Table API(batch) jobs and jobs that mix the DataStream API with the Table API, respectively. Both classes
live in the existing
streampark-flink-shims-basemodule and follow the sameready → handle → destroylifecycle established byFlinkStreamingJobin sub-issue 0.1, so allthree "Job" base classes share a consistent shape.
Brief change log
FlinkTableJobabstract class with lifecycle:ready → handle → destroy, exposing theunderlying
TableEnvironmentviagetTableEnv()(composition) instead of extending / mimickingthe interface like the legacy trait did
FlinkStreamTableJobabstract class with the same lifecycle, exposing bothStreamExecutionEnvironment(getEnv()) andStreamTableEnvironment(getTableEnv()) viacomposition
sql(String)/sql(String, Consumer<String>)on both classes, bridging to the existingScala
FlinkSqlExecutor.executeSqlso the public API surface stays Scala-freemarkConvertedToDataStream()onFlinkStreamTableJob, replacing the legacy trait'sautomatic
Table → DataStreamconversion detection (no longer possible under composition) withan explicit call
org.apache.streampark.flink.core.javaapirather than theissue-specified
org.apache.streampark.flink.core.java— see note belowgetTableEnv()/getEnv()composition, SQL execution, and the
markConvertedToDataStream()flagAbstractFlinkJob(package-private) — commonready → handle → execute → destroylifecycle,sql(...), andapp.namelookup shared by both job classesFlinkJobSupport— SQL callback bridging (JavaConsumer<-> ScalaFunction1)JobTestParams— sharedParameterToolbuilder used by both test classesSystem.out.printlnwith SLF4J logging in bothexecute(...)overridesA note on the package name
The issue body specifies
org.apache.streampark.flink.core.java. That name breaks Scalacompilation of every sibling file in
streampark-flink-shims-basethat referencesjava.util/java.io/java.lang— a nested package literally namedjavashadows the realjava.*standard library for any Scala file sharing the enclosing package. Reproduced both waysin a clean build:
org.apache.streampark.flink.core.java→BUILD FAILURE, 33 errors, alljava.util/java.io/java.langunresolved in unrelated sibling.scalafilesorg.apache.streampark.flink.core.javaapi→BUILD SUCCESS, no other files affectedWent with
javaapifor this PR. Open to other naming suggestions ifjavaapiisn't preferred.Design note: composition instead of full delegation
The legacy traits extend/mimic Flink's
TableEnvironment/StreamTableEnvironmentdirectly,re-implementing ~40-50 delegate methods each (
FlinkStreamTableTraitadditionally needed a$-prefix naming hack to avoid clashes between the two APIs). The new classes hold theenvironment(s) as fields instead, exposed via
getTableEnv()/getEnv(). This removes thedelegate boilerplate and the naming hack, and means the classes won't go stale as the Flink Table
API changes — but it is a breaking change for code that currently calls Table API methods
directly on a trait instance (e.g.
job.sqlQuery(...)→job.getTableEnv().sqlQuery(...)).Flagging for discussion since the Phase 11 migration guide will need to cover it.
Verifying this change
mvn test -pl streampark-flink/streampark-flink-shims/streampark-flink-shims-baseTests run: 9, Failures: 0, Errors: 0
Does this pull request potentially affect one of the following parts?
Documentation