feat(build): bump Scala/Java services to Java 17 LTS by bobbai00 · Pull Request #4938 · apache/texera

bobbai00 · 2026-05-05T04:00:13Z

What changes were proposed in this PR?

Bump all Scala/Java microservices from Java 11 to Java 17 LTS, and
centralize the JDK 17 --add-opens flags through .jvmopts as a
single source of truth.

Toolchain bumps

8 Dockerfiles under bin/: build base
sbtscala/scala-sbt:eclipse-temurin-jammy-11.0.17_8_1.9.3_2.13.11
→ ...-17.0.5_8_1.9.3_2.13.11 (same sbt 1.9.3 + Scala 2.13.11);
runtime eclipse-temurin:11-{jdk,jre}-jammy → 17-{jdk,jre}-jammy.
.github/workflows/build.yml and build-and-push-images.yml:
java-version: 11 → 17 (matrix entries + 5 setup-java steps).
AGENTS.md: toolchain table.

.jvmopts as single source of truth

.jvmopts lists the seven --add-opens flags JDK 17 needs for the
runtime (Pekko, Apache Arrow, Ehcache SizeOf, Pekko Kryo
serialization). The same file reaches every JVM the build launches:

sbt's own JVM: sbt's launcher reads .jvmopts directly.
forked test JVMs (sbt-jacoco): ThisBuild / Test / javaOptions ++= JdkOptions.jvmFlags(...) in build.sbt.
sbt-native-packager bin/<svc> launchers in production dists:
Universal / javaOptions ++= JdkOptions.jvmFlags(...).map("-J" + _),
bundled per-project via asfLicensingSettings (Universal does not
cascade from ThisBuild).
IntelliJ run configs .run/ComputingUnit{Master,Worker}.run.xml:
VM_PARAMETERS=@$PROJECT_DIR$/.jvmopts (JDK 9+ argfile expansion).

project/JdkOptions.scala (modeled on Pekko's) reads the file once
and exposes a Seq[String] for the build to wire up.

Any related issues, documentation, discussions?

Closes #4937. Refs discussion #4001.

How was this PR tested?

CI's amber job now finishes in
single-digit minutes, with all 5 jacoco tasks passing. Local
verification: sbt show ThisBuild / Test / javaOptions and
sbt show <project> / Universal / javaOptions confirm flag
propagation across all subprojects; java @.jvmopts -version
confirms the JDK accepts the argfile cleanly; running amber
jacoco locally hits zero InaccessibleObjectException.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.7)

codecov-commenter · 2026-05-05T04:02:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.71%. Comparing base (2652315) to head (c3c5414).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4938      +/-   ##
============================================
- Coverage     42.72%   42.71%   -0.02%     
+ Complexity     2185     2184       -1     
============================================
  Files          1031     1031              
  Lines         38152    38152              
  Branches       4004     4004              
============================================
- Hits          16302    16298       -4     
- Misses        20831    20836       +5     
+ Partials       1019     1018       -1

Flag	Coverage Δ
access-control-service	`39.53% <ø> (ø)`
agent-service	`33.72% <ø> (ø)`
amber	`43.20% <ø> (-0.02%)`	⬇️
computing-unit-managing-service	`0.00% <ø> (ø)`
config-service	`0.00% <ø> (ø)`
file-service	`32.18% <ø> (ø)`
frontend	`33.08% <ø> (ø)`
python	`88.87% <ø> (-0.03%)`	⬇️
workflow-compiling-service	`47.72% <ø> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

aglinxinyuan

Do we have to include it as part of the release? If it's not required, let's not include it in the release and do more testing.

The discussion also mentioned that we need to have JDK_JAVA_OPTIONS. Is that still required? I don't see it anywhere in this PR.

### What changes were proposed in this PR? Backport #4549 (initial AGENTS.md + CLAUDE.md) and #4825 (AGENTS.md rewrite) to `release/v1.1.0-incubating`. Two cherry-picks: #4549 applies clean; #4825 has trivial content conflicts that resolve by taking the post-rewrite state (i.e. `upstream/main:AGENTS.md` verbatim — what the squashed result of both PRs would produce on main). ### Any related issues, documentation, discussions? Backport of #4549 and #4825. Unblocks the auto-backport simulation for #4938 (Java 17 bump) and any future PR that touches `AGENTS.md`. ### How was this PR tested? Doc-only change. Verified `diff <(git show HEAD:AGENTS.md) <(git show upstream/main:AGENTS.md)` is empty. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) --------- Co-authored-by: Xinyuan Lin <xinyual3@uci.edu> Co-authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bobbai00 · 2026-05-05T04:22:22Z

add ENV JDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED". This is required by Apache Arrow Java under JDK 17+ (see https://arrow.apache.org/docs/java/install.html). The other 6 service images don't exercise Arrow off-heap memory.

Got it. I mentioned it in the PR description and AGENT.md

The main reason I want to include it in the release branch is because Java 11 already doesn't have an LTS version. Since we use Java as the runtime image, it would be great to have an LTS base image as the runtime.

Yicong-Huang · 2026-05-05T04:23:53Z

add ENV JDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED". This is required by Apache Arrow Java under JDK 17+ (see https://arrow.apache.org/docs/java/install.html). The other 6 service images don't exercise Arrow off-heap memory.

Got it. I mentioned it in the PR description and AGENT.md

The main reason I want to include it in the release branch is because Java 11 already doesn't have an LTS version. Since we use Java as the runtime image, it would be great to have an LTS base image as the runtime.

Java 11 is extended to be supported till 2032. https://www.oracle.com/java/technologies/java-se-support-roadmap.html

Do we HAVE to upgrade before release??

aglinxinyuan · 2026-05-05T04:25:35Z

add ENV JDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED". This is required by Apache Arrow Java under JDK 17+ (see https://arrow.apache.org/docs/java/install.html). The other 6 service images don't exercise Arrow off-heap memory.

Got it. I mentioned it in the PR description and AGENT.md

The main reason I want to include it in the release branch is because Java 11 already doesn't have an LTS version. Since we use Java as the runtime image, it would be great to have an LTS base image as the runtime.

Sorry, I missed the JDK_JAVA_OPTIONS part.

bobbai00 · 2026-05-05T04:41:40Z

add ENV JDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED". This is required by Apache Arrow Java under JDK 17+ (see https://arrow.apache.org/docs/java/install.html). The other 6 service images don't exercise Arrow off-heap memory.

Got it. I mentioned it in the PR description and AGENT.md
The main reason I want to include it in the release branch is because Java 11 already doesn't have an LTS version. Since we use Java as the runtime image, it would be great to have an LTS base image as the runtime.

Java 11 is extended to be supported till 2032. https://www.oracle.com/java/technologies/java-se-support-roadmap.html

Do we HAVE to upgrade before release??

We don't have to. I removed the tag.

Yicong-Huang · 2026-05-05T05:09:02Z

How about bash script in bin/? do they need to be updated for JDK_JAVA_OPTIONS?

aglinxinyuan

Do you want to add that to the IntelliJ configuration in the .run folder so developers can also run with JDK_JAVA_OPTIONS on their local machines? Please also don't solely rely on test cases for this big change. Let's run a real workflow to manually test it.

bobbai00 · 2026-05-05T05:27:46Z

Do you want to add that to the IntelliJ configuration in the .run folder so developers can also run with JDK_JAVA_OPTIONS on their local machines? Please also don't solely rely on test cases for this big change. Let's run a real workflow to manually test it.

Added

bobbai00 · 2026-05-05T05:29:05Z

Do you want to add that to the IntelliJ configuration in the .run folder so developers can also run with JDK_JAVA_OPTIONS on their local machines? Please also don't solely rely on test cases for this big change. Let's run a real workflow to manually test it.

During my daily development, I use Java 17 all the time and manually ran lots of workflows. So I am pretty confident with the changes.

To be safe, can you try this branch and manually run some workflows to test it?

Forking changed the test JVM's CWD to the subproject directory, so tests reading repo-root-relative paths (e.g. MockTexeraDB -> sql/texera_ddl.sql) failed with NoSuchFileException. Set ThisBuild / Test / baseDirectory back to the build root.

Earlier commits introduced Test / fork := true and a JdkOptions plugin to inject JDK 17 --add-opens flags. Forking changed the test JVM's CWD to each subproject directory, which broke tests that read repo-root-relative paths (sql/texera_ddl.sql from MockTexeraDB), and required a baseDirectory pin to fix. Replace all of that with a single .jvmopts at the repo root: sbt's launcher reads it and applies the flags to sbt's own JVM startup. With Test / fork at its default (false), tests run in sbt's JVM and inherit the flags — no fork overhead, no CWD change, no baseDirectory pin, no sbt plugin file. Drops: - project/JdkOptions.scala - ThisBuild / Test / fork := true - ThisBuild / Test / javaOptions ++= ... - ThisBuild / Test / baseDirectory := ... Adds: - .jvmopts (5 --add-opens lines, attribution comments)

# Conflicts: # bin/computing-unit-master.dockerfile # bin/computing-unit-worker.dockerfile

Yicong-Huang · 2026-05-06T08:35:14Z

The amber run again hangs and drains our CI. canceled manually. please check https://github.com/apache/texera/actions/runs/25420608272/job/74561719299?pr=4938 @bobbai00 @aglinxinyuan

…telliJ The amber CI job hung at the 6h timeout because sbt-jacoco's forked test JVM did not inherit .jvmopts (which is read only by sbt's own launcher). On JDK 17, Pekko's serializer reflected on java.lang.invoke.SerializedLambda and threw InaccessibleObjectException; tests waited on the resulting unresolved future and never exited. Make .jvmopts the single source of truth, consumed by: - sbt's own JVM (already) - forked test JVMs (ThisBuild / Test / javaOptions) - sbt-native-packager bin/<svc> launchers (per-project Universal / javaOptions, bundled into asfLicensingSettings since the Universal config does not propagate from ThisBuild) - IntelliJ Application run configs (VM_PARAMETERS = @\$PROJECT_DIR\$/ .jvmopts; JDK 9+ argfile expansion at JVM start) Also adds the missing --add-opens=java.base/java.lang.invoke flag. Drops the partial JDK_JAVA_OPTIONS env in ComputingUnitMaster / ComputingUnitWorker run configs (had only 3 of 6 flags; redundant once VM_PARAMETERS points at .jvmopts). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…figs These were dropped from the committed XMLs at some point during the JDK 17 bump work. ASF source files require the header. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CheckpointSpec failed on JDK 17 because Kryo (via altoo pekko-kryo) reflects on AtomicBoolean#value when serializing WorkflowExecutionCoordinator.completionNotified. Without the open, ScalaTest reports the InaccessibleObjectException as a fast test failure rather than a hang. Single-line edit to .jvmopts; the previously committed plumbing (ThisBuild / Test / javaOptions, per-project Universal / javaOptions, .run/*.xml VM_PARAMETERS = @\$PROJECT_DIR\$/.jvmopts) propagates the new flag to every JVM the build launches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Apply agent-style passes to the comments added in this PR: - build.sbt and project/JdkOptions.scala: drop passive voice and column-aligned bullets; keep the why (Universal does not cascade from ThisBuild) front-and-center. - .jvmopts: replace one em-dash with a period. - AGENTS.md: replace the outdated three-flag manual export snippet with a pointer to .jvmopts as the canonical list, and to project/JdkOptions.scala as the propagation glue. Mention the java @.jvmopts -jar argfile form for raw-java launches. No flag-list change; the JVM-side behavior is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The texera.amber → texera.amber.main rename is an unrelated IntelliJ sbt-import naming change that rode along with the JDK 17 fix commit. Revert to match origin/main; the rename will land in a follow-up PR covering all eight .run/*.xml files together. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…flags Without this, sbt running inside the Docker build sees no .jvmopts at the build root, so JdkOptions.jvmFlags returns Seq.empty, Universal / javaOptions ends up empty, and sbt-native-packager generates bin/<svc> launchers with no --add-opens flags. The runtime container then starts the JVM without strong-encapsulation opens, and any Pekko / Kryo / Arrow reflection throws InaccessibleObjectException. Add COPY .jvmopts .jvmopts to all eight Scala dockerfiles, right after COPY build.sbt. The runtime stage does not need it; the launcher script bakes the flags in at build time. Also rewrite the JDK 17 paragraph in AGENTS.md to align with .jvmopts: the file holds every flag Texera needs, with each group annotated by its upstream source (Kryo, Apache Arrow, Apache Pekko). Drops the Ehcache reference (its required flags are already covered by the Kryo and Pekko documentation). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

"Adding a missing flag" read as if a flag were currently missing. Reframe as future-tense: when a library upgrade or new code path triggers InaccessibleObjectException, add the open to .jvmopts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bobbai00 · 2026-05-09T04:34:19Z

@Yicong-Huang @aglinxinyuan can you take a look at this PR again ?

Yicong-Huang

I think we can merge it. There will be issues for post fixing.

github-actions Bot added feature ci changes related to CI docs Changes related to documentations dev labels May 5, 2026

github-actions Bot assigned bobbai00 May 5, 2026

bobbai00 force-pushed the feat/bump-java-17 branch from 25ae13f to 64ffded Compare May 5, 2026 04:05

bobbai00 requested review from Yicong-Huang and aglinxinyuan May 5, 2026 04:05

bobbai00 added the release/v1.1.0-incubating back porting to release/v1.1.0-incubating label May 5, 2026

bobbai00 mentioned this pull request May 5, 2026

docs(1.1): backport AGENTS.md #4939

Merged

aglinxinyuan requested changes May 5, 2026

View reviewed changes

bobbai00 force-pushed the feat/bump-java-17 branch from 64ffded to c16f305 Compare May 5, 2026 04:22

bobbai00 force-pushed the feat/bump-java-17 branch from c16f305 to 18f8813 Compare May 5, 2026 04:23

bobbai00 force-pushed the feat/bump-java-17 branch 2 times, most recently from fc43932 to d7b171c Compare May 5, 2026 04:37

bobbai00 removed the release/v1.1.0-incubating back porting to release/v1.1.0-incubating label May 5, 2026

bobbai00 requested a review from aglinxinyuan May 5, 2026 04:41

bobbai00 mentioned this pull request May 5, 2026

feat(docker): run service containers as non-root user #4941

Merged

Yicong-Huang reviewed May 5, 2026

View reviewed changes

Comment thread AGENTS.md Outdated

bobbai00 force-pushed the feat/bump-java-17 branch from 3563991 to 40e94aa Compare May 5, 2026 05:23

aglinxinyuan requested changes May 5, 2026

View reviewed changes

Merge branch 'main' into feat/bump-java-17

04afce7

github-actions Bot removed the engine label May 5, 2026

bobbai00 added 2 commits May 5, 2026 16:47

github-actions Bot removed dependencies Pull requests that update a dependency file common labels May 5, 2026

bobbai00 and others added 2 commits May 5, 2026 21:36

Merge branch 'main' into feat/bump-java-17

268ff38

# Conflicts: # bin/computing-unit-master.dockerfile # bin/computing-unit-worker.dockerfile

Merge branch 'main' into feat/bump-java-17

b898b48

aglinxinyuan reviewed May 6, 2026

View reviewed changes

Comment thread .jvmopts Outdated

aglinxinyuan reviewed May 6, 2026

View reviewed changes

Comment thread .jvmopts Outdated

aglinxinyuan reviewed May 6, 2026

View reviewed changes

Comment thread .jvmopts Outdated

add asf header

8d74587

bobbai00 and others added 4 commits May 7, 2026 20:41

Merge branch 'main' into feat/bump-java-17

010f622

fix: restore ASF license header on ComputingUnitMaster/Worker run con…

f262ddb

…figs These were dropped from the committed XMLs at some point during the JDK 17 bump work. ASF source files require the header. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge branch 'main' into feat/bump-java-17

bb7ee3d

github-actions Bot added dependencies Pull requests that update a dependency file common labels May 8, 2026

bobbairobot and others added 8 commits May 8, 2026 17:13

Merge branch 'main' into feat/bump-java-17

588723c

simplify comments and jvmopts

aceae40

improve agents.md

c3c5414

Yicong-Huang approved these changes May 9, 2026

View reviewed changes

bobbai00 merged commit 75ff07e into apache:main May 9, 2026
26 checks passed

Conversation

bobbai00 commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

codecov-commenter commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aglinxinyuan left a comment

Choose a reason for hiding this comment

Uh oh!

bobbai00 commented May 5, 2026

Uh oh!

Yicong-Huang commented May 5, 2026

Uh oh!

aglinxinyuan commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bobbai00 commented May 5, 2026

Uh oh!

Yicong-Huang commented May 5, 2026

Uh oh!

Uh oh!

aglinxinyuan left a comment

Choose a reason for hiding this comment

Uh oh!

bobbai00 commented May 5, 2026

Uh oh!

bobbai00 commented May 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Yicong-Huang commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bobbai00 commented May 9, 2026

Uh oh!

Yicong-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bobbai00 commented May 5, 2026 •

edited

Loading

codecov-commenter commented May 5, 2026 •

edited

Loading

aglinxinyuan commented May 5, 2026 •

edited

Loading

Yicong-Huang commented May 6, 2026 •

edited

Loading