[SPARK-58200][BUILD] Add a scala-3 build profile and make the version switch reversible#57343
Draft
fangchenli wants to merge 1 commit into
Draft
[SPARK-58200][BUILD] Add a scala-3 build profile and make the version switch reversible#57343fangchenli wants to merge 1 commit into
fangchenli wants to merge 1 commit into
Conversation
… switch reversible
Adds the plumbing needed to build the tree against Scala 3 (3.3.8) alongside 2.13,
using the mechanism Spark already used to cross-build 2.12/2.13: rewrite the POMs with
dev/change-scala-version.sh, then select the target with a Maven profile. Both Maven and
sbt follow, the latter through the existing -P passthrough into sbt-pom-reader.
Nothing here is expected to build the whole reactor under Scala 3 yet. The point is to
have a build target so progress can be measured module by module, which is how the
scala-2.13 profile was introduced in SPARK-29296 roughly two years before 2.13 worked.
Version handling. scala.version is now a reference (${scala213.version} or
${scala3.version}) that the switch flips, rather than a literal interpolated from
`mvn help:evaluate -Pscala-<v>`. Each full version is therefore defined once and the two
cannot drift; the previous scheme required a scala-<version> profile to exist purely to
be evaluated, and the commented-out scala-2.13 profile it read had drifted to 2.13.11.
Coordinates that must not follow scala.binary.version get their own property, which also
keeps the artifactId rewrite away from them: mvn-scalafmt (a build-time plugin with no _3
build), chill (SPARK-55152, consumed as chill_2.13 from Scala 3), and the Scala standard
library itself, whose coordinate differs between 2 and 3 (scala-library vs
scala3-library_3). The scala-3 profile keeps scala-library managed even though it no
longer names it, or a transitive 2.13 stdlib silently downgrades the one underneath
scala3-library_3.
Fixes to dev/change-scala-version.sh, most of which are latent today rather than new:
- the artifactId rewrite matched any line containing the text "artifactId", so it also
corrupted property declarations whose name merely ends in it; it is now anchored to
the <artifactId> element
- FROM_VERSION was hardcoded to 2.12, which made a switch to any newer version one-way
- the scala.binary.version pattern required a dot and so could never match a bare 3
- the scaladoc/mima branch injected -Pscala-2.13 when switching *to* 2.13, from when
2.12 was the default; 2.13 is now the default and that profile no longer exists, so
running the script corrupted dev/mima and docs/_plugins/build_api_docs.rb
genjavadoc is dropped under Scala 3 in the sbt build: it is a Scala 2 compiler plugin
published against the full Scala version, and leaving its -P: options behind makes the
Scala 3 compiler fail with "bad option".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a Scala 3 build target alongside 2.13: a scala-3 Maven profile (Scala 3.3.8) and a reversible dev/change-scala-version.sh (2.13 <-> 3). Coordinates with no Scala 3 build are pinned to their own property (chill as chill_2.13, mvn-scalafmt), and genjavadoc is disabled under Scala 3. It is not expected to compile the whole reactor yet; it is the target modules get ported against.
Why are the changes needed?
There is currently no way to build the project against Scala 3, since the cross-version tooling only targets 2.13. A Scala 3 build target is a prerequisite for porting modules and measuring progress one at a time, the same way the Scala-2.13 profile (SPARK-29296) was added well before 2.13 was actually built.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manually tested.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)