chore: sync spark4.1 with master - #2645
Merged
Rana Singh (ranadeepsingh) merged 100 commits intoAug 17, 2026
Merged
Conversation
Add a ReleaseBranchCompat job that runs on every PR to master. It rebases each release branch (starting with spark4.0) onto the PR HEAD and runs sbt compile test:compile to catch breakage before it lands in master. - Non-blocking (continueOnError: true) - Matrix-based for easy expansion to more release branches - Reports merge conflicts and compile failures as warnings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ease demo image (#2557) Addresses MSRC case 110886 / incident 31000000570827. The mmlspark/release image (built from tools/docker/demo/Dockerfile) ships Spark 3.5.4, which pins netty 4.1.96.Final. That version is flagged for multiple CVEs (CVE-2023-44487, CVE-2024-29025, CVE-2025-24970, ...). Spark has not bumped netty in any 3.5.x release. netty 4.1.x is binary-compatible, so we replace all netty-*-4.1.96.Final*.jar files in /opt/spark/jars/ with 4.1.118.Final right after the Spark extract. This includes netty-codec-http2 (the specific artifact named by the finder). Also removes 'pyspark' from the conda install line. It was pulling a complete second Spark install (PySpark 4.0.1) into /usr/local/lib/python*/site-packages/pyspark/ that nothing in the demo image actually used (SPARK_HOME points at /opt/spark) and that doubled the surface area scanners report on. Validated locally: - /opt/spark/jars/netty-*-4.1.96.Final*.jar: 0 matches after build - /opt/spark/jars/netty-*-4.1.118.Final*.jar: full set present - /usr/local/lib/.../pyspark: no longer exists - spark-submit --version: works - spark.range(5).count(): returns 5 Jetty (shaded inside hadoop-client-runtime-3.3.4.jar at 9.4.43) is OUT OF SCOPE for this PR; that requires a Spark/Hadoop swap and will be tracked separately.
* chore: add SynapseML local setup skill ## Summary Add a project-scoped SynapseML agent skill that diagnoses local toolchain state, selects JDK 11 for SBT commands, runs a safe local Spark smoke test, and flags live-service tests before agents run them. ## Prompting Intent The engineer asked the agent to create a skill that helps any future agent get SynapseML working locally after the PR 2556 review exposed a local Java 21 and Scala 2.12 compiler-bridge failure. The engineer also asked to create a PR for the skill addition before continuing the original external PR review. ## Linked Sources - User request in current session: create a skill that will help any agent be able to get SynapseML working locally. - Follow-up user request in current session: create a PR for that skill addition and continue using it to review PR 2556. - Existing project-scoped skill convention: .agents/skills/code-review/SKILL.md. - Local validation output: doctor_status=ok, JDK 11 dry-run selected JAVA_HOME, smoke test passed, Azure Search tests flagged review_required. ## Rationale A project-scoped SynapseML skill keeps local setup guidance with the repository where future agents need it. The scripts use explicit parameters rather than session state, force JDK 11 for Scala 2.12 SBT commands, and include a live-service guard so agents do not accidentally create or delete Azure Search resources while validating changes. * chore: move SynapseML setup skill to Copilot path ## Summary Move the SynapseML local setup skill from `.agents/skills/` to `.github/skills/` so it uses the documented Copilot project-skill discovery path. ## Prompting Intent The engineer asked whether the `.agents` folder was correct and whether Copilot would pick it up. Investigation found that the local skill-authoring reference documents `.github/skills/<name>/` and `.claude/skills/<name>/` as project skill locations, so the open skill PR needed a path correction. ## Linked Sources - User question in current session: is this .agent folder correct? will copilot pick this up? - Skill-authoring reference: /home/brwals/.copilot/installed-plugins/copilot-toolkit-marketplace/common/skills/create-skill/references/REFERENCE.md - Existing PR: #2558 ## Rationale The existing `.agents/skills/code-review` directory was only evidence of a repo-local convention, not evidence of Copilot discovery. Moving the new skill to `.github/skills/synapseml-local-setup/` keeps the same skill content while placing it in the documented project-skill path.
#2560) * Add v1 OpenAI Endpoint support and remove legacy completions API * Fix FuzzingUnitTest * Add test to increase code coverage * Make v1 api assumption cleaner * Add OpenAICompletion deprecation * Remove deprecation warnings * Fix RAI test for OpenAIPrompt * Revert "Add OpenAICompletion deprecation" This reverts commit fa708e2. * Revert "Fix RAI test for OpenAIPrompt" This reverts commit 3ed6044. * Revert "Remove deprecation warnings" This reverts commit 9a40c5c. * Reapply "Remove deprecation warnings" This reverts commit 987484c. * Reapply "Fix RAI test for OpenAIPrompt" This reverts commit f06f1ad. * Reapply "Add OpenAICompletion deprecation" This reverts commit 10715cd.
## Summary Move the remaining SynapseML repo skill from `.agents/skills/` to `.github/skills/` so Copilot CLI can discover all repo-versioned skills from the documented project-skill path. Add README pointers under `.agents/` for tools or agents that inspect the older convention. ## Prompting Intent The engineer asked to migrate everything to the correct Copilot CLI path and suggested keeping a generic agents pointer. The goal was to make existing skills discoverable by Copilot while avoiding future confusion about `.agents/skills`. ## Linked Sources - User request in current session: migrate everything to the correct path for Copilot CLI and keep a generic agents pointer. - Skill location reference: /home/brwals/.copilot/installed-plugins/copilot-toolkit-marketplace/common/skills/create-skill/references/REFERENCE.md - Prior merged skill PR: #2558 ## Rationale `.github/skills/<name>/` is the documented Copilot CLI project-skill location. Keeping only README pointers under `.agents/` preserves a breadcrumb for other agent conventions without leaving duplicate or stale `SKILL.md` files in a path Copilot CLI may not load.
* add speechtotextsdk improvements * Fix ffmpeg output args * add ffmpeg url check * fix: address speech recording review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: make OpenAIPrompt RAI test resilient Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "test: make OpenAIPrompt RAI test resilient" This reverts commit fccce86. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: remove Acrolinx integration config AB#5391146 AB#5391147 ## Summary Remove the retired Acrolinx repository configuration from SynapseML and add the Feature Registry pointer and repo-specific design notes for Feature 5391136. ## Prompting Intent Engineer asked the agent to complete the Acrolinx removal request from the Microsoft Learn authoring tools PM. The repository cleanup needed to remove stale source-controlled Acrolinx state while preserving Feature Registry traceability for the administrative webhook removal and the June 30 contract-expiration risk. ## Linked Sources - ADO Feature: https://msdata.visualstudio.com/A365/_workitems/edit/5391136 - Design Spec task: https://msdata.visualstudio.com/A365/_workitems/edit/5391146 - Deployment task: https://msdata.visualstudio.com/A365/_workitems/edit/5391147 - Feature Registry specs: https://msdata.visualstudio.com/A365/_git/FeatureRegistry?path=/Features/active/5391136 - Teams request: https://teams.microsoft.com/l/message/19:81ff723c-eac9-4b2a-ba9f-844542135555_cc1adbf9-6510-43d6-a849-adba51e66d59@unq.gbl.spaces/1782314980087?context=%7B%22contextType%22%3A%22chat%22%7D - Acrolinx config before cleanup: https://github.com/microsoft/SynapseML/blob/b0fa222cfdde5d0a2cbb2bc6a35630bbb61bc0e3/.acrolinx-config.edn ## Rationale Deleting `.acrolinx-config.edn` is the least invasive source change because the Acrolinx contract is ending and the repo-level webhook was already removed through GitHub administration. Keeping the Feature Registry folder in the repo gives future maintainers a durable pointer to the reason for the cleanup without adding runtime or build behavior. * chore: keep Feature Registry metadata out of SynapseML AB#5391146 AB#5391147 ## Summary Remove the Feature Registry scaffold files from the SynapseML cleanup branch so the public repository PR only deletes the retired Acrolinx config. ## Prompting Intent Engineer clarified that Feature Registry metadata must not be included in the external SynapseML repository. The agent adjusted the existing cleanup PR to keep registry tracking in FeatureRegistry only while preserving the Acrolinx source cleanup. ## Linked Sources - ADO Feature: https://msdata.visualstudio.com/A365/_workitems/edit/5391136 - SynapseML PR: #2570 - FeatureRegistry PR: https://msdata.visualstudio.com/A365/_git/FeatureRegistry/pullrequest/2169703 - User correction: do not include Feature Registry metadata in the external repo ## Rationale Keeping the public SynapseML PR scoped to `.acrolinx-config.edn` avoids adding internal Feature Registry process artifacts to an external repository. Feature-level tracking remains in the FeatureRegistry PR and ADO work items.
* fix: route AnalyzeText document errors to errorCol Move Azure AI Language document-level errors returned inside HTTP 200 AnalyzeText responses from the response payload into the configured error column after auto-batch flattening. Preserve transport error precedence and add a no-network regression test for mixed document success/error responses. AB#4638662 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: pin PR validation sbt launcher Use the sbt launcher version from project/build.properties instead of installing the latest apt sbt package. This keeps the JDK 11 PR validation job on the repository's sbt 1.10.11 launcher and avoids sbt 2.x rejecting JDK 11 before scalastyle can run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use pinned sbt wrapper in PR validation Invoke the downloaded sbt launcher explicitly so the GitHub runner does not resolve its preinstalled sbt 2.x binary under JDK 11. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: prefer pinned sbt on PATH Keep PR validation commands as plain sbt while placing the repository-version launcher first on PATH for subsequent workflow steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: avoid ordering assumption in AnalyzeText error test Partition collected rows by error nullability instead of relying on collect order, addressing PR review feedback about Spark DataFrames being unordered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fix: restore Azure pipeline
Pin the shared Python test environment to MLflow 2.21.3, matching the Databricks test dependency. This constrains protobuf to a compatible major version and invalidates the stale conda cache that breaks Python test collection.
test: migrate OpenAI tests and examples to GPT-5.1
ci: migrate Databricks GPU pool to T4
fix: correct LightGBM improvement tolerance semantics
Bumps [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request) from 5.4.0 to 6.1.1. - [Release notes](https://github.com/amannn/action-semantic-pull-request/releases) - [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md) - [Commits](amannn/action-semantic-pull-request@v5.4.0...v6.1.1) --- updated-dependencies: - dependency-name: amannn/action-semantic-pull-request dependency-version: 6.1.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com>
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.1 to 2.4.4. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@0864cf1...2d11466) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-version: 2.4.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5.6.0 to 5.7.0. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@03ad4de...b6effb0) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: 5.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com>
* docs: add T4 GPU local RAG quickstart ## Summary Add an end-to-end local RAG notebook that performs sentence embedding, exact retrieval, and Phi-4-mini generation on a Databricks T4 worker. Register the notebook in the active GPU smoke suite and documentation sidebar with pinned model dependencies. ## Prompting Intent Reassess the unmerged GPU demo from PR #2271 against current master. Add a maintainable integration example only if it fills a gap beyond the standalone GPU KNN, Hugging Face CausalLM/Phi, and PDF Q&A notebooks; use current T4 assumptions, avoid TensorRT-LLM and custom CUDA setup, provide deterministic smoke assertions, and make no unrelated pipeline changes. ## Linked Sources - Original proposal: #2271 - GPU KNN component: #2157 - Local embedding component: #2236 - Hugging Face CausalLM/Phi component: #2301 - Current Databricks T4 validation platform: #2579 - PDF Q&A reference: https://github.com/microsoft/SynapseML/blob/master/docs/Explore%20Algorithms/AI%20Services/Quickstart%20-%20Document%20Question%20and%20Answering%20with%20PDFs.ipynb ## Rationale The existing notebooks document the individual building blocks but not their local, service-free composition. Exact PyTorch cosine scoring keeps the tutorial small and fully testable on the active T4 suite without reviving the disabled RAPIDS pipeline or its obsolete CUDA/TensorRT initialization. The notebook uses supported current-master models, max_new_tokens rather than conflicting sequence limits, and a PR smoke mode that exercises every GPU stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: harden GPU RAG reproducibility checks ## Summary Pin both Hugging Face repositories to immutable commit snapshots, load the Phi model and tokenizer from the same local snapshot with remote code disabled, and strengthen retrieval validation against input-order fallback. ## Prompting Intent Address independent review findings on PR #2588 by removing mutable model resolution and trust_remote_code, then make the smoke test prove that GPU similarity ranking—not corpus order—selects the answer document. ## Linked Sources - Follow-up pull request: #2588 - Original proposal: #2271 - Pinned embedding snapshot: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/tree/1110a243fdf4706b3f48f1d95db1a4f5529b4d41 - Pinned Phi snapshot: https://huggingface.co/microsoft/Phi-4-mini-instruct/tree/cfbefacb99257ffa30c83adab238a50856ac3083 ## Rationale SentenceTransformer accepts an immutable revision for its complete model/tokenizer snapshot. HuggingFaceCausalLM loads its tokenizer separately, so Phi is first resolved to one pinned worker-local snapshot and both loaders receive that path. Transformers 4.49 natively supports the checkpoint's phi3 architecture, allowing remote model code to remain disabled. A persisted corpus ordinal and independent Python sort over all GPU scores prove the top-k result differs from the first input rows and has strict score ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: format GPU RAG notebook cells ## Summary Apply the repository-pinned Black 22.3 Jupyter formatter to the updated GPU RAG notebook cells. ## Prompting Intent Resolve the Python Style CI failure on PR #2588 without changing notebook behavior or broadening the patch. ## Linked Sources - Pull request: #2588 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229241355 ## Rationale Black's Jupyter formatter omits the terminal newline stored in each code cell. Formatting only the touched notebook aligns its JSON representation with the CI environment while preserving all model-pinning and retrieval assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: pin GPU RAG hub client and FP16 ## Summary Pin huggingface-hub 0.26.0 in the Databricks GPU libraries and notebook setup, verify the Hugging Face dependency set in unit tests, and force Phi model loading to FP16 on T4 hardware. ## Prompting Intent Address the second independent re-review of PR #2588 by making snapshot_download's client version reproducible and preventing Phi's BF16 checkpoint metadata from selecting an unsupported native dtype on T4 GPUs. ## Linked Sources - Pull request: #2588 - Repository environment pin: environment.yml - Hugging Face Hub 0.26.0: https://pypi.org/project/huggingface-hub/0.26.0/ - Pinned Phi configuration: https://huggingface.co/microsoft/Phi-4-mini-instruct/blob/cfbefacb99257ffa30c83adab238a50856ac3083/config.json ## Rationale Version 0.26.0 is already the repository-pinned lower bound used with Transformers 4.49.0, so installing that exact version on the GPU cluster makes snapshot resolution deterministic without introducing a new dependency choice. Phi advertises bfloat16 in its configuration, while NVIDIA T4 compute capability 7.5 lacks native BF16; passing the supported float16 dtype explicitly avoids architecture-dependent auto selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: guard accelerate GPU dependency pin ## Summary Assert that the Databricks GPU library manifest retains accelerate==0.26.0 alongside the pinned Hugging Face dependencies. ## Prompting Intent Address the remaining actionable review feedback on PR #2588 by preventing the runtime dependency used for distributed Phi loading from drifting without a focused unit-test failure. ## Linked Sources - Pull request: #2588 - Reviewed GPU library manifest: core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala ## Rationale The package is already explicitly pinned in GPULibraries, so extending the existing parsed-manifest test is the smallest regression guard and avoids duplicating library configuration or changing runtime behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#2601) * chore(deps): bump github/codeql-action/autobuild from 4.37.3 to 4.37.5 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ranadeepsingh <16433904+ranadeepsingh@users.noreply.github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com>
Bumps [postcss](https://github.com/postcss/postcss) from 8.5.19 to 8.5.25. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@8.5.19...8.5.25) --- updated-dependencies: - dependency-name: postcss dependency-version: 8.5.25 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com>
* chore: migrate artifact links off retiring Azure CDN ## Summary Replace all 400 current-master references to mmlspark.azureedge.net with the repository-owned mmlspark Blob Storage origin across runtime package configuration, release output, examples, documentation, notebooks, and every published documentation version. ## Prompting Intent Recreate the intent of the stale CDN-removal PR on current master only after verifying the supported artifact destination and Azure CDN retirement path. Audit each endpoint use by semantics, preserve package and content paths, validate live artifacts and package resolution, and avoid changing or closing the original PR. ## Linked Sources - Original proposal: #2326 - Azure CDN retirement FAQ: https://learn.microsoft.com/en-us/azure/cdn/classic-cdn-retirement-faq - Azure CDN migration guidance: https://learn.microsoft.com/en-us/azure/cdn/migrate-tier - Azure Front Door/CDN comparison: https://learn.microsoft.com/en-us/azure/frontdoor/front-door-cdn-comparison ## Rationale SynapseML's release pipeline publishes artifacts directly to the mmlspark storage account, the repository already uses that public Blob Storage origin extensively, and byte-for-byte URL checks confirmed the CDN currently proxies the same content. Using the verified origin removes the retiring CDN hostname without inventing an unverified Front Door name, while preserving Maven, documentation, R-package, model, dataset, and icon path semantics. Historical links that already return 404 retain the same status and are not broadened into unrelated artifact-repair work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: make R setup independent of retired CDN ## Summary Repair current and versioned R setup guidance so each release installs its six published, version-matched component archives and resolves SynapseML JVM artifacts through Blob Storage. Document the compatibility bypass required by already-published wrappers, correct the Databricks setup and LightGBM example, remove invalid HTML-page Maven repositories from the Docker demo, and add generator/docs regressions. ## Prompting Intent Investigate the review finding that published R archives still register the retired Azure CDN resolver. Make repository-controlled R installation work with that hostname unavailable, avoid claiming that externally published archives were rewritten, validate local and Databricks-oriented resolution paths, and state the exact external publishing prerequisite for a full artifact migration. ## Linked Sources - Original migration PR: #2326 - Current migration PR: #2589 - Maven repository review: #2589 (comment) - Azure CDN retirement FAQ: https://learn.microsoft.com/en-us/azure/cdn/classic-cdn-retirement-faq - Azure Front Door migration guidance: https://learn.microsoft.com/en-us/azure/cdn/migrate-tier - Apache Spark package repository configuration: https://spark.apache.org/docs/3.5.0/configuration.html#runtime-environment ## Rationale Existing release archives cannot be repaired by a source-only change because their generated sparklyr metadata is already published. Version-matched component downloads plus an explicit Blob resolver and `extensions = character()` provide a tested repository-controlled path without racing or misrepresenting external publication. Future generated archives inherit the corrected resolver from PackageUtils; fully repairing historical metadata still requires an authorized regeneration and publish to the `mmlspark/rrr` container (or a replacement release). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: correct Spark 3.4 compatibility guidance ## Summary Correct the Spark Packages and Python installation snippets so both identify SynapseML 1.0.15 as the compatible release for Spark 3.4 while retaining SynapseML 1.1.3 for Spark 3.5. ## Prompting Intent Address the remaining actionable review feedback on PR #2589 in the existing branch, verify the surrounding compatibility guidance stays consistent, run targeted website validation and code review, and rerun the full PR checks. ## Linked Sources - Pull request and review feedback: #2589 - Original migration context: #2326 ## Rationale The Databricks, Fabric, and SBT guidance already distinguishes SynapseML 1.1.3 for Spark 3.5 from 1.0.15 for Spark 3.4. Updating only the two stale explanatory references restores consistency without changing the Spark 3.5 commands that the snippets demonstrate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
feat: add backward-compatible AAD auth for Azure Search
## Summary Count rows on the original DataFrame RDD so adaptive execution cannot coalesce a projected counting query into a different partition topology. Add a regression that exposes the old 20-to-fewer-partitions drift and verifies exact per-partition counts. ## Prompting Intent Recreate the valid intent behind ancient PR #2282 from current master only after reproducing issue #2278. Isolate distributed startup, feature-width bounds, and native pointer lifetime separately; use TDD and submit only a proven root cause with real regression coverage. ## Linked Sources - Reported failure: #2278 - Superseded ancient proposal: #2282 ## Rationale The literal-only projection was cheaper, but AQE could optimize it to fewer partitions than the training DataFrame. LightGBM then indexed that shortened count array with real task partition IDs, causing the primary ArrayIndexOutOfBoundsException and secondary connection failures. Counting the exact DataFrame RDD trades projection pruning for topology correctness. Feature-width validation and innerPredict cleanup were deliberately excluded because neither was demonstrated as the cause of #2278 or backed by a stable leak regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.3 to 4.37.4. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@e4fba86...f205ea1) --- updated-dependencies: - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.4 to 4.37.5. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@f205ea1...d1ba80a) --- updated-dependencies: - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The Azure AI Anomaly Detector service has been retired by Microsoft. Every `anomalydetector` REST endpoint now answers HTTP 410 (Gone), verified across paths, API versions and regions.
* fix: correct LightGBM improvement tolerance semantics ## Summary Require lower-is-better validation metrics to improve by more than improvementTolerance before resetting the early-stopping counter. Clarify the parameter documentation and add focused regression coverage for both metric directions and zero tolerance. ## Prompting Intent Investigate GitHub issue #2565 from a new branch based on master, determine whether the report is valid, and implement a complete fix suitable for an upstream SynapseML pull request. ## Linked Sources - GitHub issue: #2565 ## Rationale The existing higher-is-better comparison already treats improvementTolerance as a minimum delta, while lower-is-better metrics accepted small regressions. A package-internal comparison helper makes the intended symmetric behavior directly testable without adding a slow native LightGBM fixture or changing public APIs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: harden LightGBM early stopping parameters ## Summary Expand improvement-tolerance coverage across representative LightGBM metrics and tolerance values. Preserve disabled early stopping when earlyStoppingRound is zero, validate both early-stopping parameters, and document their accepted ranges. ## Prompting Intent The engineer requested broader parameter testing to ensure the issue #2565 fix does not introduce downstream regressions. Cover related defaults, boundaries, metric families, invalid values, and early-stopping-round interactions before updating the pull request. ## Linked Sources - GitHub issue: #2565 - Pull request: #2578 - LightGBM 3.3.5 parameters: https://lightgbm.readthedocs.io/en/v3.3.5/Parameters.html#early-stopping-round ## Rationale Correct tolerance semantics classify more rounds as non-improving, so the wrapper must explicitly preserve LightGBM's zero-means-disabled behavior. Shared Spark parameter validators reject values that LightGBM does not support, while deterministic matrix tests cover the decision logic without depending on platform-specific native binaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: prevent sbt bootstrap Maven rate limits SynapseML's Azure Pipelines fans out ~30 hosted-agent matrix jobs that each cold-bootstrap the sbt launcher (org.scala-sbt:sbt:1.10.11, pinned in project/build.properties) and resolve Ivy dependencies from public Maven Central. When many fresh agents -- and several overlapping PR builds -- do this simultaneously, Maven Central returns HTTP 429 (rate limit) and "Setup repo" fails before any test runs (e.g. ADO build 229124511, UnitTests flaky). The pre-existing jittered retries only widened the window against a sustained throttle; they did not remove the thundering herd. Durable fix (cache-first, stagger as supplement): * templates/sbt_cache.yml (primary): Azure Cache@2 for the sbt launcher boot dir (~/.sbt/boot -- the artifact that 429s) and the Ivy cache (~/.ivy2/cache). In steady state, jobs restore these from Azure's cache service and never touch Maven Central. Keys derive from the bootstrap inputs (project/build.properties, project/plugins.sbt, build.sbt) so they invalidate exactly when those change; restoreKeys give a safe partial fallback and continueOnError keeps a cache miss/corruption non-fatal. * BuildAndCacheSbt prewarm job: warms those caches once per run, mirroring the existing BuildAndCacheCondaEnv job. * tools/ci/sbt_retry.sh: single tested helper replacing the duplicated inline retry blocks. Smooths only the cold-cache path with a bounded random start stagger (desynchronises concurrent cold bootstraps) plus bounded jittered exponential-backoff retries. Fails visibly on exhaustion -- no success fallback masking. Wired the shared cache template into every sbt-running job (Style, Publish, Databricks/Fabric E2E, BuildDocker, PythonTests, RTests, WebsiteSamplesTests, UnitTests, ReleaseBranchCompat) by reviving the dormant ivy_cache placeholders, and routed all `sbt setup` bootstraps through the helper. Tests (python -m pytest tools/ci/tests/): deterministically exercise the retry/backoff/stagger + visible-failure behaviour with a fake sbt, and assert pipeline.yaml parses, the cache keys invalidate on bootstrap inputs, and every sbt job is wired to the cache template + prewarm job. No LightGBM, Isolation Forest, GPU, or application changes. TLS verification, job coverage, and all tests are preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: serialize sbt cache prewarm before fan-out ## Summary Make the sbt bootstrap prewarm a mandatory gate before Azure Pipeline matrix jobs start. Add Coursier caching, require exact hits on the boot, Ivy, and Coursier caches before disabling the cold-cache stagger, wire the conditional release job, and strengthen pipeline tests around the dependency graph and cache lifecycle. ## Prompting Intent The engineer asked to fix Maven Central HTTP 429 setup failures in a new stacked PR. The solution must prevent fresh hosted agents from cold-bootstrapping sbt concurrently, allow at least the existing job fan-out after bootstrap is safe, retain bounded retry behavior for cache-service failures, and keep bootstrap failures visible rather than masking them. ## Linked Sources - Failing Azure job: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229124511&view=logs&jobId=e97036a1-bcdb-5cd5-905e-b0cf2c8f33cf - Parent PR investigation: #2578 (comment) - Stacked PR: #2581 - Prewarm concurrency review: #2581 (comment) ## Rationale A best-effort prewarm running beside the matrix does not protect the first run for a new dependency key, so every sbt-running job now waits for one successful warm job. Cache-service errors remain non-fatal and fall back to staggered retries, but a failed warm blocks fan-out to avoid recreating the thundering herd. Coursier is cached alongside sbt boot and Ivy because modern resolution uses all three stores, and the stagger is suppressed only when every cache is an exact hit so dependency-only changes remain desynchronized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: skip Databricks E2E for non-impacting PRs ## Summary Add conservative pull-request impact detection for the six-leg Databricks E2E matrix. Clearly non-impacting documentation, website, GitHub metadata, CI helper, and isolated test-source changes skip Databricks, while all uncertain or runtime-affecting changes continue to run it. ## Prompting Intent The engineer asked to extend PR #2581 so expensive Databricks Azure Pipeline jobs are skipped when the pull request cannot affect notebook execution. The gate must preserve scheduled and branch coverage, avoid brittle CPU-shard mapping, and default to running whenever impact detection is incomplete or uncertain. ## Linked Sources - Stacked CI PR: #2581 - Full green baseline build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229164855 - Azure multi-job output variables: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops#job-output-variables-used-in-other-job-conditions ## Rationale Use one fail-open decision for the complete Databricks matrix because the five CPU partitions mix notebooks across modules and are not stable ownership boundaries. The detector skips only a narrow allowlist of clearly inert paths; runtime code, notebooks, build and pipeline files, Databricks test utilities, shared TestBase infrastructure, unknown paths, empty diffs, and fetch or classifier failures all keep E2E enabled. Non-PR builds always run to preserve scheduled and release coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: gate Databricks CPU and GPU tests independently (#2582) * Gate Databricks CPU and GPU tests independently ## Summary Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently. ## Prompting Intent The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR #2581. ## Linked Sources - Base CI hardening PR: #2581 - GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests - ADO timing audit: build 229176406 ## Rationale CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: fix and streamline release branch compatibility (#2583) * Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: #2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: extend Docker validation timeout ## Summary Raise the BuildDocker job timeout from 60 to 120 minutes and add a pipeline regression test that preserves enough time for both sequential image builds. ## Prompting Intent The engineer asked to diagnose and fix the remaining failure on #2581 and to continue full validation until the parent PR is ready, without hiding genuine test failures. ## Linked Sources - Parent PR: #2581 - Failed PR build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229579403 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229578121 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229580525 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229576499 ## Rationale The Dockerfiles and image behavior were unchanged, but recent hosted-agent builds required roughly 51 minutes when successful and exceeded the default one-hour job cap in multiple master and PR runs. A 120-minute job budget keeps both image validations mandatory while tolerating current registry and package download latency. This is safer and more targeted than skipping an image or doubling agent usage by splitting the builds into parallel jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: address PR review and protect package publishing ## Summary Resolve the blocking PR #2581 review findings by making ACR archival digest-safe, correcting PipelineRun names and sbt cache invalidation, warming cold agents before direct sbt calls, and validating the canonical package version before publishing. ## Prompting Intent The engineer asked to rebase PR #2581 onto current master, audit the new review feedback, fix valid actions, ensure the pipelines continue to publish package versions safely, review the complete change, and rerun Azure validation. ## Linked Sources - Integration PR and review threads: #2581 - Stacked CI changes: #2582 - Stacked CI changes: #2583 - Stacked CI changes: #2584 - Stacked CI changes: #2585 - ACR transfer guidance: https://learn.microsoft.com/azure/container-registry/container-registry-transfer-images - ACR image deletion behavior: https://learn.microsoft.com/azure/container-registry/container-registry-delete ## Rationale Immutable manifest digests prevent mutable tags such as latest from reusing the wrong backup or deleting an unarchived manifest. Per-agent warming is limited to unavailable or inexact cache restores so exact hits remain fast, while the prewarm job still verifies dependency resolution. Package versions are resolved from the SBT source of truth and release publication fails before side effects when the v-tag disagrees. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: Add lossless string identifier support to SAR Refs #2275 Refs #2283 ## Summary Add deterministic, reversible user and item identifier mappings to SAR so string and wide numeric IDs are never cast into lossy caller-visible values. Persist mappings with the model, preserve identifier types in scores and recommendations, define null and unknown-ID behavior, restore typed item recommendation APIs, and add Scala and Python regression coverage. ## Prompting Intent Recreate the intent of the stale SAR string-ID change on current master without copying its lossy casts. Keep the SparkML API coherent and backward compatible for numeric users, use TDD, validate serialization and schema behavior, expose Python wrappers, and exercise targeted compile, style, code generation, Scala, and Python/JVM checks before opening a replacement PR. ## Linked Sources - Feature request: #2275 - Original pull request: #2283 - Current SAR implementation at the starting revision: https://github.com/microsoft/SynapseML/tree/7d9fabcc/core/src/main/scala/com/microsoft/azure/synapse/ml/recommendation - Repository review policy: .github/skills/code-review/SKILL.md ## Rationale Use model-owned typed mappings instead of composing RecommendationIndexer because that stage stringifies numeric identifiers, exposes index columns, and cannot recover every original type. Contiguous deterministic indices keep the existing matrix implementation viable, while persisted DataFrame parameters make decoding reversible after save/load. Inner mapping joins intentionally drop null or unseen scoring IDs, strict type validation prevents ambiguous conversions, and legacy numeric models fall back to identity mappings. The approach accepts a deterministic global sort and persisted mapping storage in exchange for lossless, reproducible SparkML behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Address SAR identifier compatibility review Refs #2275 Refs #2594 ## Summary Resolve the four independent review findings on SAR string identifier support. Preserve typed IDs in ranking train/validation splits, accept only round-trip-safe numeric scoring casts, retain established integer recommendation schemas for safely representable numeric IDs, and rank only factor IDs that have real mappings. Add focused Scala and Python regressions and remove unnecessary mapping cache and interaction-count work identified during review. ## Prompting Intent The engineer asked to fix all medium correctness and compatibility findings on PR #2594, add a regression for each, rerun targeted Scala, code generation, formatting, and Python/JVM validation, then update the existing PR and request re-review without weakening lossless string or wide numeric behavior. ## Linked Sources - Pull request and review context: #2594 - Feature request: #2275 - Original pull request: #2283 - Repository review policy: .github/skills/code-review/SKILL.md ## Rationale Use Spark structs and array functions instead of Double UDF payloads so split schemas remain typed. Numeric scoring IDs are temporarily cast only when casting back reproduces the input, preventing overflow and fractional aliasing while retaining unknown-ID drop semantics. Recommendation decoding conditionally uses the historical integer schema only when every ID round-trips through Int; strings and wide or fractional numeric IDs remain lossless. Candidate indices are intersected with both factors and mappings before top-K so gaps cannot consume recommendation slots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Make SAR numeric identifier handling ANSI-safe Refs #2275 Refs #2594 ## Summary Use ANSI-safe try_cast expressions for numeric identifier compatibility and legacy mappings. Persist whether model-owned user and item mappings safely round-trip through IntegerType, reuse those flags when selecting recommendation output schemas, and limit destination-index collection to mapping-less legacy models. Add ANSI overflow, persisted-flag, legacy-default, and recommendation-planning regressions. ## Prompting Intent The engineer asked to resolve the second independent review of PR #2594: prevent CAST_OVERFLOW under spark.sql.ansi.enabled=true, eliminate repeated mapped-model recommendation scans and index collection, add focused regressions, rerun Scala/codegen/Python validation, update the existing PR, trigger Azure Pipelines, and request another re-review. ## Linked Sources - Pull request and review context: #2594 - Feature request: #2275 - Original pull request: #2283 - Repository review policy: .github/skills/code-review/SKILL.md ## Rationale Use Spark SQL try_cast in both cast directions rather than pre-cast comparisons so out-of-range values become null and are filtered even with ANSI mode enabled. Compute compatibility once while fitting and persist it with conservative false defaults for legacy models, avoiding full mapping scans on every recommendation call. New model mappings are contiguous, so mapped models rank the score vector directly; only mapping-less legacy models collect actual candidate indices to preserve gapped-ID correctness. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Configure a deterministic repository-local Git committer identity before replaying PR commits onto the Spark 3.5 and Spark 4.1 release branches. Distinguish genuine merge conflicts from rebase infrastructure failures and preserve successful rebase diagnostics.
* ci: preserve sbt retry helper during release replay ## Summary Stage the sbt retry helper outside the repository before switching to Spark release branches, and parameterize the shared cache template so it can invoke that stable path after rebase. ## Prompting Intent Investigate why Spark 3.5 and Spark 4.1 compatibility checks still failed after PR #2608, reproduce the failure with PR #2595 changes, implement the complete hotfix, and validate the real release replay path. ## Linked Sources - Failing PR: #2595 - Prior identity hotfix: #2608 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229969678 ## Rationale The rebased working tree intentionally comes from the Spark release branch, so master-only CI helpers cannot remain repository-relative. Copying the helper to Agent.TempDirectory preserves release-specific dependency resolution and avoids moving cache warming ahead of the rebase, where exact cache hits could hide missing release dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: replay only release-relevant PR changes ## Summary Replace commit-history rebasing with a three-way application of the synthetic PR merge tree's release-relevant patch onto each Spark release branch. ## Prompting Intent Validate the compatibility hotfix with PR #2595's real source changes while ensuring CI-only commits do not conflict with old Spark branches that predate the current pipeline and helper files. ## Linked Sources - Validation PR source: #2595 - Prior identity hotfix: #2608 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229969678 ## Rationale The compatibility job needs the effective PR content on the release tree, not unrelated CI and documentation commits. Building the patch from the synthetic merge commit preserves GitHub's merge result, handles source branches behind master, retains three-way conflict detection, and avoids requiring commit identity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: focus release compatibility on Spark 4.1 compilation ## Summary Remove the redundant Spark 3.5 release matrix leg and replace broad Spark 4.1 runtime suites with full test compilation of the effective PR patch. ## Prompting Intent Explain why the release compatibility jobs exist and keep fixing the failures exposed by validation PR #2610, accounting for master already targeting Spark 3.5. ## Linked Sources - Original compatibility PR: #2550 - Streamlining PR: #2583 - Integration validation PR: #2610 - Azure validation build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229984834 ## Rationale Normal PR validation already compiles and tests master on Spark 3.5, so replaying onto the older spark3.5 maintenance snapshot duplicates coverage and introduces unrelated JVM drift. Spark 4.1 test compilation catches cross-version source and test API breakage, while the existing master test fan-out supplies runtime coverage without rerunning broad, memory-heavy suites on a constrained compatibility agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 17, 2026
…ll (#2651) * docs: preserve Spark 4 branch operating knowledge in the branches skill The `spark4.0` and `spark4.1` sync PRs (#2646, #2645) each carry a branch-local `AGENTS_spark4.*.md`. Those files are being deleted, so this moves the detail that had not yet been transferred into `.github/skills/synapseml-branches/`, where it is reachable from master and applies to future rebases. Transferred into the shared reference: - The hand-written `__init__.py` policy. `PythonInitMerger` made these files live code instead of inert text, so a stale one is a real bug; the per-path table records which are empty, removed, or kept, and why. - Scala 2.13's `ClassCastException` on `ArraySeq`, which fails at runtime rather than compile time, and `toIndexedSeq` as the O(1) fix. - The GPU suite split and re-merge: #2538 split it into three clusters of two workers, #2573 reverted it because that needs six GPU nodes against a pool sized for three, and the split's hardcoded notebook indices silently skip any fourth notebook. Recorded so it is not reintroduced. - `DatabricksCPUStreamingTests` is scheduled only on live `spark4.0`; master and `spark4.1` define the class but do not schedule it. This is a convergence, not a regression the sync introduced. - `requestedFor=GitHub` alongside `reason=pullRequest` for telling a trigger-driven build from a hand-queued one. Transferred into `branch-spark4p0.md`: - Runtime strings, and why the DBR version is not a free knob: 17.3 LTS ML ships Spark 4.0 and 18.0 ML ships Spark 4.1, so bumping it stops testing the branch. - The sparklyr 1.9.3 failure, which is live on the branch today. Under dbplyr 2.6 its `tidyselect_data_proxy.tbl_spark` drops the Spark connection, so failures surface as `invoke_static` on `NULL` and read like a dead session. Interleaving is the tell: 21 of 69 failed scattered among passes. - The `SPARK_HOME` connect form, recorded honestly as alignment with 4.1 rather than a fix, because the previous form measured identically. - The Horovod wheel gap, with the `ensure_petastorm_compatibility()` ordering marked explicitly unproven so it is not quoted as a cause. - That the GPU notebook denominator moves from three to four after the sync. Transferred into `branch-spark4p1.md`: runtime strings, the two halves of the Petastorm layer, the `LongOffset` importers, the `np.asarray` `ValueError`, and the exact line needed to re-enable Fabric E2E. Two claims in the branch-local files were deliberately not carried over because measurement contradicts them: the Petastorm shim is a pyarrow compatibility layer with no version gating rather than a Python 3.13/cloudpickle workaround, and NumPy is unpinned on both live branches rather than pinned on `spark4.0`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: drop the fabricated central Seq conversion claim The Copilot review on #2651 is correct and the text it flagged was wrong. �sImmutableCollection exists in no branch, and CognitiveServiceBase.getValueOpt returns the row or default value with no Seq conversion at all. The claim came from the branch-local AGENTS notes, which described a design that never shipped: the helper appears only in two abandoned commits (745b342, 6cab133) that are contained in no tip. Carrying it into the skill would have preserved the error in the place people are told to trust. Keeps what is verifiable and useful -- the ClassCastException surfaces at runtime rather than compile time, and toIndexedSeq preserves O(1) indexing -- and records that there is no central conversion today, with the command to re-check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: scope the guard-test and Fabric-condition claims to their branches Both suppressed Copilot comments on 4a1190b were worth acting on, and measuring them turned up something the text had missed. test_http_package.py and test_package_exports.py: the reviewer is right that they do not exist, on master. They are on spark4.1 and reach spark4.0 through #2646, but master carries PythonInitMerger without either test. Stating them unqualified in a master-resident file implied a guard that is not there, so the text now names the full paths, says which branches have them, and calls out that master has the merger without the tests -- a real gap rather than just a wording fix. FabricE2E condition: the claim was accurate for spark4.1, which does use a bare `condition: false`, but the reviewer checked master and found the parameterised form, which is exactly the confusion a master-resident file describing another branch invites. The text now scopes the claim and records all three forms, since they differ on every branch: master has and(succeeded(), eq(...)), spark4.0 has eq(...) with no succeeded(), spark4.1 has false. The restore target is named as master's form rather than quoted loose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: fix line length and punctuation review comments Addresses the two remaining review items on this PR. - branch-spark4-common.md: the __init__.py table had two rows of 248 and 153 characters, over the repo's 120-char limit. Moved the long explanations out of the Why cells into the prose immediately below, so the table stays a quick index and no line exceeds 120. - branch-spark4p1.md:41: replaced the bare double hyphen with a sentence break. Also fixed the check that missed these. The earlier length lint skipped lines starting with a pipe, on the assumption that table rows were exempt, which is exactly why these two slipped through. Re-linted every line of every file in the skill: 0 lines now exceed 120 characters, and the only remaining double hyphen is the git pathspec separator inside backticks in a code example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: record where the Java version is declared and how to resolve its sync conflict The Java version is declared in up to five files per branch and none of that was captured. Adds a measured table and, more importantly, the conflict rule. Corrects a wrong claim while doing it. The old text said Spark 4's .github/workflows/pr-validation.yml uses JDK 17. That holds for spark4.1 but spark4.0 pins java-version: 11 there while the rest of the branch is on 17, so the workflow is not evidence of a branch's Java version. Measured across ms/master, ms/spark4.0 and ms/spark4.1 rather than taken from the old notes. The part that can actually break a sync: PR #2652 adds templates/java_setup.yml to master at versionSpec 11, and both Spark 4 branches already have that file at 17. Verified with git merge-tree that this is an add/add conflict which leaves markers rather than silently overwriting, so the sync stops and asks. The hazard is that the intuitive resolution -- take master, it is newer -- is the wrong one and drops the branch to Java 11, reintroducing the 'Class java.lang.Record not found' failure #2652 exists to fix. Documented the rule as always keep the branch's own 17, noted that the conflict is one-time, and gave a one-line command to verify the result. Also records that spark4.0 has no JAVA_VERSION because it is not yet in the ReleaseBranchCompat matrix, and that spark4.1 has java_setup.yml but nothing includes it yet, so neither absence is mistaken for a regression later. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: close the gaps found by an independent transfer audit Ran two independent audits, one per source file, against the whole skill directory, to check that AGENTS_spark4.0.md and AGENTS_spark4.1.md can be deleted without losing anything. Both flagged the same top two gaps. Verified every claim before writing it, because the source files are already known to contain three false ones. Closed: 1. The CI trigger lives on the ADO definition, not in pipeline.yaml. Both audits ranked this first and it verified more sharply than the source put it. Definition 17563 reports triggers[].branchFilters = +master, +spark3.5, +spark4.0, +spark4.1 and settingsSourceType 2, meaning UI-defined. The proof that the YAML is a red herring is on the branch itself: spark4.0's own pipeline.yaml pr: block lists master, spark3.3 and spark3.5 and does not list spark4.0, yet PRs targeting spark4.0 build. Recorded with the REST call to re-read it, and the consequence that a future release branch gets no PR builds from a pipeline.yaml edit alone. 2. Root cause of the VerifyTrainClassifier fixture change: Spark 4 does not tolerate a NaN feature reaching logistic regression the way 3.5 did. The destination had kept the "value was replaced rather than the assertion weakened" reasoning but dropped the reason. Confirmed Double.NaN is still on master at VerifyTrainClassifier.scala:121 and absent on spark4.1, so a sync will try to restore it; that is now stated. 3. Why re-enabling Fabric E2E belongs in its own PR: the pipeline run is the test, and a Fabric provisioning failure should not block an unrelated merge. 4. The pyarrow rationale, with measured values rather than the audit's. The audit reported "bumped to 18.0.0 for cp312 wheels", which conflates the two branches. Measured: spark4.0 is pyarrow 22.0.0 on Python 3.12.11, spark4.1 is 18.0.0 on Python 3.13, master is held down at 10.0.1 because Petastorm uses Parquet and fsspec APIs removed after PyArrow 10. Added the table and noted each branch reached its value by a different route, so the reasoning does not carry across. Also recorded spark4.0's unexplained, unvalidated mlflow downgrade to 1.26.1 against master's 2.21.3. Both audits independently confirmed the destination is otherwise a superset of the source, and that the three deliberate corrections (asImmutableCollection, the Petastorm shim's real cause, and the pr-validation.yml Java version) read as intended rather than as losses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: address review on Java/Dockerfile scope and java_setup.yml state Scope the Dockerfile JAVA_HOME claim explicitly to the Spark 4 branches and state master's value (11) for contrast; the bullet sat under a Spark-4 heading but read as a repo-wide claim. Fixes the plural-subject grammar in the same sentence. Add tools/docker/*/Dockerfile as a sixth row to the Java declaration table (master 11, both Spark 4 branches 17). Phrase templates/java_setup.yml on master conditionally: it does not exist there until #2652 merges, so tell readers to expect it absent and give the command to confirm on the live branch. Reflow two paragraphs where an earlier edit left a mid-sentence fragment on its own line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: ranadeepsingh <ranadeepsingh@users.noreply.github.com>
* fix: pin the JDK for InternalCompat to the branch under test The SynapseML-Internal compatibility check never set a JDK, so it ran on the agent default. That works for master and spark3.5 but cannot compile the Spark 4 branches, which target Java 17: build 231477746 (#2645, spark4.1) failed publishing the OSS tree and build 231456784 (#2646, spark4.0) failed compiling the Internal tree, both with 'Class java.lang.Record not found', a type that arrived in Java 16. This is structural rather than intermittent. Across recent builds of definition 17563 the job succeeds on every master-targeting PR and fails on both Spark 4 PRs, so it would be a permanent red on those branches. The job is advisory by design, and an advisory check that is always red is one people stop reading, which loses the signal it exists to carry. The JDK now follows the branch under test, derived in the step that already parses and validates System.PullRequest.TargetBranch to pick the Internal branch, since both sides of the comparison are built from that branch's Spark line. Java 17 also needs the java.prefs --add-opens flag for sbt, mirroring the ReleaseBranchCompat job above, which builds the same branches with the same task and flag on the same pool. Only Spark 4 branches are pinned. master and spark3.5 pass on the agent default, so pinning them would change working behaviour to fix a problem they do not have; an empty value skips the JDK step and leaves that path unchanged. The two variables are declared with empty job-level defaults because an undefined ADO macro is passed to the script verbatim, and an unexpanded COMPAT_SBT_OPTS macro would be read by bash as a command substitution rather than an empty argument. Two later steps run under succeededOrFailed(), so they can execute even when the step that sets the variables did not. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: pin InternalCompat JDK per branch via templates/java_setup.yml Replaces the runtime target-branch mapping from the first commit on this PR with the pin the repository already uses. Why the first approach was wrong -------------------------------- It read the target branch at runtime and mapped spark4.* to JDK 17 inside pipeline.yaml. That added a fifth place where a branch's Java version is declared, alongside .github/workflows/pr-validation.yml, environment.yml, templates/java_setup.yml and the ReleaseBranchCompat matrix. It also carried a hardcoded spark4.* glob that a future spark5.0 would silently fall through, and two job-level variables that existed only to keep an undefined ADO macro from reaching bash as a command substitution. What this does instead ---------------------- templates/java_setup.yml is already this repo's per-branch JDK pin: it exists on spark4.0 and spark4.1 with versionSpec 17, and spark4.0's pipeline.yaml already includes it. This commit adds master's copy with versionSpec 11 -- measured as master's current effective JDK, so master's behaviour does not change -- and includes the template from InternalCompat. The include is the same single line on every branch, so the sync never has to reconcile pipeline.yaml for this, and the per-branch value lives in exactly one file that already carries the right value on the branches that need it. A future release branch is correct by construction: it gets its JDK from its own java_setup.yml rather than from a glob in master's pipeline. The sbt --add-opens flag is now passed unconditionally rather than through a variable. It is only required on 17, but --add-opens has existed since Java 9 and an unknown module is a warning rather than an error, so it is inert on 11. That removes both job-level variables and the macro-expansion hazard with them. Verification ------------ - master's InternalCompat runs Temurin 11.0.32 today (build 231341458 echoes JAVA_HOME and java -version), so versionSpec 11 is a no-op there. - templates/java_setup.yml is byte-identical to spark4.1's copy apart from the two version tokens. - build.sbt declares nothing about Java on any branch, so sbt cannot make this choice; it runs on whatever JDK is on PATH. - Net change is 23 insertions against master, 47 deletions against the previous commit; 11 of the insertions are the explanatory comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: cover every JVM in InternalCompat with JAVA_TOOL_OPTIONS The Copilot review on this PR was right and my previous commit was wrong. I claimed "all four sbt invocations carry the flag". There are not four. The review identified two more, and both checks out: * templates/sbt_cache.yml runs `bash tools/ci/sbt_retry.sh update`, an sbt invocation in a template this job includes, which no per-command flag in pipeline.yaml can reach. * the `Run Internal Scala tests` loop runs `if ! sbt "testOnly com.microsoft...$pkg.**"`, which my grep for lines starting with `sbt ` missed because of the `if ! ` prefix. Per-command flags are the wrong shape for this regardless of how many I find, because they also cannot reach a JVM the tests fork. Replaced them with a single job-level JAVA_TOOL_OPTIONS, which every JVM started anywhere in the job picks up. This is not a new mechanism for the repo. spark4.0's pipeline.yaml already does `export JAVA_TOOL_OPTIONS="--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED"` in two places for the same reason. Set unconditionally rather than behind a Spark 4 condition: --add-opens has existed since Java 9 and an unknown module is a warning rather than an error, so it is inert on master's Java 11. That keeps a branch conditional out of the file entirely. Checked the one hazard this introduces. JAVA_TOOL_OPTIONS makes every JVM print "Picked up JAVA_TOOL_OPTIONS: ..." to stderr, and this job captures sbt output with 2>&1 and parses a version out of it. The parse greps '^\[info\] ([0-9]+\.[0-9]+|HEAD-)', anchored to the start of the line, which the "Picked up" line cannot match. Safe. Also worth recording: spark4.0's Style job runs `sbt scalastyle test:scalastyle` on Java 17 with no --add-opens and passes, so the flag is not needed for sbt startup, only for particular tasks. That is why the failure showed up where it did rather than immediately. Net result against master is now 34 insertions and zero deletions, 25 of them comment, so the functional change is 10 lines: the new template file, one template include, and one variable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: correct the rationale comment for setting JAVA_TOOL_OPTIONS unconditionally The review is right: the previous comment justified this as 'inert on master's Java 11' on the grounds that an unknown module only warns. That reasoning describes a case that does not apply here. java.prefs has existed since Java 9, so on Java 11 the option is genuinely applied, not ignored. The conclusion is unchanged -- it is still safe to set unconditionally -- but for the correct reason: it opens a package that nothing on master's path reflects into, which is harmless, and setting it for every target keeps a branch conditional out of pipeline.yaml. Comment-only change; no behavioural difference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brings in #2651 (Spark 4 branch operating knowledge under .github/skills/synapseml-branches/) and #2652 (InternalCompat now pins its JDK via templates/java_setup.yml instead of the agent default, which was compiling this branch's Scala 2.13 tree on a pre-16 JDK and failing with 'Class java.lang.Record not found'). Conflict: templates/java_setup.yml add/add. Master added it at versionSpec 11 (master's already-effective JDK); this branch already carried it at 17. Resolved by keeping this branch's 17 -- taking master's side would silently downgrade the branch and reintroduce the Record failure. This conflict is one-time; the file histories are now connected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ranches skill #2651 moved this branch's operating knowledge onto master under .github/skills/synapseml-branches/, which the merge above brings here: references/branch-spark4-common.md for what both Spark 4 branches share and references/branch-spark4p1.md for this branch's specifics. The skill is the better home: it is resolved automatically by branch, it lives on master so it no longer has to survive every sync, and its content was checked against this file by two independent audits before the move. Three claims were deliberately corrected rather than copied, because measurement contradicted them. Verified before deleting: nothing in the tree references AGENTS_spark4.1.md, and the base AGENTS.md is untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/azp run microsoft.SynapseML |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Collaborator
Author
Sync completeness audit — spark4.1 ← masterPosting the evidence behind "everything from master is in, and everything branch-specific survived", since a sync PR of this size can't be eyeballed. 1. Ancestry — every master commit is inZero master commits are unreachable from this head. Structurally complete. Ancestry alone isn't proof, though — a conflict resolution can keep ancestry while dropping master's content. So: 2. Content — master's added lines are presentCollected every line master added in the last 120 days for the files that still differ, and checked each is present in the branch. Everything came back 95–100% except two files, both investigated:
The three recent LightGBM fixes were confirmed present by ancestry:
3. Test parity
An earlier pass flagged missing 4. Branch-unique values preservedThe merge did not overwrite what makes this branch a Spark 4.1 branch:
5. AGENTS file removal
6. Note on automated reviewCopilot cannot review this PR — at 23,788 changed lines it is past the 20,000-line reviewer limit, so it returns "exceeds the maximum number of lines" rather than findings. That is inherent to a sync PR and not something shrinking the diff can fix. Worth noting for reviewers: the bulk of this diff is master's own already-reviewed code. The genuinely new work here is the merge resolution plus the doc deletion — 5 files, +275/−294 versus the previous head, which is reviewable on its own. Currently unresolved review threads: 0. Suppressed review comments across all reviews: 0. This PR is |
Collaborator
Author
Per-commit scan of master → this branchRan the same three checks on this PR that surfaced a gap on #2646, to confirm nothing from master was dropped here. A file-level diff can't see a file master has that the branch simply lacks, so these checks target that specifically. Check 1 — files master has that this branch does not0 files. Clean. (#2646 had 2 — two Check 2 — per-commit line-level validationFor all 165 master commits in the last year: take the lines each commit added, keep only those still present in master's current content (this controls for master later rewriting its own lines, which otherwise produces false alarms), then verify each against this branch. All 44 are deliberate adaptations. A representative sample:
One that looked like a drop and is not: Check 3 — silent content lossEvery file differing from master, comparing blob sizes: no file is below 60% of master's size. Nothing was quietly gutted. Sibling-branch difference worth recording
Result
No changes needed on this PR from the scan. |
Rana Singh (ranadeepsingh)
pushed a commit
that referenced
this pull request
Aug 17, 2026
…r published 'Run Internal Python tests (ExcludeAIFunc)' fails on essentially every PR. All ~21 tests error at setup and the only visible cause is an opaque JAVA_GATEWAY_EXITED, which hides an Ivy resolve failure for an Internal jar that no step ever published. Internal's version comes from sbt-dynver, which appends a live '-<yyyyMMdd>-<HHmm>' suffix whenever the working tree is dirty and recomputes it from the wall clock on every sbt load. The 'Retarget Internal to this build' step edits build.sbt with two sed commands, which is exactly what makes the tree dirty for the rest of the job. Each later sbt session therefore picks a different version. Observed on build 231488245 (PR #2645): 11:32 'sbt packagePython publishM2' bakes ...-1132-SNAPSHOT into the generated Python package and publishes that same jar to ~/.m2 11:36 'sbt testPythonExcludeAIFunc' re-runs CodeGen in a new sbt session, rebakes the package as ...-1136-SNAPSHOT and pip-installs it 11:37 pytest fixtures read the baked coordinate out of the installed package via synapse.ml.ebm.__spark_package_version__ and ask Ivy for ...-1136-SNAPSHOT, which nobody published make_mlflow_models.py runs in between and succeeds: it resolves the 1132 jar correctly from local-m2-cache, because the rebake has not happened yet. The existing same-session mitigation for packagePython and publishM2 is therefore necessary but not sufficient, since a later sbt session recomputes the version regardless. Committing the retarget makes the tree clean, so dynver stops appending a timestamp and every sbt session in the job computes the same version. The OSS side already demonstrates the end state: its checkout is never edited, so its version carries no timestamp at all (1.1.3-python3.13-102-bfba9c82-SNAPSHOT in that same build). Verified against git directly: clean tree v1.1.3.0-1-g0a826a4f tracked file modified (the sed) v1.1.3.0-1-g0a826a4f+DIRTY after commit v1.1.3.0-2-gdc0c4fca + gitignored target/ + untracked v1.1.3.0-2-gdc0c4fca The last row matters for correctness here: 'git describe --dirty', which is what dynver reads, only considers modifications to tracked files, so sbt's own build output cannot reintroduce the drift later in the job. A diff-index guard fails the step loudly if the tree is ever dirty at that point anyway. The commit is local to the agent's checkout and never pushed. Targeted at spark4.1 rather than master so the fix is validated on a branch where this check actually fails, and against SynapseML-Internal's own spark4.1 branch. Also fixes the diagnostic listing published versions: it hardcoded '*-internal_2.12', so on this branch, which builds _2.13, it printed nothing - exactly where a version mismatch most needed to be visible. Internal Scala tests were never affected: sbt resolves through its own classpath and never goes through Ivy for the Internal artifact. Fixes #2653 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 17, 2026
## Summary Remove five stale release compatibility prerequisite entries after the Spark 4.1 sync incorporated their changes, while retaining the configuration file required by tooling tests. ## Prompting Intent Make PR microsoft#2634 engineering-ready after rebasing onto the post-sync master, diagnose the exact Azure compatibility failure, and fix the deterministic blocker without weakening the release replay gate or changing the verified OpenAI patch. ## Linked Sources - GitHub issue: microsoft#2246 - Pull request: microsoft#2634 - Spark 4.1 sync: microsoft#2645 - Blocked Azure validation: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231553224 ## Rationale PR microsoft#2645 squash-merged all five prerequisite changes into Spark 4.1, so ancestry checks can no longer identify their original master SHAs and incorrectly replay them into files that already contain the changes. Retaining a comment-only file preserves the tested configuration contract and allows future prerequisites to be added normally, while avoiding special-case logic or weakening conflict detection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 17, 2026
## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR microsoft#2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request microsoft#2630: microsoft#2630 - Spark 4.1 master sync microsoft#2645: microsoft#2645 - Azure Search auth prerequisite microsoft#2591: microsoft#2591 - Azure Search dependent change microsoft#2604: microsoft#2604 - LightGBM retry prerequisite microsoft#2612: microsoft#2612 - LightGBM IPv6 dependent change microsoft#2637: microsoft#2637 - Coverage baseline microsoft#2507: microsoft#2507 - Precision-recall AUC prerequisite microsoft#2635: microsoft#2635 - Metadata prerequisite microsoft#2632: microsoft#2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync microsoft#2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
that referenced
this pull request
Aug 18, 2026
…r published (#2655) * fix: stop InternalCompat requesting an Internal version that was never published 'Run Internal Python tests (ExcludeAIFunc)' fails on essentially every PR. All ~21 tests error at setup and the only visible cause is an opaque JAVA_GATEWAY_EXITED, which hides an Ivy resolve failure for an Internal jar that no step ever published. Internal's version comes from sbt-dynver, which appends a live '-<yyyyMMdd>-<HHmm>' suffix whenever the working tree is dirty and recomputes it from the wall clock on every sbt load. The 'Retarget Internal to this build' step edits build.sbt with two sed commands, which is exactly what makes the tree dirty for the rest of the job. Each later sbt session therefore picks a different version. Observed on build 231488245 (PR #2645): 11:32 'sbt packagePython publishM2' bakes ...-1132-SNAPSHOT into the generated Python package and publishes that same jar to ~/.m2 11:36 'sbt testPythonExcludeAIFunc' re-runs CodeGen in a new sbt session, rebakes the package as ...-1136-SNAPSHOT and pip-installs it 11:37 pytest fixtures read the baked coordinate out of the installed package via synapse.ml.ebm.__spark_package_version__ and ask Ivy for ...-1136-SNAPSHOT, which nobody published make_mlflow_models.py runs in between and succeeds: it resolves the 1132 jar correctly from local-m2-cache, because the rebake has not happened yet. The existing same-session mitigation for packagePython and publishM2 is therefore necessary but not sufficient, since a later sbt session recomputes the version regardless. Committing the retarget makes the tree clean, so dynver stops appending a timestamp and every sbt session in the job computes the same version. The OSS side already demonstrates the end state: its checkout is never edited, so its version carries no timestamp at all (1.1.3-python3.13-102-bfba9c82-SNAPSHOT in that same build). Verified against git directly: clean tree v1.1.3.0-1-g0a826a4f tracked file modified (the sed) v1.1.3.0-1-g0a826a4f+DIRTY after commit v1.1.3.0-2-gdc0c4fca + gitignored target/ + untracked v1.1.3.0-2-gdc0c4fca The last row matters for correctness here: 'git describe --dirty', which is what dynver reads, only considers modifications to tracked files, so sbt's own build output cannot reintroduce the drift later in the job. A diff-index guard fails the step loudly if the tree is ever dirty at that point anyway. The commit is local to the agent's checkout and never pushed. Targeted at spark4.1 rather than master so the fix is validated on a branch where this check actually fails, and against SynapseML-Internal's own spark4.1 branch. Also fixes the diagnostic listing published versions: it hardcoded '*-internal_2.12', so on this branch, which builds _2.13, it printed nothing - exactly where a version mismatch most needed to be visible. Internal Scala tests were never affected: sbt resolves through its own classpath and never goes through Ivy for the Internal artifact. Fixes #2653 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: only commit the retarget when it changed something Addresses review feedback on the guard added by this PR: 'git commit -am' exits non-zero when nothing is staged, and this step runs under 'set -e', so an unconditional commit would turn the 'already retargeted' case into a hard step failure and make the step non-idempotent on retry. Verified against git directly: git commit --quiet --no-verify -am (nothing staged) -> exit 1 git diff-index --quiet HEAD -- (clean tree) -> exit 0 git diff-index --quiet HEAD -- (dirty tree) -> exit 1 The preceding grep -qF checks already prove build.sbt holds the requested synapseMLVersion and Resolver.mavenLocal, so an unchanged tree is a valid state rather than an error: it means the edits were already in place. Either way dynver sees a clean tree and emits one stable version, which is the property the rest of the job depends on. The post-commit diff-index guard is unchanged and still fails loudly if the tree is dirty at that point. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: make the Scala-suffix comment accurate on every branch The glob comment was written on spark4.1 and asserted 'this branch builds _2.13', which is false on master (scalaVersion 2.12.17). pipeline.yaml is kept identical across master, spark4.1 and spark4.0, so the comment has to hold on all of them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: check git status directly in the post-retarget dirty guard The guard tested whether 'git diff-index --name-only' printed anything. That output is empty both when the tree is clean and when git itself fails, so a broken checkout read as 'clean' and the job continued with a version that was not actually stable - the exact state the guard exists to catch. Measured (unborn HEAD): git diff-index --name-only HEAD -- stdout empty, exit 128 git diff-index --quiet HEAD -- exit 128 Isolating the guard with the commit already skipped: old guard, git broken -> exit 0 (passes) new guard, git broken -> exit 1 (fails loudly) both guards, dirty -> exit 1 both guards, clean -> exit 0 '--quiet' is non-zero for both 'dirty' and 'git failed', neither of which is safe to continue from. The 'if' condition is exempt from 'set -e' and the diagnostic is guarded with '|| true' so a failing git cannot abort the step before the explicit exit 1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: separate the two comment paragraphs in the retarget step Every other paragraph in this block is separated by a bare comment line; these two were run together, so the dynver rationale and the idempotency rationale read as one paragraph. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make the Internal retarget genuinely idempotent Copilot suppressed a low-confidence comment pointing out that sed -i '/^resolvers ++= Seq(/a\ Resolver.mavenLocal,' build.sbt appends unconditionally, unlike the version edit above it which is a substitution. Running the step twice against one tree adds a second Resolver.mavenLocal line each time. Measured: 1 -> 2 -> 3 over three runs. That cannot accumulate on a hosted agent, which starts from a fresh VM, but the comment exposed a real defect: because the append always dirtied the tree, the 'tree is already clean' branch added in 6ced9d8 could never be taken, so the state it documents was unreachable. Executing the real step three times against a simulated Internal checkout then surfaced a second, larger problem. Runs 2 and 3 exited 1: Retargeting Internal to OSS version ... On branch master nothing to commit, working tree clean >> step exit=1 'sed -i' rewrites build.sbt in place, so its mtime and inode change even when the bytes are identical. 'git diff-index' trusts stat info before comparing content and reported a phantom modification, so the step took the commit branch, 'git commit' found nothing staged and exited 1, and 'set -e' failed the step. The earlier fix for the same Copilot comment was therefore only half a fix - the guard it added was reading a stale index. Three changes: - 'git update-index -q --refresh || true' before testing the tree, so the guard compares content rather than stat data. '|| true' because --refresh exits non-zero when a file really did change. - Insert Resolver.mavenLocal only when the resolvers block lacks it. - Verify with the same block-scoped predicate the insert decides on, so the two cannot disagree. A file-wide test could be satisfied by the unrelated Resolver.mavenLocal in usage/build.sbt and report success for a resolvers block that never received one - a false green. Re-running the real extracted step three times now gives: run 1 commits, exit 0 run 2 'Retarget produced no changes; tree is already clean', exit 0 run 3 same, exit 0 Resolver.mavenLocal count stays 1 git describe stays v1.1.3.0-1-g22c27875 with no +DIRTY Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 18, 2026
## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR microsoft#2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request microsoft#2630: microsoft#2630 - Spark 4.1 master sync microsoft#2645: microsoft#2645 - Azure Search auth prerequisite microsoft#2591: microsoft#2591 - Azure Search dependent change microsoft#2604: microsoft#2604 - LightGBM retry prerequisite microsoft#2612: microsoft#2612 - LightGBM IPv6 dependent change microsoft#2637: microsoft#2637 - Coverage baseline microsoft#2507: microsoft#2507 - Precision-recall AUC prerequisite microsoft#2635: microsoft#2635 - Metadata prerequisite microsoft#2632: microsoft#2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync microsoft#2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 18, 2026
## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR microsoft#2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request microsoft#2630: microsoft#2630 - Spark 4.1 master sync microsoft#2645: microsoft#2645 - Azure Search auth prerequisite microsoft#2591: microsoft#2591 - Azure Search dependent change microsoft#2604: microsoft#2604 - LightGBM retry prerequisite microsoft#2612: microsoft#2612 - LightGBM IPv6 dependent change microsoft#2637: microsoft#2637 - Coverage baseline microsoft#2507: microsoft#2507 - Precision-recall AUC prerequisite microsoft#2635: microsoft#2635 - Metadata prerequisite microsoft#2632: microsoft#2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync microsoft#2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 18, 2026
## Summary Remove release compatibility prerequisites already incorporated by the Spark 4.1 sync while retaining PR microsoft#2638, which PR microsoft#2629 still requires. ## Prompting Intent Make PR microsoft#2629 merge-ready after its current-head Azure compatibility job failed, without weakening conflict detection or dropping the Featurize dependency used by its production and test changes. ## Linked Sources - PR microsoft#2629: microsoft#2629 - PR microsoft#2638: microsoft#2638 - Spark 4.1 sync microsoft#2645: microsoft#2645 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231604593 ## Rationale Spark 4.1 already contains port-adapted equivalents of the five older manifest entries, so replaying their original master patches conflicts before CI reaches the PR. PR microsoft#2638 is not yet present and remains necessary because microsoft#2629 uses its missing-numeric Featurize API and test imports. Keeping only that exact commit matches the successful local replay and preserves the pipeline's existing validation semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 18, 2026
## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR microsoft#2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request microsoft#2630: microsoft#2630 - Spark 4.1 master sync microsoft#2645: microsoft#2645 - Azure Search auth prerequisite microsoft#2591: microsoft#2591 - Azure Search dependent change microsoft#2604: microsoft#2604 - LightGBM retry prerequisite microsoft#2612: microsoft#2612 - LightGBM IPv6 dependent change microsoft#2637: microsoft#2637 - Coverage baseline microsoft#2507: microsoft#2507 - Precision-recall AUC prerequisite microsoft#2635: microsoft#2635 - Metadata prerequisite microsoft#2632: microsoft#2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync microsoft#2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
pushed a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 18, 2026
## Summary Remove release compatibility prerequisites already incorporated by the Spark 4.1 sync while retaining PR microsoft#2638, which PR microsoft#2629 still requires. ## Prompting Intent Make PR microsoft#2629 merge-ready after its current-head Azure compatibility job failed, without weakening conflict detection or dropping the Featurize dependency used by its production and test changes. ## Linked Sources - PR microsoft#2629: microsoft#2629 - PR microsoft#2638: microsoft#2638 - Spark 4.1 sync microsoft#2645: microsoft#2645 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231604593 ## Rationale Spark 4.1 already contains port-adapted equivalents of the five older manifest entries, so replaying their original master patches conflicts before CI reaches the PR. PR microsoft#2638 is not yet present and remains necessary because microsoft#2629 uses its missing-numeric Featurize API and test imports. Keeping only that exact commit matches the successful local replay and preserves the pipeline's existing validation semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
that referenced
this pull request
Aug 18, 2026
* fix: handle empty featurized text features AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make #2629 engineering-ready on current master after PRs #2638 and #2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue #1667: #1667 - GitHub pull request #2629: #2629 - Automated review overview: #2629 (review) - Suppressed review findings: #2629 (review) - Merged pull request #2638: #2638 - Merged pull request #2633: #2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained #2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: declare the Featurize release prerequisite ## Summary Declare PR #2638 as a release compatibility prerequisite for PR #2629. ## Prompting Intent Make PR #2629 merge-ready after rebasing it onto the latest master, including the Spark 4.1 compatibility replay required by SynapseML CI. ## Linked Sources - PR #2629: #2629 - PR #2638: #2638 ## Rationale The PR #2629 tests intentionally exercise the missing-numeric Featurize API merged by PR #2638. Spark 4.1 has not yet received that change, so replaying only the #2629 delta cannot compile. Declaring the merged commit lets the existing compatibility job apply the exact dependency first and keeps the prerequisite removable after the next release-branch sync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: retire synced compatibility prerequisites ## Summary Remove release compatibility prerequisites already incorporated by the Spark 4.1 sync while retaining PR #2638, which PR #2629 still requires. ## Prompting Intent Make PR #2629 merge-ready after its current-head Azure compatibility job failed, without weakening conflict detection or dropping the Featurize dependency used by its production and test changes. ## Linked Sources - PR #2629: #2629 - PR #2638: #2638 - Spark 4.1 sync #2645: #2645 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231604593 ## Rationale Spark 4.1 already contains port-adapted equivalents of the five older manifest entries, so replaying their original master patches conflicts before CI reaches the PR. PR #2638 is not yet present and remains necessary because #2629 uses its missing-numeric Featurize API and test imports. Keeping only that exact commit matches the successful local replay and preserves the pipeline's existing validation semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rana Singh (ranadeepsingh)
added a commit
that referenced
this pull request
Aug 18, 2026
* fix(core): include reference-only distribution categories ## Summary Fix DistributionBalanceMeasure so custom comparisons cover the union of observed and positive-probability reference categories. Add typed validation, robust persisted-number decoding, one-category and empty-input handling, public Scala/Python regressions, documentation, and conflict-free Spark 4.1 replay. ## Prompting Intent Make GitHub issue #2010 and PR #2630 engineering-ready end to end. Preserve public signatures, serialized parameter shapes, output schema, empty-map uniform behavior, and existing observed-only semantics while covering reference-only categories, null and invalid inputs, supported category types, copy/save-load, generated Python, SQL-native scalability, and release-branch compatibility. ## Linked Sources - Bug report: #2010 - Pull request: #2630 - Related Jensen-Shannon normalization: #2631 - Automated review finding: #2630 (comment) - Spark 4.1 compatibility failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231496504 - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale Build a typed DataFrame from the already driver-resident reference parameter, then union and aggregate it with distributed observed statistics. This aligns support without collecting observed categories, RDD APIs, Python UDFs, or persistent caches. Normalize complex-param numeric values during Param validation so Python and persisted integral numbers are safe while preserving the Spark 4 branch's strict-map adaptation and clean patch replay. Dynamic support cardinality keeps chi-square behavior correct without changing public schema or signatures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(core): keep distribution measurement lazy ## Summary Remove eager input and support-count actions from DistributionBalanceMeasure. Build observed and configured support in one SQL-native plan, preserve empty-input and category semantics with bounded sentinel rows, and add a regression proving transform does not launch Spark jobs or cache its input. ## Prompting Intent Continue PR #2630 through current-head automated review and address the reported double-scan performance regression without RDD APIs, driver collection, persistent caches, unbounded materialization, public signature changes, schema changes, or serialized parameter changes. Preserve reference-only, observed-only, null, integral, string, uniform, persistence, and Spark 4.1 behavior. ## Linked Sources - Bug report: #2010 - Pull request: #2630 - Current-head automated review: #2630 (review) - Performance finding: #2630 (comment) - Related Jensen-Shannon normalization: #2631 ## Rationale Avoid the suggested cache because input-size materialization and cache lifetime are unsafe for a reusable transformer. Instead, project supported string and integral sensitive columns into a long SQL representation, aggregate the input once, and perform row/support totals over the reduced category support. Small configured-reference and sentinel rows stay driver-bounded; sentinels preserve empty-input rejection lazily. This removes eager full scans while retaining all metric and compatibility contracts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(core): correct infinite chi-square p-values ## Summary Return a right-tail chi-square p-value of zero when the test statistic is positive infinity. Align the independent Scala calculator, strengthen Scala and generated-wrapper Python regressions, and clarify the documented interpretation. ## Prompting Intent Continue PR #2630 through current-head automated review, audit every new finding, and fix real metric defects without weakening tests or changing public signatures, output schema, serialized parameters, or supported category behavior. Verify the corrected metric through Scala, generated Python, Spark 3.5, and Spark 4.1 paths before pushing. ## Linked Sources - Bug report: #2010 - Pull request: #2630 - Current-head review: #2630 (review) - Chi-square p-value finding: #2630 (comment) - Data balance documentation: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md ## Rationale The implementation computes the upper-tail probability as `1 - CDF(score)`, whose limit is zero as the score approaches positive infinity. Returning one inverted the statistical meaning and reported maximum compatibility for an impossible observed category. Preserve the one-category degree-of-freedom special case at one, while making infinite statistics consistently produce zero across production, test calculators, Scala tests, Python wrappers, and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: retire synced release compatibility prerequisites ## Summary Drain the release compatibility prerequisite manifest after the Spark 4.1 master sync incorporated every configured baseline. This prevents old master patches from being replayed over their Spark 4.1-adapted equivalents before pull request changes are validated. ## Prompting Intent Make PR #2630 fully engineering-ready by resolving the release-compatibility failure at its root. Verify each configured prerequisite against live master and Spark 4.1 history, preserve replay ordering, run the exact release replay and affected CI tests, and avoid unrelated pipeline changes. ## Linked Sources - Pull request #2630: #2630 - Spark 4.1 master sync #2645: #2645 - Azure Search auth prerequisite #2591: #2591 - Azure Search dependent change #2604: #2604 - LightGBM retry prerequisite #2612: #2612 - LightGBM IPv6 dependent change #2637: #2637 - Coverage baseline #2507: #2507 - Precision-recall AUC prerequisite #2635: #2635 - Metadata prerequisite #2632: #2632 - Failed current-head Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231542808 ## Rationale The release matrix currently validates only Spark 4.1, and sync #2645 contains every commit named by the manifest. Their source was deliberately adapted during the port, so exact reverse-patch detection cannot recognize them and sequential replay conflicts before the PR patch. An exact local replay showed the first two stale entries failing in turn; an empty manifest then applied the PR patch cleanly, compiled the complete Spark 4.1 test surface, and passed the 30-test distribution suite. Keeping the supported empty manifest is narrower and safer than weakening conflict detection or special-casing individual patches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Merges
masterintospark4.1. The branch was 67 commits / 305 files behind — last synced on 2026-08-10 by #2617 — so it was missing every fix merged since, including the Spark 4 correctness fix in #2643.master@eb76ff6bd0→ merged intospark4.1@1d1d0dadf8. 144 files changed, +14363/−635. 33 files conflicted (~86 hunks); each was resolved individually rather than by a blanket--ours/--theirs.Why this matters
#2643 fixed model metadata read/write to go through
SparkSessioninstead of RDD APIs, which is what makes SynapseML work on Spark Connect and Databricks Unity Catalog standard/serverless clusters. That fix landed onmasteronly — so the branch that exists to support Spark 4 was still running the code path Spark 4 users hit. This sync closes that gap.Resolution rule
Two things made this decidable rather than a judgement call:
git log <merge-base>..spark4.1 -- <file>separates files touched only by the previous sync (take master) from files touched by a real Spark 4 upgrade commit (preserve).ManualInitPackageFoldersis base=0, master=0, spark4.1=2 ⇒ spark4.1 added it ⇒ keep ours.license_expressionis base=0, master=1, spark4.1=0 ⇒ master added it ⇒ take theirs.Spark 4 commits preserved:
b76c391be4(Spark 4.0 / Java 17 / Scala 2.13 / Python 3.12) and791af3237d(Spark 4.1.1 / Scala 2.13.17 / Python 3.13).Kept from spark4.1 (Spark 4 identity)
pipeline.yamlSPARK_VERSION=4.1.1, Java-17-safeSBT_OPTSSBT_OPTScarries-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled, removed in Java 14+ — they abort JVM startup on Java 17pipeline.yamlcondition: falseon FabricE2Eenvironment.ymlpip=21.3,numpy==1.26.4(no 3.13 wheels),torch==2.1.2.github/skills/*(5 files)pr-validation.yml17java-version: 17had already auto-merged; only the label conflictedDatabricksUtilities.scalaDatabricksCPUStreamingTestsneedsStreamingNotebooks; dropping it breaks compilationOpenAIChatCompletion.scalascala.collection.Seq+.toSeqis required in 2.13, where bareSeqis immutableTaken from master
SparkSession-based model metadata I/O, plus theSynapseMLLoggingandSerializerfollow-ups. All four fixes confirmed present on the merged tree.NetworkManager.scala— theNetworkManagerSocketSupportrefactor anddriverUnreachableException(new fileNetworkManagerSocketSupport.scalacame in cleanly).pipeline.yaml— theInternalCompatjob, the release-compat prerequisite replay,publish_coverage_ado, andcontinueOnError: trueon "Load Codecov token".Deliberate non-adoptions
numpy==1.26.4— master pins it; 1.26.4 publishes no Python 3.13 wheels. An inline guard comment now sits abovenumpyinenvironment.ymlso a future sync doesn't silently re-apply the pin.test_pipeline_yaml.py— hunk 1 taken from master dropped theBUILD_SBT/PLUGINS_SBTconstants, which are still used; they were re-added by hand. The brittleassert 'branches: [ "master", "spark4.1" ]' in workflowstring match was replaced with a real YAML-parsing helper,_pr_validation_branches()(which also handles YAML 1.1 folding a bareon:key to booleanTrue). The FabricE2E assertion stayscondition is Falseto match this branch.One test adapted — and why it is not a regression
EnsembleByKeySuite→ "no active session should expose the documented case-resolution limitation" fails on Spark 4.1 as written.Both
EnsembleByKey.scalaandEnsembleByKeySuite.scalaare byte-identical to master on this branch (git diff ms/master --is empty), so the merge did not touch them. Measured on Spark 4.1.1:transformer.transformSchema(input.schema)key,id,score,featureskey,id,score,features— unchangedassembler.transformSchema(thatSchema)FEATURES does not existvectorthatSchema("FEATURES")FIELD_NOT_FOUND— unchangedSpark 3.5 resolved
VectorAssemblerinput columns with a case-sensitiveStructTypelookup, so it disagreed with the case-insensitive fallbackEnsembleByKeyapplies when no session is active, and the pipeline was rejected. Spark 4 routes that lookup throughSQLConf.get.resolver, which applies the same session-less fallback — so the two now agree and the pipeline builds. SynapseML's own behaviour is identical; only the downstream Spark consequence moved, and it moved in the safer direction.The assertion is retargeted at the transformed schema — the stable contract across both versions, and what the test set out to document: the fallback really did drop
FEATURES. The replacement passes on Spark 3.5 as well, somasterandspark4.1can carry the identical test and this does not become a recurring merge conflict.Validation
Every row below was executed, not inferred — Scala 2.13.17 / Spark 4.1.1 / Java 17.
sbt compile(all 6 modules)sbt Test/compilesbt scalastyle Test/scalastyle(all modules)core/testOnlyserialize + logging + train + stagespytest tools/ci/tests/test_pipeline_yaml.pyFollow-ups (not in this PR)
spark4.0to the GitHub Actions and ADO branch filters —spark4.0currently runs no CI at all.spark4.0needs the same sync; it is 547 files / ~4.5 months behind and should land after ci: run PR validation on the spark3.5, spark4.0 and spark4.1 branches #2644 so it isn't merged blind.spark4.0is deliberately still absent from theReleaseBranchCompatmatrix — enabling it before its sync lands would make it permanently red.EnsembleByKeySuitechange should be applied tomasterso the two branches converge.Updates since the description above
Correction to this description
The table above justifies
condition: falseonFabricE2Ewith "no Spark 4.1 Fabric runtime exists yet". That is no longer true. Fabric Runtime 2.0 has since reached general availability on Apache Spark 4.1 (Scala 2.13, Python 3.13, Java 21, Delta 4.x), which makes this the one Spark 4 branch Fabric can actually host, and makes the disabled job real lost coverage rather than an unavoidable skip.It is not re-enabled in this PR, deliberately. Checking what that would take turned up a concrete blocker beyond the pipeline condition:
core/src/test/scala/.../fabric/FabricOperations.scalahardcodes'SparkVersion': '3.5'in the workspace-creation payload, so flipping the condition alone would provision a 3.5 workspace and fail. Enabling it needs that change, the restored condition, and a Fabric capacity in thesempy-integration-regionable to provision Runtime 2.0 workspaces — and the only real test of the last one is a live pipeline run. That belongs in its own PR where the pipeline run is the test, rather than inside a sync PR where a Fabric provisioning failure would block an unrelated merge.The reasoning, the blocker, and the exact steps are recorded in
AGENTS_spark4.1.md(added here) so this is not rediscovered later. Note thatspark4.0reaches the opposite conclusion correctly — Fabric has no Spark 4.0 runtime at all, so it stays disabled there permanently.Additional fix
OpenAIPromptoverrides via MRO instead of a hardcoded class namepyInternalWrappermakes codegen emitclass _OpenAIPrompt, sosuper(OpenAIPrompt, self)raisesNameError— that name does not exist inside the generated moduleDocumentation
Adds
AGENTS_spark4.1.mdrecording every divergence on this branch and why, plus the sharedAGENTS.md/CONTRIBUTING.mdpair (byte-identical acrossmaster,spark4.0andspark4.1— verified by git blob hash; proposed for master in #2648).The branch file inverts the framing used on
spark4.0. That branch's file says "check spark4.1 first"; this one says "ask whether spark4.0 needs this too", because this branch descends from that one's upgrade commit and is the more actively maintained of the pair. It therefore lists the four things that are genuinely 4.1-only and states that everything else is a back-port candidate.Two of those four were measured rather than assumed, by running the same probe against real Spark 4.0.1 and 4.1.1 installs:
ImageTransformer.toNDArrayusingnp.frombufferbytesfor aBinaryTypecolumn where 4.0 returnsbytearray.np.asarrayhandlesbytearraybut raisesValueErroronbytesspark_utilsusingspark.createDataFrame(rdd, schema)rdd.toDF(schema)works on both 4.0.1 and 4.1.1df.rdd.zipWithIndex()is still an RDD callAdding these files also surfaced a bug:
.gitignorehas ignoredAGENTS.mdand.agents/sinceb76c391be4, the Spark 4.0 upgrade commit this branch inherits from. Master ignores neither. The failure mode is the quiet kind —git add AGENTS.mdprints a hint and exits zero, so the file is simply absent from the commit..agents/is worse: it holds tracked repo content, and a probe confirmed any new file under it is silently dropped. Both rules removed.Completeness audit
Reachability was not treated as sufficient evidence that the sync landed — an empty
git log master ^branchonly proves the commits are ancestors, and a conflict resolution can discard master's side while leaving the merge commit intact. Every file master changed since the merge base was compared line by line:All 71 were either false positives where master's intent survived a refactor, or deliberate documented adaptations — the
environment.ymlpins for Python 3.13 (whose comments now state the sync policy explicitly, e.g. "Intentionally unpinned: master pins numpy==1.26.4, which has no Python 3.13 wheels. Do not adopt master's pin when syncing."), and the CMS GC flags removed fromSBT_OPTSbecause they are invalid on Java 17.The same audit run against
spark4.0did find two real gaps, both fixed in #2646.