Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 1 addition & 69 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@ name: CI Native CLI
on:
schedule:
- cron: '0 12 * * 3' # This runs the workflow every wednesday day at 12:00 UTC
env:
NATIVE_VERSION: 100.100.100

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
CI:
strategy:
fail-fast: false
matrix:
os: [ mulesoft-ubuntu, mulesoft-windows ]
include:
Expand Down Expand Up @@ -70,70 +69,3 @@ jobs:
- name: Run Node.js Tests
run: ./gradlew --stacktrace --no-problems-report native-lib:nodeTest
shell: bash

# Derive lowercase OS + runtime arch tokens for artifact names.
# OS comes from matrix.script_name (linux/windows); arch from uname -m
# (x86_64 today; stays correct if an arm64 runner is ever added).
- name: Derive platform tokens
run: echo "ARCH=$(uname -m)" >> "$GITHUB_ENV"
shell: bash

# archive:false makes the artifact name equal the uploaded file name.
# Copy the Gradle-named distro zip to the convention name so the CLI
# artifact reads dw-cli-<ver>-<os>-<arch>.zip.
- name: Stage renamed CLI distro
run: cp "native-cli/build/distributions/native-cli-${{env.NATIVE_VERSION}}-native-distro-${{ matrix.script_name }}.zip" "native-cli/build/distributions/dw-cli-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.zip"
shell: bash

# Upload the artifact file
- name: Upload generated script
uses: actions/upload-artifact@v7.0.1
with:
# archive:false skips the redundant outer zip (the distro is already a
# .zip) and names the artifact after the file — which already carries
# ${matrix.script_name}, so the matrix legs don't collide. No `name:`:
# it would be ignored under archive:false.
path: native-cli/build/distributions/dw-cli-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.zip
archive: false

# Upload the Python wheel
- name: Upload Python wheel
uses: actions/upload-artifact@v7.0.1
with:
# archive:false skips the redundant outer zip and names the artifact
# after the wheel — whose platform tag (manylinux/win_amd64/…) is
# already OS-unique, so the matrix legs don't collide. No `name:`: it
# would be ignored under archive:false.
path: native-lib/python/dist/dataweave_native-0.0.1-py3-*.whl
archive: false

# npm pack emits the same filename (dataweave-native-0.0.1.tgz) on every
# OS. With archive:false the file NAME becomes the artifact name (the
# `name:` input is ignored), so copy to an OS-qualified name first to keep
# the matrix legs from colliding.
- name: Stage OS-qualified Node package
run: cp native-lib/node/dataweave-native-0.0.1.tgz "native-lib/node/dataweave-node-0.0.1-${{ matrix.script_name }}-${{ env.ARCH }}.tgz"
shell: bash

# Upload the Node.js package
- name: Upload Node package
uses: actions/upload-artifact@v7.0.1
with:
path: native-lib/node/dataweave-node-0.0.1-${{ matrix.script_name }}-${{ env.ARCH }}.tgz
# Single .tgz (already gzip-compressed); skip the redundant outer zip
# (v7+ feature). archive:false ignores `name:` and uses the file name,
# which the copy above made OS-unique.
archive: false

# Upload the native shared library + header together per OS. Multiple
# files, so this stays archived (archive:false allows only one file); the
# zip is wanted here and `name:` still applies.
- name: Upload native shared library
uses: actions/upload-artifact@v7.0.1
with:
name: dwlib-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}
path: |
native-lib/python/src/dataweave/native/dwlib.dylib
native-lib/python/src/dataweave/native/dwlib.so
native-lib/python/src/dataweave/native/dwlib.dll
native-lib/python/src/dataweave/native/dwlib.h
22 changes: 16 additions & 6 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,23 @@ jobs:
# This workflow contains a single job called "build"
BUILD:
strategy:
fail-fast: false
matrix:
os: [ mulesoft-ubuntu, mulesoft-windows ]
os: [ ubuntu-latest, windows-2022, macos-latest ]
include:
- os: mulesoft-ubuntu
# script_name → artifact-name OS token (our naming convention).
# distro_os → Gradle distro classifier (native-cli getOsName():
# macOS is "osx", not "macos"), used for the source
# filename produced by native-cli:distro.
- os: ubuntu-latest
script_name: linux
- os: mulesoft-windows
distro_os: linux
- os: windows-2022
script_name: windows
distro_os: windows
- os: macos-latest
script_name: macos
distro_os: osx
runs-on: ${{ matrix.os }}
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
Expand Down Expand Up @@ -65,7 +75,7 @@ jobs:

# Install Python build dependencies (setuptools/wheel may be missing on Windows runners)
- name: Install Python build dependencies
run: python3 -m pip install --upgrade setuptools wheel
run: python3 -m pip install --break-system-packages --upgrade setuptools wheel
shell: bash

# Generate native-lib python wheel
Expand Down Expand Up @@ -110,7 +120,7 @@ jobs:

# Derive lowercase OS + runtime arch tokens for artifact names.
# OS comes from matrix.script_name (linux/windows); arch from uname -m
# (x86_64 today; stays correct if an arm64 runner is ever added).
# (x86_64 on linux/windows, arm64 on macos).
- name: Derive platform tokens
run: echo "ARCH=$(uname -m)" >> "$GITHUB_ENV"
shell: bash
Expand All @@ -119,7 +129,7 @@ jobs:
# Copy the Gradle-named distro zip to the convention name so the CLI
# artifact reads dw-cli-<ver>-<os>-<arch>.zip.
- name: Stage renamed CLI distro
run: cp "native-cli/build/distributions/native-cli-${{env.NATIVE_VERSION}}-native-distro-${{ matrix.script_name }}.zip" "native-cli/build/distributions/dw-cli-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.zip"
run: cp "native-cli/build/distributions/native-cli-${{env.NATIVE_VERSION}}-native-distro-${{ matrix.distro_os }}.zip" "native-cli/build/distributions/dw-cli-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.zip"
shell: bash

# Upload the artifact file
Expand Down
30 changes: 25 additions & 5 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,23 @@ jobs:
# This workflow contains a single job called "build"
RELEASE_EXTENSION:
strategy:
fail-fast: false
matrix:
os: [ mulesoft-ubuntu, mulesoft-windows ]
os: [ ubuntu-latest, windows-2022, macos-latest ]
include:
- os: mulesoft-ubuntu
# script_name → artifact-name OS token (our naming convention).
# distro_os → Gradle distro classifier (native-cli getOsName():
# macOS is "osx", not "macos"), used for the source
# filename produced by native-cli:distro.
- os: ubuntu-latest
script_name: linux
- os: mulesoft-windows
distro_os: linux
- os: windows-2022
script_name: windows
distro_os: windows
- os: macos-latest
script_name: macos
distro_os: osx
runs-on: ${{ matrix.os }}

# Steps represent a sequence of tasks that will be executed as part of the job
Expand Down Expand Up @@ -54,7 +64,7 @@ jobs:

# Install Python build dependencies (setuptools/wheel may be missing on Windows runners)
- name: Install Python build dependencies
run: python3 -m pip install --upgrade setuptools wheel
run: python3 -m pip install --break-system-packages --upgrade setuptools wheel
shell: bash

# Generate native-lib python wheel
Expand Down Expand Up @@ -83,7 +93,7 @@ jobs:
uses: svenstaro/upload-release-action@v2
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
file: native-cli/build/distributions/native-cli-${{env.NATIVE_VERSION}}-native-distro-${{ matrix.script_name }}.zip
file: native-cli/build/distributions/native-cli-${{env.NATIVE_VERSION}}-native-distro-${{ matrix.distro_os }}.zip
asset_name: dw-cli-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.zip
tag: ${{ github.ref }}
overwrite: true
Expand Down Expand Up @@ -129,6 +139,16 @@ jobs:
tag: ${{ github.ref }}
overwrite: true

- name: Upload native shared library to release (macOS)
if: runner.os == 'macOS'
uses: svenstaro/upload-release-action@v2
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
file: native-lib/python/src/dataweave/native/dwlib.dylib
asset_name: dwlib-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.dylib
tag: ${{ github.ref }}
overwrite: true

# Upload the native library header
- name: Upload native library header to release
uses: svenstaro/upload-release-action@v2
Expand Down
2 changes: 1 addition & 1 deletion build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ buildscript {
mavenCentral()
}
dependencies {
classpath "org.graalvm.buildtools.native:org.graalvm.buildtools.native.gradle.plugin:0.11.2"
classpath "org.graalvm.buildtools.native:org.graalvm.buildtools.native.gradle.plugin:0.11.5"
}
}

Expand Down
182 changes: 182 additions & 0 deletions docs/superpowers/specs/2026-07-29-github-runners-migration-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# GitHub Actions Runner Migration + macOS Artifacts

**Date:** 2026-07-29

## Goal

Update the runners used across the three GitHub Actions workflows, and add
macOS artifact production:

- **A) `ci.yml`** — keep the `mulesoft` runners, but make it a pure build+test
workflow: remove all artifact output.
- **B) `main.yml` + `release.yml`** — replace the `mulesoft` runners with
standard GitHub-hosted runners.
- **C) `main.yml` + `release.yml`** — add a macOS runner leg so a
macOS-compatible artifact is produced.

## Current state

All three workflows use the same matrix:

```yaml
matrix:
os: [ mulesoft-ubuntu, mulesoft-windows ]
include:
- os: mulesoft-ubuntu
script_name: linux
- os: mulesoft-windows
script_name: windows
runs-on: ${{ matrix.os }}
```

- `ci.yml` — scheduled (Wednesdays 12:00 UTC). Builds, creates distro, builds
Python wheel + Node package, runs Node tests. Then 3 upload-prep steps
(*Derive platform tokens*, *Stage renamed CLI distro*, *Stage OS-qualified
Node package*) and 4 `upload-artifact` steps (CLI distro, Python wheel, Node
package, native shared library).
- `main.yml` — push/PR to master. Same build steps plus master-only regression
+ Node TCK conformance lanes, plus the same 3 prep + 4 upload-artifact steps.
- `release.yml` — on `v*` tags. Builds, then uploads assets to the GitHub
Release via `svenstaro/upload-release-action`. The native shared library
upload is split into **per-OS conditional** steps: Linux (`.so`), Windows
(`.dll`), plus a shared header upload.

Artifact naming already derives arch from `uname -m` (`ARCH`), so adding a new
OS leg produces correctly-named artifacts automatically.

## Changes

### A) `ci.yml` — build + run only

- Matrix and `runs-on` **unchanged** (keep `mulesoft-ubuntu` + `mulesoft-windows`).
- **Remove** these steps (they exist only to produce/stage uploads):
- *Derive platform tokens*
- *Stage renamed CLI distro*
- *Upload generated script*
- *Upload Python wheel*
- *Stage OS-qualified Node package*
- *Upload Node package*
- *Upload native shared library*
- **Keep** everything up to and including *Run Node.js Tests*.
- Result: `ci.yml` builds and runs tests, produces no artifacts.

### B) `main.yml` + `release.yml` — GitHub-hosted runners

Replace the matrix runner labels in both files:

- `mulesoft-ubuntu` → `ubuntu-latest`
- `mulesoft-windows` → `windows-2022`

`script_name` mappings (`linux`, `windows`) and all other steps stay the same.

> **Windows label: `windows-2022`, not `windows-latest`.** The migration
> originally targeted `windows-latest`, but that image rolled forward to
> Visual Studio 18 (VS 2026, install path `…\Microsoft Visual Studio\18\…`),
> which the pinned `node-gyp` 11.5.0 cannot detect (`find VS unknown version
> "undefined"`) — the Node addon build fails. `windows-2022` ships the VS 2022
> / v17 toolchain that node-gyp 11 supports (the same toolchain class the
> self-hosted `mulesoft-windows` runner uses) and is still a standard
> GitHub-hosted runner, satisfying goal B.
>
> **Follow-up:** when `windows-2022` is eventually retired, the durable fix is
> bumping `node-gyp` to `^12.1.0` (which added VS 18/2026 support) and
> returning to `windows-latest`.

### C) macOS leg on `main.yml` + `release.yml`

Add a third matrix entry to both files:

```yaml
matrix:
os: [ ubuntu-latest, windows-2022, macos-latest ]
include:
# script_name → artifact-name OS token (our naming convention).
# distro_os → Gradle distro classifier (native-cli getOsName():
# macOS is "osx", not "macos"), used for the source
# filename produced by native-cli:distro.
- os: ubuntu-latest
script_name: linux
distro_os: linux
- os: windows-2022
script_name: windows
distro_os: windows
- os: macos-latest
script_name: macos
distro_os: osx
```

- Chosen runner: **`macos-latest` (Apple Silicon, arm64)**. `uname -m` returns
`arm64`, so all artifacts read `...-macos-arm64...` with no other change.
- **`distro_os` field.** `native-cli:distro` names its zip with the Gradle
`getOsName()` classifier, which is **`osx`** on macOS (not `macos`). The
staging step that renames the distro to the `dw-cli-<ver>-<os>-<arch>.zip`
convention (and `release.yml`'s upload `file:`) must reference the *source*
file by its `distro_os` classifier, while the *convention* artifact name
keeps `script_name` (`macos`). For linux/windows the two tokens coincide, so
only macOS needs the distinction.
- **`main.yml`** — macOS is just another matrix leg; every existing step runs
for it. The *Upload native shared library* step already lists `dwlib.dylib`
in its `path`, so macOS is covered with no edit to that step.
- **`release.yml`** — add a new conditional step mirroring the existing
Linux/Windows branches:

```yaml
- name: Upload native shared library to release (macOS)
if: runner.os == 'macOS'
uses: svenstaro/upload-release-action@v2
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
file: native-lib/python/src/dataweave/native/dwlib.dylib
asset_name: dwlib-${{env.NATIVE_VERSION}}-${{ matrix.script_name }}-${{ env.ARCH }}.dylib
tag: ${{ github.ref }}
overwrite: true
```

The shared *header* upload step is not OS-conditional and needs no change.

## Orchestration

A coordinator (main session) dispatches parallel subagents — one per workflow
file (`ci.yml`, `main.yml`, `release.yml`) since the edits are independent. All
work happens on a single new branch; a single PR is opened at the end.

## Risks / notes

- macOS `native-lib` native build uses `-J-Xmx6G`; `macos-latest` (arm64) has
~7GB RAM. Should fit but is the tightest leg — monitor the first run.
- The master-only regression + Node TCK steps in `main.yml` will also execute
on the macOS leg for master pushes (expected, more coverage).
- macOS GitHub-hosted minutes bill at a higher multiplier than Linux/Windows.
- `windows-2022` is pinned (see section B); it is not deprecated today but will
eventually be retired — track the node-gyp `^12` follow-up before then.

## Deviations discovered during CI

These fixes were required to get all three legs green and are outside the
original A/B/C scope; recorded here so the as-shipped state is complete.

- **GraalVM buildtools `0.11.2 → 0.11.5`** (`build.gradle`). On `windows-latest`
the toolcache (C:) and workspace (D:) live on different drives; buildtools
0.11.2 called `Path.relativize` across drive roots and crashed
`nativeCompile` (`'other' has different root`). 0.11.5 guards relativize to
same-root paths.
- **`node-gyp` bumped to `^11`** (`native-lib/node/package.json` +
regenerated lockfile). Intended to fix VS detection on `windows-latest`; it
did **not** (11.5.0 still can't see VS 18), which is why the Windows leg was
pinned to `windows-2022` instead. The bump is harmless and left in place.
- **`--break-system-packages`** added to the Python build-deps step in
`main.yml` + `release.yml`. macOS system Python enforces PEP 668 and refuses
the plain `pip install --upgrade setuptools wheel`.

## Testing

No unit tests apply. Validation is:
- YAML lint / parse sanity for all three files.
- Confirm no `mulesoft-*` labels remain in `main.yml` / `release.yml`, and that
`ci.yml` still uses them.
- Confirm the Windows leg uses `windows-2022` (not `windows-latest`).
- Confirm `ci.yml` has zero `upload-artifact` steps.
- Confirm the three-way matrix + macOS release branch exist, and that the
distro staging/upload references `matrix.distro_os` for the source filename.
- End-to-end: all three legs (`ubuntu-latest`, `windows-2022`, `macos-latest`)
of the `Build Native CLI` workflow pass — confirmed green on commit `4bb7e46`.
Loading
Loading