From 20b3d0171e2b6925366ed25d947e3551f6a3b59f Mon Sep 17 00:00:00 2001 From: Anuj Hydrabadi Date: Tue, 15 Sep 2026 12:57:44 +0530 Subject: [PATCH] feat(module-ci): apply output types before modules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A modules repository now carries the output types its modules exchange, at outputs/{namespace}/{name}.yaml. CI had no step for them. The order is not a preference. A module names its output type by reference and the control plane refuses a module upload whose type it does not already hold ("Output @ns/name not found"), so the types are applied first in both preview and publish mode. Preview and publish are not the same operation at different strengths, because an output type is global and unversioned: the control plane keys it on name and namespace alone, with no stage and no version, and a write replaces it for every module at once. There is no slot to hold a proposed change and nothing to roll back to. - preview (pull request): `raptor create output-type -f FILE --if-absent`. A type the control plane has never seen is created — it has no consumers, so the write is safe, and it is the only way a pull request that adds a module together with its new type can validate its own module. A type that already exists is only reported in the PR comment. - publish (push): the full apply. raptor still refuses a removal or a retype while some module uses the type. - cleanup (PR close): nothing. CI never deletes an output type — there is no preview slot to restore one from, and a type carries no provenance, so there is no ownership check like the module cleanup relies on. The PR comment is now assembled in the upsert step rather than in the module step, because a pull request can touch output types only, modules only, or both, and the marker must appear exactly once in the final body. New input `apply-output-types` (default true) turns the whole thing off for a repository whose types are managed elsewhere. Co-Authored-By: Claude Opus 5 --- README.md | 18 ++-- module-ci-action/README.md | 56 +++++++++++-- module-ci-action/action.yml | 161 ++++++++++++++++++++++++++++++++++-- 3 files changed, 217 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 7371f4f..02cc540 100644 --- a/README.md +++ b/README.md @@ -7,17 +7,23 @@ Releases are semver tags with a moving major alias (`v1` always points at the la ## 1. Facets Module CI (current) The **Facets Module CI** GitHub Action provides end-to-end CI for a Facets **modules -repository** (`modules/{intent}/{flavor}/{version}`) using the **`raptor`** CLI. It runs +repository** — the modules at `modules/{intent}/{flavor}/{version}` and the output types +they exchange at `outputs/{namespace}/{name}.yaml` — using the **`raptor`** CLI. It runs in three modes, derived automatically from the triggering event: [module-ci-action README](./module-ci-action/README.md). -- **Preview (on pull request)**: Validates each changed module, then registers an - unpublishable feature-branch preview pinned to the PR head commit (optionally posts a - PR comment). -- **Publish (on push)**: Uploads and publishes each changed module (PREVIEW → PUBLISHED). +- **Preview (on pull request)**: Creates each changed output type the control plane does + not have yet, then validates each changed module and registers an unpublishable + feature-branch preview pinned to the PR head commit (optionally posts a PR comment). +- **Publish (on push)**: Applies each changed output type, then uploads and publishes each + changed module (PREVIEW → PUBLISHED). - **Cleanup (on pull request close)**: Deletes each changed module's preview, but only - the previews this PR's own commits created (ownership-checked). + the previews this PR's own commits created (ownership-checked). Output types are never + deleted — they are global and unversioned. + +Output types are always applied **before** any module: a module names its output type by +reference, and the control plane refuses a module whose type it does not already hold. This is the action the Facets control plane wires into bootstrapped modules repositories, and the recommended CI for all modules repos. diff --git a/module-ci-action/README.md b/module-ci-action/README.md index fdb72f4..8b02796 100644 --- a/module-ci-action/README.md +++ b/module-ci-action/README.md @@ -9,8 +9,8 @@ triggering event: | Event | Mode | What it does | |-------|------|--------------| -| `pull_request` opened / synchronize / reopened | **preview** | Validates each changed module, then registers a **feature-branch preview** pinned to the PR head commit. Optionally upserts a PR comment. | -| `push` (to your default branch) | **publish** | Uploads each changed module and **publishes** it (PREVIEW → PUBLISHED). | +| `pull_request` opened / synchronize / reopened | **preview** | Creates each changed output type the Control Plane does not have yet, then validates each changed module and registers a **feature-branch preview** pinned to the PR head commit. Optionally upserts a PR comment. | +| `push` (to your default branch) | **publish** | Applies each changed output type, then uploads each changed module and **publishes** it (PREVIEW → PUBLISHED). | | `pull_request` **closed without merge** | **cleanup** | Deletes the preview each changed module owns — **only if** the preview belongs to a commit from this PR. | | `pull_request` **closed by merge** | **no-op** | Skipped: the concurrent `push` (publish) run re-uploads and publishes the module at the merge commit, so it owns the slot. Running cleanup here would be redundant and could race the publish. | @@ -20,7 +20,8 @@ triggering event: ## Modules-repo layout -Each module lives at a versioned path and contains a `facets.yaml` plus its Terraform: +A modules repository holds two trees: the modules, and the **output types** they +exchange. ``` modules/ @@ -31,14 +32,54 @@ modules/ main.tf variables.tf outputs.tf +outputs/ + / + .yaml # name: "@namespace/name", properties:, providers: ``` The action discovers a module as **any directory under `modules/` that contains a `facets.yaml`**. `intent`, `flavor`, and `version` are read from that `facets.yaml` and form the module reference `intent/flavor/version` used by `raptor`. -If your modules tree is not at the repo root, set `path-prefix` (e.g. `infra/` when -modules live at `infra/modules/...`). +It discovers an output type as **any `.yaml`, `.yml` or `.json` file under +`outputs/`**. Each file names itself through its `name` field; the directory layout is +a convention, not a lookup key. `raptor get output-type @namespace/name -o yaml` +prints exactly this shape, so an existing type can be saved to a file, edited, and +committed. + +If your trees are not at the repo root, set `path-prefix` (e.g. `infra/` when they +live at `infra/modules/...` and `infra/outputs/...`). + +## Output types + +A module names its output type by reference (`@namespace/name`). The Control Plane +**refuses a module upload whose output type it does not already hold** +(`Output @ns/name not found`), so the action always applies the output types first, +in both preview and publish mode. + +An output type is different from a module in one way that drives everything here: it +is **global and unversioned**. The Control Plane keys it on `name` + `namespace` +alone — there is no preview stage, no version, and a write replaces it for every +module and every environment at once. There is no slot to hold a proposed change and +nothing to roll back to. + +So the two modes are not the same operation at different strengths: + +| Mode | What happens to an output type | +|------|-------------------------------| +| **preview** (pull request) | `raptor create output-type -f --if-absent`. A type the Control Plane has never seen is **created** — it has no consumers, so the write cannot break anything, and it is the only way a PR that adds a module *and* its new type can validate its own module. A type that already exists is **only reported**; the PR comment lists what would change. | +| **publish** (push) | The full apply. raptor still refuses a change that removes or retypes a field while some module produces or consumes the type, and names those modules. Run `raptor create output-type -f --allow-breaking` yourself when you really mean it. | +| **cleanup** (PR closed) | Nothing. See below. | + +**The action never deletes an output type.** Removing a file from `outputs/` leaves +the type in place; delete it deliberately with +`raptor module delete-output-type @namespace/name --yes`. Two reasons: a type deleted +on a PR-close would break every module that reads it with no preview slot to restore +it from, and a type has no provenance fields, so there is no ownership check like the +one the module cleanup relies on. + +Set `apply-output-types: 'false'` when the output types are managed outside this +repository. ## Requirements @@ -76,9 +117,10 @@ modules live at `infra/modules/...`). | `raptor-download-url` | no | `""` | Exact URL to download the raptor `linux-amd64` binary from, bypassing the default `Facets-cloud/raptor-releases` location. When set, `raptor_version` is ignored. An escape hatch for testing / pre-release raptor builds and enterprise mirrors. | | `terraform-version` | no | `1.5.7` | Terraform version installed (from `releases.hashicorp.com`) for raptor's module validation. Not installed in **cleanup** mode. | | `trivy-version` | no | `0.72.0` | Trivy version installed (from `aquasecurity/trivy` releases, no `v` prefix) for raptor's module security scan. Not installed in **cleanup** mode. | -| `all-modules` | no | `false` | When `true`, operate on **every** module under `modules/` instead of only the ones the event changed. | +| `all-modules` | no | `false` | When `true`, operate on **every** module under `modules/` and **every** output type under `outputs/`, instead of only the ones the event changed. | +| `apply-output-types` | no | `true` | Apply the output type definitions under `outputs/` before touching any module. See **Output types** above. Set to `false` when they are managed outside this repository. | | `mode` | no | `auto` | `auto` \| `preview` \| `publish` \| `cleanup`. `auto` derives the mode from the event (see the table above). Set explicitly to override. | -| `path-prefix` | no | `""` | Sub-path to the `modules/` tree relative to the repo root (e.g. `infra/`). Empty means `modules/` is at the root. | +| `path-prefix` | no | `""` | Sub-path to the `modules/` and `outputs/` trees relative to the repo root (e.g. `infra/`). Empty means both are at the root. | | `auto-create-intents` | no | `true` | Pass `--auto-create` to raptor, so a module whose **intent** the Control Plane has never seen registers it. Without this the upload fails `404 Intent not found`. See **Intents** below before turning it off. | ### Secrets diff --git a/module-ci-action/action.yml b/module-ci-action/action.yml index 6dff594..644fec1 100644 --- a/module-ci-action/action.yml +++ b/module-ci-action/action.yml @@ -33,9 +33,13 @@ inputs: required: false default: "0.72.0" all-modules: - description: "When 'true', operate on every module under modules/ instead of only the ones changed by the triggering event." + description: "When 'true', operate on every module under modules/ and every output type under outputs/, instead of only the ones changed by the triggering event." required: false default: "false" + apply-output-types: + description: "When 'true' (default), apply the output type definitions under outputs/ before touching any module. A module names its output type by reference and the Control Plane refuses a module whose type it does not already hold, so the order matters. Set to 'false' only when the output types are managed outside this repository." + required: false + default: "true" mode: description: "One of auto|preview|publish|cleanup. 'auto' (default) derives the mode from the event: pull_request opened/synchronize/reopened -> preview; push -> publish; pull_request closed -> cleanup." required: false @@ -252,6 +256,140 @@ runs: fi echo "dirs=${DIRS}" >> "$GITHUB_ENV" + - name: Detect target output type definitions + if: (env.MODE == 'preview' || env.MODE == 'publish') && inputs.apply-output-types == 'true' + shell: bash + env: + ALL_MODULES: ${{ inputs.all-modules }} + PATH_PREFIX: ${{ inputs.path-prefix }} + EVENT_NAME: ${{ github.event_name }} + PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} + PUSH_BEFORE: ${{ github.event.before }} + PUSH_AFTER: ${{ github.sha }} + run: | + set -uo pipefail + + PREFIX="${PATH_PREFIX}" + if [ -n "$PREFIX" ] && [ "${PREFIX%/}" = "$PREFIX" ]; then + PREFIX="${PREFIX}/" + fi + OUTPUTS_ROOT="${PREFIX}outputs" + echo "Outputs root: ${OUTPUTS_ROOT}" + + # Same change detection as the modules step — kept as its own copy rather than shared, + # because an output type is ONE FILE, not a directory: there is no facets.yaml to walk + # up to, and a changed file IS the unit of work. + changed_files() { + if [ "$EVENT_NAME" = "push" ]; then + if [ -z "$PUSH_BEFORE" ] || [ "$PUSH_BEFORE" = "0000000000000000000000000000000000000000" ] \ + || ! git rev-parse --verify --quiet "${PUSH_BEFORE}^{commit}" >/dev/null 2>&1; then + git diff-tree --no-commit-id --name-only -r "$PUSH_AFTER" + else + git diff --name-only "${PUSH_BEFORE}..${PUSH_AFTER}" + fi + else + git diff --name-only "${PR_BASE_SHA}...HEAD" + fi + } + + is_definition() { + case "${1,,}" in + *.yaml|*.yml|*.json) return 0 ;; + *) return 1 ;; + esac + } + + FILES="" + if [ "$ALL_MODULES" = "true" ]; then + echo "all-modules=true: scanning ${OUTPUTS_ROOT} for every definition" + if [ -d "$OUTPUTS_ROOT" ]; then + FILES="$(find "$OUTPUTS_ROOT" -type f \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' \) | sort)" + fi + else + while IFS= read -r f; do + [ -z "$f" ] && continue + case "$f/" in + "$OUTPUTS_ROOT"/*) ;; + *) continue ;; + esac + # A deleted file is intentionally NOT a target: CI never deletes an output type. + # See the README — an output type is global and unversioned, so removing its file + # leaves the type in place rather than breaking every module that reads it. + [ -f "$f" ] || continue + is_definition "$f" || continue + FILES="${FILES} ${f}" + done <<< "$(changed_files || true)" + fi + + # shellcheck disable=SC2086 # deliberately re-split the accumulated space-separated list + FILES="$(printf '%s\n' $FILES | sed '/^$/d' | sort -u | tr '\n' ' ' | sed 's/[[:space:]]*$//')" + + if [ -z "$FILES" ]; then + echo "No target output type definitions detected." + else + echo "Target output type definitions:" + for f in $FILES; do echo " - $f"; done + fi + echo "output_files=${FILES}" >> "$GITHUB_ENV" + + - name: Apply output types + if: (env.MODE == 'preview' || env.MODE == 'publish') && env.output_files != '' + shell: bash + env: + CONTROL_PLANE_URL: ${{ inputs.control_plane_url }} + FACETS_USERNAME: ${{ inputs.username }} + FACETS_TOKEN: ${{ inputs.token }} + run: | + set -uo pipefail + FAILURES="${RUNNER_TEMP}/ci_failures" + SECTION="${RUNNER_TEMP}/preview_outputs.md" + + # An output type is GLOBAL and UNVERSIONED — the Control Plane keys it on name+namespace + # only, with no stage and no version, and a write replaces it for every module at once. + # So the two modes differ in kind, not in degree: + # + # preview (pull request): --if-absent. A type the Control Plane has never seen has no + # consumers, so creating it is safe AND necessary — without it, a pull request that + # adds a module together with its new output type cannot validate its own module + # (the upload 404s with "Output @ns/name not found"). A type that already exists is + # only reported, never written: there is no preview slot to hold a change and no + # rollback if it is wrong. + # + # publish (push): the full apply. raptor still refuses a change that removes or retypes + # a field while some module uses the type; that is a real gate, not advice. + MODE_FLAG="" + if [ "$MODE" = "preview" ]; then + MODE_FLAG="--if-absent" + fi + + { + echo "### Facets output types" + echo "" + echo "| Definition | Result |" + echo "|---|---|" + } > "$SECTION" + + # shellcheck disable=SC2086,SC2154 # $output_files is an intentional space-separated list injected via $GITHUB_ENV + for file in $output_files; do + echo "::group::Output type ${file}" + # shellcheck disable=SC2086 # $MODE_FLAG is a deliberate empty-or-one-flag word + if raptor create output-type -f "$file" $MODE_FLAG; then + RESULT="applied" + if [ "$MODE" = "preview" ]; then RESULT="created if absent"; fi + else + echo "::error::Output type apply failed for ${file}" + echo "${file} (output-type)" >> "$FAILURES" + RESULT="failed" + fi + echo "| \`${file}\` | ${RESULT} |" >> "$SECTION" + echo "::endgroup::" + done + + { + echo "" + echo "_On a pull request an output type is only created when the control plane does not have it. A change to an existing type is reported here and applied on merge, because an output type has no version and no preview stage._" + } >> "$SECTION" + - name: Preview modules (pull request) if: env.MODE == 'preview' && env.dirs != '' shell: bash @@ -269,7 +407,7 @@ runs: run: | set -uo pipefail FAILURES="${RUNNER_TEMP}/ci_failures" - COMMENT="${RUNNER_TEMP}/preview_comment.md" + COMMENT="${RUNNER_TEMP}/preview_modules.md" # Read a top-level scalar key (intent/flavor/version) from a module's # facets.yaml. Top-level keys have no leading whitespace; surrounding @@ -280,7 +418,6 @@ runs: } { - echo "" echo "### Facets module preview" echo "" echo "This branch now owns the single **preview slot** for each module below (concurrent PRs on the same module: last push wins)." @@ -343,7 +480,7 @@ runs: } >> "$COMMENT" - name: Upsert preview PR comment - if: env.MODE == 'preview' && env.dirs != '' && inputs.github_token != '' + if: env.MODE == 'preview' && (env.dirs != '' || env.output_files != '') && inputs.github_token != '' shell: bash env: GH_TOKEN: ${{ inputs.github_token }} @@ -352,7 +489,21 @@ runs: run: | set -uo pipefail COMMENT="${RUNNER_TEMP}/preview_comment.md" - if [ ! -f "$COMMENT" ]; then + MODULES_SECTION="${RUNNER_TEMP}/preview_modules.md" + OUTPUTS_SECTION="${RUNNER_TEMP}/preview_outputs.md" + + # Assemble the body here rather than in the producing steps: a PR can touch output types + # only, modules only, or both, and the marker must sit in the final body exactly once. + echo "" > "$COMMENT" + if [ -f "$OUTPUTS_SECTION" ]; then + cat "$OUTPUTS_SECTION" >> "$COMMENT" + echo "" >> "$COMMENT" + fi + if [ -f "$MODULES_SECTION" ]; then + cat "$MODULES_SECTION" >> "$COMMENT" + fi + + if [ ! -f "$MODULES_SECTION" ] && [ ! -f "$OUTPUTS_SECTION" ]; then echo "No preview comment body found; nothing to post." exit 0 fi