diff --git a/README.md b/README.md index 7371f4f..02cc540 100644 --- a/README.md +++ b/README.md @@ -7,17 +7,23 @@ Releases are semver tags with a moving major alias (`v1` always points at the la ## 1. Facets Module CI (current) The **Facets Module CI** GitHub Action provides end-to-end CI for a Facets **modules -repository** (`modules/{intent}/{flavor}/{version}`) using the **`raptor`** CLI. It runs +repository** — the modules at `modules/{intent}/{flavor}/{version}` and the output types +they exchange at `outputs/{namespace}/{name}.yaml` — using the **`raptor`** CLI. It runs in three modes, derived automatically from the triggering event: [module-ci-action README](./module-ci-action/README.md). -- **Preview (on pull request)**: Validates each changed module, then registers an - unpublishable feature-branch preview pinned to the PR head commit (optionally posts a - PR comment). -- **Publish (on push)**: Uploads and publishes each changed module (PREVIEW → PUBLISHED). +- **Preview (on pull request)**: Creates each changed output type the control plane does + not have yet, then validates each changed module and registers an unpublishable + feature-branch preview pinned to the PR head commit (optionally posts a PR comment). +- **Publish (on push)**: Applies each changed output type, then uploads and publishes each + changed module (PREVIEW → PUBLISHED). - **Cleanup (on pull request close)**: Deletes each changed module's preview, but only - the previews this PR's own commits created (ownership-checked). + the previews this PR's own commits created (ownership-checked). Output types are never + deleted — they are global and unversioned. + +Output types are always applied **before** any module: a module names its output type by +reference, and the control plane refuses a module whose type it does not already hold. This is the action the Facets control plane wires into bootstrapped modules repositories, and the recommended CI for all modules repos. diff --git a/module-ci-action/README.md b/module-ci-action/README.md index fdb72f4..8b02796 100644 --- a/module-ci-action/README.md +++ b/module-ci-action/README.md @@ -9,8 +9,8 @@ triggering event: | Event | Mode | What it does | |-------|------|--------------| -| `pull_request` opened / synchronize / reopened | **preview** | Validates each changed module, then registers a **feature-branch preview** pinned to the PR head commit. Optionally upserts a PR comment. | -| `push` (to your default branch) | **publish** | Uploads each changed module and **publishes** it (PREVIEW → PUBLISHED). | +| `pull_request` opened / synchronize / reopened | **preview** | Creates each changed output type the Control Plane does not have yet, then validates each changed module and registers a **feature-branch preview** pinned to the PR head commit. Optionally upserts a PR comment. | +| `push` (to your default branch) | **publish** | Applies each changed output type, then uploads each changed module and **publishes** it (PREVIEW → PUBLISHED). | | `pull_request` **closed without merge** | **cleanup** | Deletes the preview each changed module owns — **only if** the preview belongs to a commit from this PR. | | `pull_request` **closed by merge** | **no-op** | Skipped: the concurrent `push` (publish) run re-uploads and publishes the module at the merge commit, so it owns the slot. Running cleanup here would be redundant and could race the publish. | @@ -20,7 +20,8 @@ triggering event: ## Modules-repo layout -Each module lives at a versioned path and contains a `facets.yaml` plus its Terraform: +A modules repository holds two trees: the modules, and the **output types** they +exchange. ``` modules/ @@ -31,14 +32,54 @@ modules/ main.tf variables.tf outputs.tf +outputs/ + / + .yaml # name: "@namespace/name", properties:, providers: ``` The action discovers a module as **any directory under `modules/` that contains a `facets.yaml`**. `intent`, `flavor`, and `version` are read from that `facets.yaml` and form the module reference `intent/flavor/version` used by `raptor`. -If your modules tree is not at the repo root, set `path-prefix` (e.g. `infra/` when -modules live at `infra/modules/...`). +It discovers an output type as **any `.yaml`, `.yml` or `.json` file under +`outputs/`**. Each file names itself through its `name` field; the directory layout is +a convention, not a lookup key. `raptor get output-type @namespace/name -o yaml` +prints exactly this shape, so an existing type can be saved to a file, edited, and +committed. + +If your trees are not at the repo root, set `path-prefix` (e.g. `infra/` when they +live at `infra/modules/...` and `infra/outputs/...`). + +## Output types + +A module names its output type by reference (`@namespace/name`). The Control Plane +**refuses a module upload whose output type it does not already hold** +(`Output @ns/name not found`), so the action always applies the output types first, +in both preview and publish mode. + +An output type is different from a module in one way that drives everything here: it +is **global and unversioned**. The Control Plane keys it on `name` + `namespace` +alone — there is no preview stage, no version, and a write replaces it for every +module and every environment at once. There is no slot to hold a proposed change and +nothing to roll back to. + +So the two modes are not the same operation at different strengths: + +| Mode | What happens to an output type | +|------|-------------------------------| +| **preview** (pull request) | `raptor create output-type -f --if-absent`. A type the Control Plane has never seen is **created** — it has no consumers, so the write cannot break anything, and it is the only way a PR that adds a module *and* its new type can validate its own module. A type that already exists is **only reported**; the PR comment lists what would change. | +| **publish** (push) | The full apply. raptor still refuses a change that removes or retypes a field while some module produces or consumes the type, and names those modules. Run `raptor create output-type -f --allow-breaking` yourself when you really mean it. | +| **cleanup** (PR closed) | Nothing. See below. | + +**The action never deletes an output type.** Removing a file from `outputs/` leaves +the type in place; delete it deliberately with +`raptor module delete-output-type @namespace/name --yes`. Two reasons: a type deleted +on a PR-close would break every module that reads it with no preview slot to restore +it from, and a type has no provenance fields, so there is no ownership check like the +one the module cleanup relies on. + +Set `apply-output-types: 'false'` when the output types are managed outside this +repository. ## Requirements @@ -76,9 +117,10 @@ modules live at `infra/modules/...`). | `raptor-download-url` | no | `""` | Exact URL to download the raptor `linux-amd64` binary from, bypassing the default `Facets-cloud/raptor-releases` location. When set, `raptor_version` is ignored. An escape hatch for testing / pre-release raptor builds and enterprise mirrors. | | `terraform-version` | no | `1.5.7` | Terraform version installed (from `releases.hashicorp.com`) for raptor's module validation. Not installed in **cleanup** mode. | | `trivy-version` | no | `0.72.0` | Trivy version installed (from `aquasecurity/trivy` releases, no `v` prefix) for raptor's module security scan. Not installed in **cleanup** mode. | -| `all-modules` | no | `false` | When `true`, operate on **every** module under `modules/` instead of only the ones the event changed. | +| `all-modules` | no | `false` | When `true`, operate on **every** module under `modules/` and **every** output type under `outputs/`, instead of only the ones the event changed. | +| `apply-output-types` | no | `true` | Apply the output type definitions under `outputs/` before touching any module. See **Output types** above. Set to `false` when they are managed outside this repository. | | `mode` | no | `auto` | `auto` \| `preview` \| `publish` \| `cleanup`. `auto` derives the mode from the event (see the table above). Set explicitly to override. | -| `path-prefix` | no | `""` | Sub-path to the `modules/` tree relative to the repo root (e.g. `infra/`). Empty means `modules/` is at the root. | +| `path-prefix` | no | `""` | Sub-path to the `modules/` and `outputs/` trees relative to the repo root (e.g. `infra/`). Empty means both are at the root. | | `auto-create-intents` | no | `true` | Pass `--auto-create` to raptor, so a module whose **intent** the Control Plane has never seen registers it. Without this the upload fails `404 Intent not found`. See **Intents** below before turning it off. | ### Secrets diff --git a/module-ci-action/action.yml b/module-ci-action/action.yml index 6dff594..644fec1 100644 --- a/module-ci-action/action.yml +++ b/module-ci-action/action.yml @@ -33,9 +33,13 @@ inputs: required: false default: "0.72.0" all-modules: - description: "When 'true', operate on every module under modules/ instead of only the ones changed by the triggering event." + description: "When 'true', operate on every module under modules/ and every output type under outputs/, instead of only the ones changed by the triggering event." required: false default: "false" + apply-output-types: + description: "When 'true' (default), apply the output type definitions under outputs/ before touching any module. A module names its output type by reference and the Control Plane refuses a module whose type it does not already hold, so the order matters. Set to 'false' only when the output types are managed outside this repository." + required: false + default: "true" mode: description: "One of auto|preview|publish|cleanup. 'auto' (default) derives the mode from the event: pull_request opened/synchronize/reopened -> preview; push -> publish; pull_request closed -> cleanup." required: false @@ -252,6 +256,140 @@ runs: fi echo "dirs=${DIRS}" >> "$GITHUB_ENV" + - name: Detect target output type definitions + if: (env.MODE == 'preview' || env.MODE == 'publish') && inputs.apply-output-types == 'true' + shell: bash + env: + ALL_MODULES: ${{ inputs.all-modules }} + PATH_PREFIX: ${{ inputs.path-prefix }} + EVENT_NAME: ${{ github.event_name }} + PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} + PUSH_BEFORE: ${{ github.event.before }} + PUSH_AFTER: ${{ github.sha }} + run: | + set -uo pipefail + + PREFIX="${PATH_PREFIX}" + if [ -n "$PREFIX" ] && [ "${PREFIX%/}" = "$PREFIX" ]; then + PREFIX="${PREFIX}/" + fi + OUTPUTS_ROOT="${PREFIX}outputs" + echo "Outputs root: ${OUTPUTS_ROOT}" + + # Same change detection as the modules step — kept as its own copy rather than shared, + # because an output type is ONE FILE, not a directory: there is no facets.yaml to walk + # up to, and a changed file IS the unit of work. + changed_files() { + if [ "$EVENT_NAME" = "push" ]; then + if [ -z "$PUSH_BEFORE" ] || [ "$PUSH_BEFORE" = "0000000000000000000000000000000000000000" ] \ + || ! git rev-parse --verify --quiet "${PUSH_BEFORE}^{commit}" >/dev/null 2>&1; then + git diff-tree --no-commit-id --name-only -r "$PUSH_AFTER" + else + git diff --name-only "${PUSH_BEFORE}..${PUSH_AFTER}" + fi + else + git diff --name-only "${PR_BASE_SHA}...HEAD" + fi + } + + is_definition() { + case "${1,,}" in + *.yaml|*.yml|*.json) return 0 ;; + *) return 1 ;; + esac + } + + FILES="" + if [ "$ALL_MODULES" = "true" ]; then + echo "all-modules=true: scanning ${OUTPUTS_ROOT} for every definition" + if [ -d "$OUTPUTS_ROOT" ]; then + FILES="$(find "$OUTPUTS_ROOT" -type f \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' \) | sort)" + fi + else + while IFS= read -r f; do + [ -z "$f" ] && continue + case "$f/" in + "$OUTPUTS_ROOT"/*) ;; + *) continue ;; + esac + # A deleted file is intentionally NOT a target: CI never deletes an output type. + # See the README — an output type is global and unversioned, so removing its file + # leaves the type in place rather than breaking every module that reads it. + [ -f "$f" ] || continue + is_definition "$f" || continue + FILES="${FILES} ${f}" + done <<< "$(changed_files || true)" + fi + + # shellcheck disable=SC2086 # deliberately re-split the accumulated space-separated list + FILES="$(printf '%s\n' $FILES | sed '/^$/d' | sort -u | tr '\n' ' ' | sed 's/[[:space:]]*$//')" + + if [ -z "$FILES" ]; then + echo "No target output type definitions detected." + else + echo "Target output type definitions:" + for f in $FILES; do echo " - $f"; done + fi + echo "output_files=${FILES}" >> "$GITHUB_ENV" + + - name: Apply output types + if: (env.MODE == 'preview' || env.MODE == 'publish') && env.output_files != '' + shell: bash + env: + CONTROL_PLANE_URL: ${{ inputs.control_plane_url }} + FACETS_USERNAME: ${{ inputs.username }} + FACETS_TOKEN: ${{ inputs.token }} + run: | + set -uo pipefail + FAILURES="${RUNNER_TEMP}/ci_failures" + SECTION="${RUNNER_TEMP}/preview_outputs.md" + + # An output type is GLOBAL and UNVERSIONED — the Control Plane keys it on name+namespace + # only, with no stage and no version, and a write replaces it for every module at once. + # So the two modes differ in kind, not in degree: + # + # preview (pull request): --if-absent. A type the Control Plane has never seen has no + # consumers, so creating it is safe AND necessary — without it, a pull request that + # adds a module together with its new output type cannot validate its own module + # (the upload 404s with "Output @ns/name not found"). A type that already exists is + # only reported, never written: there is no preview slot to hold a change and no + # rollback if it is wrong. + # + # publish (push): the full apply. raptor still refuses a change that removes or retypes + # a field while some module uses the type; that is a real gate, not advice. + MODE_FLAG="" + if [ "$MODE" = "preview" ]; then + MODE_FLAG="--if-absent" + fi + + { + echo "### Facets output types" + echo "" + echo "| Definition | Result |" + echo "|---|---|" + } > "$SECTION" + + # shellcheck disable=SC2086,SC2154 # $output_files is an intentional space-separated list injected via $GITHUB_ENV + for file in $output_files; do + echo "::group::Output type ${file}" + # shellcheck disable=SC2086 # $MODE_FLAG is a deliberate empty-or-one-flag word + if raptor create output-type -f "$file" $MODE_FLAG; then + RESULT="applied" + if [ "$MODE" = "preview" ]; then RESULT="created if absent"; fi + else + echo "::error::Output type apply failed for ${file}" + echo "${file} (output-type)" >> "$FAILURES" + RESULT="failed" + fi + echo "| \`${file}\` | ${RESULT} |" >> "$SECTION" + echo "::endgroup::" + done + + { + echo "" + echo "_On a pull request an output type is only created when the control plane does not have it. A change to an existing type is reported here and applied on merge, because an output type has no version and no preview stage._" + } >> "$SECTION" + - name: Preview modules (pull request) if: env.MODE == 'preview' && env.dirs != '' shell: bash @@ -269,7 +407,7 @@ runs: run: | set -uo pipefail FAILURES="${RUNNER_TEMP}/ci_failures" - COMMENT="${RUNNER_TEMP}/preview_comment.md" + COMMENT="${RUNNER_TEMP}/preview_modules.md" # Read a top-level scalar key (intent/flavor/version) from a module's # facets.yaml. Top-level keys have no leading whitespace; surrounding @@ -280,7 +418,6 @@ runs: } { - echo "" echo "### Facets module preview" echo "" echo "This branch now owns the single **preview slot** for each module below (concurrent PRs on the same module: last push wins)." @@ -343,7 +480,7 @@ runs: } >> "$COMMENT" - name: Upsert preview PR comment - if: env.MODE == 'preview' && env.dirs != '' && inputs.github_token != '' + if: env.MODE == 'preview' && (env.dirs != '' || env.output_files != '') && inputs.github_token != '' shell: bash env: GH_TOKEN: ${{ inputs.github_token }} @@ -352,7 +489,21 @@ runs: run: | set -uo pipefail COMMENT="${RUNNER_TEMP}/preview_comment.md" - if [ ! -f "$COMMENT" ]; then + MODULES_SECTION="${RUNNER_TEMP}/preview_modules.md" + OUTPUTS_SECTION="${RUNNER_TEMP}/preview_outputs.md" + + # Assemble the body here rather than in the producing steps: a PR can touch output types + # only, modules only, or both, and the marker must sit in the final body exactly once. + echo "" > "$COMMENT" + if [ -f "$OUTPUTS_SECTION" ]; then + cat "$OUTPUTS_SECTION" >> "$COMMENT" + echo "" >> "$COMMENT" + fi + if [ -f "$MODULES_SECTION" ]; then + cat "$MODULES_SECTION" >> "$COMMENT" + fi + + if [ ! -f "$MODULES_SECTION" ] && [ ! -f "$OUTPUTS_SECTION" ]; then echo "No preview comment body found; nothing to post." exit 0 fi