Skip to content

feat(module-ci): apply output types before modules #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anujhydrabadi merged 1 commit into from
Sep 15, 2026
Merged

feat(module-ci): apply output types before modules #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,23 @@ Releases are semver tags with a moving major alias (`v1` always points at the la
## 1. Facets Module CI (current)

The **Facets Module CI** GitHub Action provides end-to-end CI for a Facets **modules
repository** (`modules/{intent}/{flavor}/{version}`) using the **`raptor`** CLI. It runs
repository** — the modules at `modules/{intent}/{flavor}/{version}` and the output types
they exchange at `outputs/{namespace}/{name}.yaml` — using the **`raptor`** CLI. It runs
in three modes, derived automatically from the triggering event:

[module-ci-action README](./module-ci-action/README.md).

- **Preview (on pull request)**: Validates each changed module, then registers an
unpublishable feature-branch preview pinned to the PR head commit (optionally posts a
PR comment).
- **Publish (on push)**: Uploads and publishes each changed module (PREVIEW → PUBLISHED).
- **Preview (on pull request)**: Creates each changed output type the control plane does
not have yet, then validates each changed module and registers an unpublishable
feature-branch preview pinned to the PR head commit (optionally posts a PR comment).
- **Publish (on push)**: Applies each changed output type, then uploads and publishes each
changed module (PREVIEW → PUBLISHED).
- **Cleanup (on pull request close)**: Deletes each changed module's preview, but only
the previews this PR's own commits created (ownership-checked).
the previews this PR's own commits created (ownership-checked). Output types are never
deleted — they are global and unversioned.

Output types are always applied **before** any module: a module names its output type by
reference, and the control plane refuses a module whose type it does not already hold.

This is the action the Facets control plane wires into bootstrapped modules
repositories, and the recommended CI for all modules repos.
Expand Down
56 changes: 49 additions & 7 deletions module-ci-action/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ triggering event:

| Event | Mode | What it does |
|-------|------|--------------|
| `pull_request` opened / synchronize / reopened | **preview** | Validates each changed module, then registers a **feature-branch preview** pinned to the PR head commit. Optionally upserts a PR comment. |
| `push` (to your default branch) | **publish** | Uploads each changed module and **publishes** it (PREVIEW → PUBLISHED). |
| `pull_request` opened / synchronize / reopened | **preview** | Creates each changed output type the Control Plane does not have yet, then validates each changed module and registers a **feature-branch preview** pinned to the PR head commit. Optionally upserts a PR comment. |
| `push` (to your default branch) | **publish** | Applies each changed output type, then uploads each changed module and **publishes** it (PREVIEW → PUBLISHED). |
| `pull_request` **closed without merge** | **cleanup** | Deletes the preview each changed module owns — **only if** the preview belongs to a commit from this PR. |
| `pull_request` **closed by merge** | **no-op** | Skipped: the concurrent `push` (publish) run re-uploads and publishes the module at the merge commit, so it owns the slot. Running cleanup here would be redundant and could race the publish. |

Expand All @@ -20,7 +20,8 @@ triggering event:

## Modules-repo layout

Each module lives at a versioned path and contains a `facets.yaml` plus its Terraform:
A modules repository holds two trees: the modules, and the **output types** they
exchange.

```
modules/
Expand All @@ -31,14 +32,54 @@ modules/
main.tf
variables.tf
outputs.tf
outputs/
<namespace>/
<name>.yaml # name: "@namespace/name", properties:, providers:
```

The action discovers a module as **any directory under `modules/` that contains a
`facets.yaml`**. `intent`, `flavor`, and `version` are read from that `facets.yaml`
and form the module reference `intent/flavor/version` used by `raptor`.

If your modules tree is not at the repo root, set `path-prefix` (e.g. `infra/` when
modules live at `infra/modules/...`).
It discovers an output type as **any `.yaml`, `.yml` or `.json` file under
`outputs/`**. Each file names itself through its `name` field; the directory layout is
a convention, not a lookup key. `raptor get output-type @namespace/name -o yaml`
prints exactly this shape, so an existing type can be saved to a file, edited, and
committed.

If your trees are not at the repo root, set `path-prefix` (e.g. `infra/` when they
live at `infra/modules/...` and `infra/outputs/...`).

## Output types

A module names its output type by reference (`@namespace/name`). The Control Plane
**refuses a module upload whose output type it does not already hold**
(`Output @ns/name not found`), so the action always applies the output types first,
in both preview and publish mode.

An output type is different from a module in one way that drives everything here: it
is **global and unversioned**. The Control Plane keys it on `name` + `namespace`
alone — there is no preview stage, no version, and a write replaces it for every
module and every environment at once. There is no slot to hold a proposed change and
nothing to roll back to.

So the two modes are not the same operation at different strengths:

| Mode | What happens to an output type |
|------|-------------------------------|
| **preview** (pull request) | `raptor create output-type -f <file> --if-absent`. A type the Control Plane has never seen is **created** — it has no consumers, so the write cannot break anything, and it is the only way a PR that adds a module *and* its new type can validate its own module. A type that already exists is **only reported**; the PR comment lists what would change. |
| **publish** (push) | The full apply. raptor still refuses a change that removes or retypes a field while some module produces or consumes the type, and names those modules. Run `raptor create output-type -f <file> --allow-breaking` yourself when you really mean it. |
| **cleanup** (PR closed) | Nothing. See below. |

**The action never deletes an output type.** Removing a file from `outputs/` leaves
the type in place; delete it deliberately with
`raptor module delete-output-type @namespace/name --yes`. Two reasons: a type deleted
on a PR-close would break every module that reads it with no preview slot to restore
it from, and a type has no provenance fields, so there is no ownership check like the
one the module cleanup relies on.

Set `apply-output-types: 'false'` when the output types are managed outside this
repository.

## Requirements

Expand Down Expand Up @@ -76,9 +117,10 @@ modules live at `infra/modules/...`).
| `raptor-download-url` | no | `""` | Exact URL to download the raptor `linux-amd64` binary from, bypassing the default `Facets-cloud/raptor-releases` location. When set, `raptor_version` is ignored. An escape hatch for testing / pre-release raptor builds and enterprise mirrors. |
| `terraform-version` | no | `1.5.7` | Terraform version installed (from `releases.hashicorp.com`) for raptor's module validation. Not installed in **cleanup** mode. |
| `trivy-version` | no | `0.72.0` | Trivy version installed (from `aquasecurity/trivy` releases, no `v` prefix) for raptor's module security scan. Not installed in **cleanup** mode. |
| `all-modules` | no | `false` | When `true`, operate on **every** module under `<path-prefix>modules/` instead of only the ones the event changed. |
| `all-modules` | no | `false` | When `true`, operate on **every** module under `<path-prefix>modules/` and **every** output type under `<path-prefix>outputs/`, instead of only the ones the event changed. |
| `apply-output-types` | no | `true` | Apply the output type definitions under `<path-prefix>outputs/` before touching any module. See **Output types** above. Set to `false` when they are managed outside this repository. |
| `mode` | no | `auto` | `auto` \| `preview` \| `publish` \| `cleanup`. `auto` derives the mode from the event (see the table above). Set explicitly to override. |
| `path-prefix` | no | `""` | Sub-path to the `modules/` tree relative to the repo root (e.g. `infra/`). Empty means `modules/` is at the root. |
| `path-prefix` | no | `""` | Sub-path to the `modules/` and `outputs/` trees relative to the repo root (e.g. `infra/`). Empty means both are at the root. |
| `auto-create-intents` | no | `true` | Pass `--auto-create` to raptor, so a module whose **intent** the Control Plane has never seen registers it. Without this the upload fails `404 Intent <kind> not found`. See **Intents** below before turning it off. |

### Secrets
Expand Down
161 changes: 156 additions & 5 deletions module-ci-action/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,13 @@ inputs:
required: false
default: "0.72.0"
all-modules:
description: "When 'true', operate on every module under <path-prefix>modules/ instead of only the ones changed by the triggering event."
description: "When 'true', operate on every module under <path-prefix>modules/ and every output type under <path-prefix>outputs/, instead of only the ones changed by the triggering event."
required: false
default: "false"
apply-output-types:
description: "When 'true' (default), apply the output type definitions under <path-prefix>outputs/ before touching any module. A module names its output type by reference and the Control Plane refuses a module whose type it does not already hold, so the order matters. Set to 'false' only when the output types are managed outside this repository."
required: false
default: "true"
mode:
description: "One of auto|preview|publish|cleanup. 'auto' (default) derives the mode from the event: pull_request opened/synchronize/reopened -> preview; push -> publish; pull_request closed -> cleanup."
required: false
Expand Down Expand Up @@ -252,6 +256,140 @@ runs:
fi
echo "dirs=${DIRS}" >> "$GITHUB_ENV"

- name: Detect target output type definitions
if: (env.MODE == 'preview' || env.MODE == 'publish') && inputs.apply-output-types == 'true'
shell: bash
env:
ALL_MODULES: ${{ inputs.all-modules }}
PATH_PREFIX: ${{ inputs.path-prefix }}
EVENT_NAME: ${{ github.event_name }}
PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
PUSH_BEFORE: ${{ github.event.before }}
PUSH_AFTER: ${{ github.sha }}
run: |
set -uo pipefail

PREFIX="${PATH_PREFIX}"
if [ -n "$PREFIX" ] && [ "${PREFIX%/}" = "$PREFIX" ]; then
PREFIX="${PREFIX}/"
fi
OUTPUTS_ROOT="${PREFIX}outputs"
echo "Outputs root: ${OUTPUTS_ROOT}"

# Same change detection as the modules step — kept as its own copy rather than shared,
# because an output type is ONE FILE, not a directory: there is no facets.yaml to walk
# up to, and a changed file IS the unit of work.
changed_files() {
if [ "$EVENT_NAME" = "push" ]; then
if [ -z "$PUSH_BEFORE" ] || [ "$PUSH_BEFORE" = "0000000000000000000000000000000000000000" ] \
|| ! git rev-parse --verify --quiet "${PUSH_BEFORE}^{commit}" >/dev/null 2>&1; then
git diff-tree --no-commit-id --name-only -r "$PUSH_AFTER"
else
git diff --name-only "${PUSH_BEFORE}..${PUSH_AFTER}"
fi
else
git diff --name-only "${PR_BASE_SHA}...HEAD"
fi
}

is_definition() {
case "${1,,}" in
*.yaml|*.yml|*.json) return 0 ;;
*) return 1 ;;
esac
}

FILES=""
if [ "$ALL_MODULES" = "true" ]; then
echo "all-modules=true: scanning ${OUTPUTS_ROOT} for every definition"
if [ -d "$OUTPUTS_ROOT" ]; then
FILES="$(find "$OUTPUTS_ROOT" -type f \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' \) | sort)"
fi
else
while IFS= read -r f; do
[ -z "$f" ] && continue
case "$f/" in
"$OUTPUTS_ROOT"/*) ;;
*) continue ;;
esac
# A deleted file is intentionally NOT a target: CI never deletes an output type.
# See the README — an output type is global and unversioned, so removing its file
# leaves the type in place rather than breaking every module that reads it.
[ -f "$f" ] || continue
is_definition "$f" || continue
FILES="${FILES} ${f}"
done <<< "$(changed_files || true)"
fi

# shellcheck disable=SC2086 # deliberately re-split the accumulated space-separated list
FILES="$(printf '%s\n' $FILES | sed '/^$/d' | sort -u | tr '\n' ' ' | sed 's/[[:space:]]*$//')"

if [ -z "$FILES" ]; then
echo "No target output type definitions detected."
else
echo "Target output type definitions:"
for f in $FILES; do echo " - $f"; done
fi
echo "output_files=${FILES}" >> "$GITHUB_ENV"

- name: Apply output types
if: (env.MODE == 'preview' || env.MODE == 'publish') && env.output_files != ''
shell: bash
env:
CONTROL_PLANE_URL: ${{ inputs.control_plane_url }}
FACETS_USERNAME: ${{ inputs.username }}
FACETS_TOKEN: ${{ inputs.token }}
run: |
set -uo pipefail
FAILURES="${RUNNER_TEMP}/ci_failures"
SECTION="${RUNNER_TEMP}/preview_outputs.md"

# An output type is GLOBAL and UNVERSIONED — the Control Plane keys it on name+namespace
# only, with no stage and no version, and a write replaces it for every module at once.
# So the two modes differ in kind, not in degree:
#
# preview (pull request): --if-absent. A type the Control Plane has never seen has no
# consumers, so creating it is safe AND necessary — without it, a pull request that
# adds a module together with its new output type cannot validate its own module
# (the upload 404s with "Output @ns/name not found"). A type that already exists is
# only reported, never written: there is no preview slot to hold a change and no
# rollback if it is wrong.
#
# publish (push): the full apply. raptor still refuses a change that removes or retypes
# a field while some module uses the type; that is a real gate, not advice.
MODE_FLAG=""
if [ "$MODE" = "preview" ]; then
MODE_FLAG="--if-absent"
fi

{
echo "### Facets output types"
echo ""
echo "| Definition | Result |"
echo "|---|---|"
} > "$SECTION"

# shellcheck disable=SC2086,SC2154 # $output_files is an intentional space-separated list injected via $GITHUB_ENV
for file in $output_files; do
echo "::group::Output type ${file}"
# shellcheck disable=SC2086 # $MODE_FLAG is a deliberate empty-or-one-flag word
if raptor create output-type -f "$file" $MODE_FLAG; then
RESULT="applied"
if [ "$MODE" = "preview" ]; then RESULT="created if absent"; fi
else
echo "::error::Output type apply failed for ${file}"
echo "${file} (output-type)" >> "$FAILURES"
RESULT="failed"
fi
echo "| \`${file}\` | ${RESULT} |" >> "$SECTION"
echo "::endgroup::"
done

{
echo ""
echo "_On a pull request an output type is only created when the control plane does not have it. A change to an existing type is reported here and applied on merge, because an output type has no version and no preview stage._"
} >> "$SECTION"

- name: Preview modules (pull request)
if: env.MODE == 'preview' && env.dirs != ''
shell: bash
Expand All @@ -269,7 +407,7 @@ runs:
run: |
set -uo pipefail
FAILURES="${RUNNER_TEMP}/ci_failures"
COMMENT="${RUNNER_TEMP}/preview_comment.md"
COMMENT="${RUNNER_TEMP}/preview_modules.md"

# Read a top-level scalar key (intent/flavor/version) from a module's
# facets.yaml. Top-level keys have no leading whitespace; surrounding
Expand All @@ -280,7 +418,6 @@ runs:
}

{
echo "<!-- facets-module-ci:preview -->"
echo "### Facets module preview"
echo ""
echo "This branch now owns the single **preview slot** for each module below (concurrent PRs on the same module: last push wins)."
Expand Down Expand Up @@ -343,7 +480,7 @@ runs:
} >> "$COMMENT"

- name: Upsert preview PR comment
if: env.MODE == 'preview' && env.dirs != '' && inputs.github_token != ''
if: env.MODE == 'preview' && (env.dirs != '' || env.output_files != '') && inputs.github_token != ''
shell: bash
env:
GH_TOKEN: ${{ inputs.github_token }}
Expand All @@ -352,7 +489,21 @@ runs:
run: |
set -uo pipefail
COMMENT="${RUNNER_TEMP}/preview_comment.md"
if [ ! -f "$COMMENT" ]; then
MODULES_SECTION="${RUNNER_TEMP}/preview_modules.md"
OUTPUTS_SECTION="${RUNNER_TEMP}/preview_outputs.md"

# Assemble the body here rather than in the producing steps: a PR can touch output types
# only, modules only, or both, and the marker must sit in the final body exactly once.
echo "<!-- facets-module-ci:preview -->" > "$COMMENT"
if [ -f "$OUTPUTS_SECTION" ]; then
cat "$OUTPUTS_SECTION" >> "$COMMENT"
echo "" >> "$COMMENT"
fi
if [ -f "$MODULES_SECTION" ]; then
cat "$MODULES_SECTION" >> "$COMMENT"
fi

if [ ! -f "$MODULES_SECTION" ] && [ ! -f "$OUTPUTS_SECTION" ]; then
echo "No preview comment body found; nothing to post."
exit 0
fi
Expand Down