Document cpflow workflow testing

.controlplane/docs/testing-cpflow-github-actions.md

@@ -0,0 +1,140 @@
+# Testing cpflow GitHub Actions Changes
+
+Use this guide when changing generated `cpflow-*` GitHub Actions, updating the
+`cpflow` generator version, or debugging review-app automation.
+
+## What To Test
+
+Test the flow in three layers:
+
+1. Local generated-file checks catch YAML, metadata, and lint problems before a PR.
+2. GitHub workflow checks prove GitHub can load the workflow and run CI.
+3. A real review-app deploy proves the default-branch trusted actions, GitHub
+   secrets, Docker build, and Control Plane deploy all work together.
+
+The third layer matters because the review-app workflow intentionally checks out
+trusted workflow sources from the repository default branch before passing
+Control Plane secrets to local composite actions. A PR branch can contain fixed
+`.github/actions/*` files, but the deploy job still loads those local actions
+from `master` until the fix is merged there.
+
+## Local Checks
+
+After regenerating the flow, run these checks from the repository root. If
+`cpflow` is installed as a gem, use `cpflow` directly:
+
+```sh
+bin/conductor-exec cpflow generate-github-actions --staging-branch master
+bin/test-cpflow-github-flow
+```
+
+When testing an unreleased upstream `control-plane-flow` checkout, replace
+`cpflow` with that checkout's `bin/cpflow`:
+
+```sh
+bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
+bin/test-cpflow-github-flow ruby /path/to/control-plane-flow/bin/cpflow
+```
+
+Why the explicit description check exists: GitHub parses expression-like snippets
+inside composite action metadata, including `description:` fields. Literal
+examples such as `${{ vars.SOME_VALUE }}` can fail action loading before any
+shell step starts. The wrapper runs `cpflow github-flow-readiness`, parses the
+generated YAML, checks action input descriptions for literal GitHub expressions,
+and runs `actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml`.
+
+## PR Checks
+
+Open a normal PR for the generated-file diff and wait for CI. The workflow PR
+itself is useful for syntax and CI validation, but it does not fully prove
+review-app deployment changes that live under `.github/actions/`.
+
+For top-level workflow edits, you can manually dispatch the PR branch workflow:
+
+```sh
+gh workflow run cpflow-deploy-review-app.yml --ref <branch> -f pr_number=<pr-number>
+```
+
+This loads the workflow file from `<branch>`, but the deploy workflow's
+`Checkout trusted workflow sources` step still checks out `master` before using
+local composite actions with secrets. Treat this as a partial smoke test, not as
+proof that PR-branch composite action changes work.
+
+## Post-Merge Review-App Test
+
+After the workflow PR merges to `master`, test a real review-app deployment:
+
+1. Pick a same-repository PR to use as the canary.
+2. If the review app does not exist yet, comment exactly `+review-app-deploy` on
+   that PR.
+3. If a previous deploy run failed, rerun the failed deploy run after the
+   workflow PR is merged.
+4. Confirm the deploy run checks out `master` at the merge commit in
+   `Checkout trusted workflow sources`.
+5. Confirm `Setup environment` succeeds and prints the expected `cpflow` version.
+6. Confirm `Check if review app exists`, `Build Docker image`, and
+   `Deploy to Control Plane` all run as expected.
+7. Open the review-app URL from the PR comment or deployment status and verify
+   it returns HTTP 200.
+
+Use the generated app name from the workflow log:
+
+```text
+APP_NAME: ${REVIEW_APP_PREFIX}-${PR_NUMBER}
+```
+
+This is a template from the workflow output, not a literal command to evaluate
+unless those environment variables are already set. For this repo, verify the
+actual `REVIEW_APP_PREFIX` repository variable before assuming the final app
+name.
+
+## Troubleshooting Signals
+
+### Composite action metadata fails before setup
+
+Error shape:
+
+```text
+Unrecognized named-value: 'vars'
+Failed to load ./.github/actions/cpflow-setup-environment/action.yml
+```
+
+Cause: GitHub parsed a literal expression inside composite action metadata,
+usually an input description. Because trusted local actions come from `master`,
+fix and merge the generated action metadata on `master`, then rerun the deploy.
+
+### Setup succeeds, then `cpflow exists` reports token format
+
+Error shape:
+
+```text
+ERROR: Unknown API token format. Please re-run 'cpln profile login' or set the correct CPLN_TOKEN env variable.
+```
+
+Cause: the workflow can read `CPLN_TOKEN_STAGING`, but the value is not a valid
+Control Plane service-account token for the installed Control Plane CLI. Rotate
+the GitHub secret, then rerun the failed deploy job.
+
+### PR pushes do not create a new review app
+
+This is expected. Pushes redeploy only after the review app already exists.
+Create the first review app by commenting exactly:
+
+```text
++review-app-deploy
+```
+
+## Ways To Make This Easier
+
+- Add a no-secret GitHub Actions smoke workflow that loads generated local
+  composite actions from the PR branch and fails fast on action metadata parsing.
+- Extend `bin/test-cpflow-github-flow` as more local cpflow GitHub Actions
+  checks become worth standardizing.
+- Add an early token sanity step after `Setup environment` so invalid
+  `CPLN_TOKEN_STAGING` and `CPLN_TOKEN_PRODUCTION` values fail with a named
+  "validate Control Plane token" step instead of surfacing later during
+  `cpflow exists`.
+- Keep a tiny canary PR open for review-app workflow testing so post-merge
+  deploy verification does not depend on whichever feature PR happens to exist.
+- Upstream the metadata-description check to `cpflow github-flow-readiness` so
+  downstream repos get the guard automatically.

.controlplane/readme.md

@@ -424,24 +424,33 @@ repository variables, secrets, and docs aligned with `.controlplane/controlplane
 For this app, validate a regenerated flow with:
 
 ```bash
-bundle exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
-bundle exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
-actionlint .github/workflows/cpflow-*.yml
-bundle exec rubocop
+bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
+bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
+actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml
+bin/conductor-exec bundle exec rubocop
 ```
 
 Then open a normal PR and let GitHub Actions prove the generated review-app,
 staging, lint, JS, and RSpec workflows before merging. For review-app workflow
 changes, test both the local workflow syntax and a real deployment. GitHub runs
 `issue_comment` workflows from the default branch, so a `+review-app-deploy`
 comment on the PR does not fully exercise command changes that are only on the
-PR branch. Before merge, run the PR branch workflow explicitly:
+PR branch. For top-level workflow edits, run the PR branch workflow explicitly:
 
 ```bash
 gh workflow run cpflow-deploy-review-app.yml --ref <branch> -f pr_number=<pr-number>
 ```
 
+This loads the workflow file from `<branch>`, but trusted local composite
+actions still come from the default branch before secrets are used. Treat it as
+a partial smoke test, then verify a real deploy after the workflow changes land
+on `master`.
+
 After the workflow reports a review-app URL, verify the URL returns HTTP 200.
 If a project needs to track generator changes automatically, use a scheduled
 maintenance PR or Renovate-style workflow that bumps the `cpflow` version,
 regenerates these files, and runs the same validation commands.
+
+For a fuller checklist, including the gotcha that review-app deploys load local
+composite actions from `master` before using Control Plane secrets, see
+[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md).

.controlplane/shakacode-team.md

@@ -66,9 +66,14 @@ flow, regenerate the `cpflow-*` actions/workflows in this repo from the target
 `cpflow` version or branch using `--staging-branch master`, review the diff, and
 keep the repository variables above aligned with `.controlplane/controlplane.yml`. Validate with
 `cpflow github-flow-readiness`, `actionlint .github/workflows/cpflow-*.yml`, and
-the normal CI checks before merging.
-
-See [readme.md](readme.md) for more details.
+the normal CI checks before merging. For review-app workflow changes, remember
+that the deploy workflow checks out trusted local actions from `master` before
+passing Control Plane secrets; PR-branch composite action changes are not fully
+tested until they land on `master` and a real review-app deploy is rerun.
+
+See [readme.md](readme.md) and
+[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md)
+for more details.
 
 ## Links
 

.github/cpflow-help.md

@@ -60,14 +60,14 @@ You asked for review app help. These commands are generated by [cpflow](https://
 <details>
 <summary>Advanced: testing changes to generated workflows</summary>
 
-When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the PR-branch workflow code before merging, dispatch the workflow manually with `gh`:
+When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the top-level PR-branch workflow file before merging, dispatch the workflow manually with `gh`:
 
 ```sh
 gh workflow run cpflow-deploy-review-app.yml --ref <your-pr-branch> -f pr_number=<pr-number>
 gh workflow run cpflow-delete-review-app.yml --ref <your-pr-branch> -f pr_number=<pr-number>
 gh workflow run cpflow-help-command.yml      --ref <your-pr-branch> -f pr_number=<pr-number>
 ```
 
-`workflow_dispatch` runs use the workflow file from the `--ref` you pass, so this is the supported way to test PR-branch workflow edits before merge. After merge, comment triggers go back to running the default-branch workflow code as usual.
+`workflow_dispatch` runs use the workflow file from the `--ref` you pass, but workflows that intentionally check out trusted local actions from the default branch will still load those local composite actions from the default branch before using secrets. Treat this as a partial smoke test for top-level workflow edits. For changes under `.github/actions/`, merge the generated fix to the default branch and rerun a real review-app deploy.
 
 </details>

bin/test-cpflow-github-flow

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+cpflow_cmd=(cpflow)
+if [[ $# -gt 0 ]]; then
+  cpflow_cmd=("$@")
+fi
+
+echo "==> cpflow github-flow-readiness"
+bin/conductor-exec "${cpflow_cmd[@]}" github-flow-readiness
+
+echo "==> parse generated GitHub Actions YAML"
+bin/conductor-exec ruby <<'RUBY'
+require "yaml"
+
+Dir[".github/actions/**/action.yml", ".github/workflows/*.yml"].sort.each do |path|
+  YAML.load_file(path, aliases: true)
+  puts "parsed #{path}"
+end
+RUBY
+
+echo "==> check composite action input descriptions"
+bin/conductor-exec ruby <<'RUBY'
+require "yaml"
+
+bad = []
+Dir[".github/actions/**/action.yml"].sort.each do |path|
+  doc = YAML.load_file(path, aliases: true)
+  doc.fetch("inputs", {}).each do |name, spec|
+    bad << "#{path}:#{name}" if spec["description"].to_s.include?("${{")
+  end
+end
+
+abort bad.join("\n") unless bad.empty?
+puts "no action metadata descriptions contain GitHub expressions"
+RUBY
+
+echo "==> actionlint"
+actionlint -ignore "SC2129" .github/workflows/cpflow-*.yml