diff --git a/.agents/references/building-block-state-doctoring.md b/.agents/references/building-block-state-doctoring.md new file mode 100644 index 00000000..748c88d0 --- /dev/null +++ b/.agents/references/building-block-state-doctoring.md @@ -0,0 +1,107 @@ +# Building block state doctoring + +A building block run keeps its Terraform state in meshStack's own HTTP backend, and that state is +reachable from outside the run. This is what makes tenant migration, adoption and run repair possible. + +Read this when a building block must take over a resource it did not create, when a run has to be +repaired after a failed apply, or when a seed state has to be built for a tenant migration. + +## The endpoint + +``` +/api/terraform/state/workspace//buildingBlock/ +``` + +`GET` reads it, `POST` with the state as the body writes it. The runner sets only `address` on the `http` +backend and passes credentials as `TF_HTTP_USERNAME=x` / `TF_HTTP_PASSWORD=`; +`TfStateRunTokenBasicAuthFilter` rewrites `Basic(x:)` into `Bearer `, so a normal API-key bearer +token works either way. No lock address is configured, so `-lock=false` is correct. + +The API key needs `ADM_TFSTATE_LIST`, `ADM_TFSTATE_SAVE` and `ADM_TFSTATE_DELETE`. + +## The adopt harness + +The safest way to build a seed state: a copy of the *new* module's `buildingblock/`, with a **local** state +file rather than the HTTP backend. Import the live cloud resource into it, plan, then use the resulting +state as the seed. + +```sh +mkdir -p /tmp/adopt && cp -r modules///buildingblock/. /tmp/adopt/ +cd /tmp/adopt +tofu init +tofu import -var-file=bbd-inputs.tfvars '' '' +tofu plan -out=tfplan -var-file=bbd-inputs.tfvars +# gate the seed on the plan — see the risk method in .agents/skills/tenant-migration/SKILL.md +cp terraform.tfstate seed.tfstate +``` + +Why this beats rewriting an old block's state: + +- it reads the resource's real attributes from the cloud API rather than trusting a stale record +- the plan is a free safety check, and it can be asserted on before anything is written +- it works for tenants that never had a building block at all + +Keep the harness a byte-identical copy of the module the runner executes. If it drifts, its plan stops +predicting the run's plan, which is the only reason the harness is worth having. In particular, copy the +non-`.tf` files too — a missing template such as `SUMMARY.md.tftpl` breaks the plan. + +**Assert before writing.** Refuse to emit a seed unless the plan shows no replacements and no destroys. +Creates of subordinate resources such as role assignments are expected. The check belongs in the script +rather than in the operator's head; the SKILL.md risk method has the exact gate. + +## A harness against a live block's state + +The same throwaway directory can point at a running block's state instead of a local file, which allows +`tofu import`, `state rm` and `state mv` against it like any other remote state. Full plan and apply +cycles work too. + +```hcl +terraform { + backend "http" { + address = "/api/terraform/state/workspace//buildingBlock/" + } +} +``` + +```sh +export TF_HTTP_USERNAME=x TF_HTTP_PASSWORD="$MT" +tofu init +tofu state list -lock=false +``` + +It also needs the BBD's inputs as tfvars, and `MESHSTACK_ENDPOINT` / `MESHSTACK_API_KEY` / +`MESHSTACK_API_SECRET` in the environment, because the buildingblock does not configure the meshstack +provider itself. + +## Normalising a state before pushing it + +| Field | What to do | Why | +|---|---|---| +| `terraform_version` | Set to the runner's version, never higher | OpenTofu refuses to read a state written by a newer version. Read the runner's version off a successful run's state. | +| `serial` | 1 | The run rewrites it. | +| `outputs` | `{}` | The run recomputes them; stale values are misleading if it fails early. | +| `lineage` | Leave as generated | The runner has no prior local state to conflict with. | +| resources with no instances | Drop them | Harmless but noisy. | +| `each` | `"map"` on any resource with string-keyed instances | State v4 needs it, and a hand-built state easily omits it. | + +## Import ids are provider-specific and worth checking + +Two examples, both from STACKIT, both non-obvious: + +- `stackit_resourcemanager_project` imports by **container id**, not project id — the `id` attribute in + state is the container id. +- `stackit_authorization_project_role_assignment` imports by the triple `,,`. + +Read the resource's `id` attribute in an existing state to work out the format rather than guessing. + +## Tokens and long applies + +**Mint the token immediately before a long apply.** A meshStack token that expires mid-apply gives `HTTP +401` on state save: the resources exist and the state does not record them. Recovery is +`tofu state push errored.tfstate`, which OpenTofu writes for exactly this case. + +## What state doctoring cannot fix + +It moves and rewrites *Terraform's* record of the world. It does not help with meshStack objects whose +blockers are about names and identity — a landing zone that needs a different `metadata.name`, or a +platform whose location is wrong. Those are replaces, and no state edit makes them otherwise. diff --git a/.agents/references/meshstack-api-cookbook.md b/.agents/references/meshstack-api-cookbook.md new file mode 100644 index 00000000..7062aad1 --- /dev/null +++ b/.agents/references/meshstack-api-cookbook.md @@ -0,0 +1,146 @@ +# meshStack API cookbook for migration work + +Every meshObject type needs its own versioned `Accept` header, and the wrong one fails in a way that looks +like an empty result rather than an error. The public spec is at +`https://docs.meshcloud.io/api/meshstack-openapi-docs.json` — download it and query it with `jq` rather +than guessing, because it documents required headers and validation rules that are not obvious from the +object shapes. + +## Getting a token + +`/api/login` only issues a redirect. Take the token from Keycloak directly: + +```sh +MT=$(curl -s -u ":" -d 'grant_type=client_credentials' \ + "https://sso./auth/realms/meshfed/protocol/openid-connect/token" | jq -r .access_token) +``` + +## Media types that matter + +The full header value is `application/vnd.meshcloud.api.`; the table lists the `` part. + +| Object | `Accept` | Notes | +|---|---|---| +| meshTenant | `meshtenant.v3.hal+json` | Has `spec.localId` — the cloud resource id — but no uuid. | +| meshTenant | `meshtenant.v4-preview.hal+json` | The **only** place tenant uuids are exposed. Plain `v4` returns 406. | +| meshBuildingBlock | `meshbuildingblock.v2-preview.hal+json` | | +| meshBuildingBlockRun | `meshbuildingblockrun.v1-preview.hal+json` | Also the only accepted type on the logs sub-resource. | +| meshBuildingBlockDefinition | `meshbuildingblockdefinition.v1-preview.hal+json` | | +| meshBuildingBlockDefinitionVersion | `meshbuildingblockdefinitionversion.v1-preview.hal+json` | | +| meshLandingZone | `meshlandingzone.v1.hal+json` | Not `v1-preview`. | +| meshPlatform | `meshplatform.v2.hal+json` | `v1-preview` renders every custom platform's config as `type: "unsupported"`, which is misleading. `v3`+ returns 406. | +| meshProject | `meshproject.v2.hal+json` | Tags are under `spec.tags`, **not** `metadata.tags`. | +| meshProjectUserBinding | `meshprojectuserbinding.v3.hal+json` | | + +A request therefore looks like this: + +```sh +curl -sS --fail-with-body -H "Authorization: Bearer $MT" \ + -H 'Accept: application/vnd.meshcloud.api.meshtenant.v4-preview.hal+json' \ + "$MESHSTACK/api/meshobjects/meshtenants?workspaceIdentifier=$WS" +``` + +## Traps that cost real time + +**A 401 looks like an empty result.** `jq` on an error body yields `null` or an empty list, so a stale +token reads as "no tenants exist". Always check the HTTP status before believing an empty list — +`--fail-with-body` is enough. + +**`meshtenants` returns nothing without a filter.** Use `?workspaceIdentifier=`. There is no +instance-wide listing, so an inventory means iterating workspaces. + +**`meshlandingzones?platformIdentifier=…` ignores the filter** and returns every landing zone on the +instance. Filter client-side, and beware that the `v1` list does not expose the owning platform at all — +fetch each zone individually if you need it. + +**Blocks of a definition** come from `meshbuildingblocks?definitionUuid=`. Do **not** filter the +full block list on `spec.buildingBlockDefinitionVersionRef.uuid` — that field holds the *version* uuid, not +the definition's, so it never matches. + +**Run logs need one exact `Accept`.** The run's `_links.downloadLogs` returns 406 for +`application/octet-stream`, `text/plain`, `application/zip` and even `*/*`. Only +`meshbuildingblockrun.v1-preview.hal+json` works, and it returns JSON with a `steps` array carrying +`displayName`, `status`, `systemMessage` and `userMessage`. The run uuid is `status.latestRunUuid` on the +block. + +**Project members are not readable via `meshusers`** without a `USER_*` permission — it returns 403. Use +the bindings instead, and note the path is `meshprojectbindings/userbindings`, not +`meshprojectuserbindings`, which 404s: + +``` +GET /api/meshobjects/meshprojectbindings/userbindings?workspaceIdentifier=&projectIdentifier=

+Accept: application/vnd.meshcloud.api.meshprojectuserbinding.v3.hal+json +``` + +Roles come back as display names — `Project Admin`, `Project User`, `Project Reader` — which map to the +lowercase identifiers a building block's `users` input carries: `admin`, `user`, `reader`. Confirm the +mapping against a real run's `users` input rather than assuming it. + +**`meshbuildingblockdefinitionversions` requires `buildingBlockDefinitionUuid`.** There is no unfiltered +listing, so auditing sources across an instance means one request per definition. + +## Landing zones + +`DELETE /api/meshobjects/meshlandingzones/` is documented as a **disable**, not a removal: +*"Deleting a meshLandingZone will disable it, preventing new tenants from being created on this landing +zone. Existing tenants will not be affected."* The object stays readable afterwards with +`lifecycle.state = DEACTIVATED`. + +**The identifier is global, not scoped per platform.** Verify uniqueness before deleting by bare +identifier — on one instance, 141 landing zones had zero duplicate names, but many carried the same +platform word in their name while belonging to different platforms. + +## Platform availability + +`PUT /api/meshobjects/meshplatforms/` with the `meshplatform.v2` media type accepts +`spec.availability`, so availability *can* be set through the API even where the panel is the usual route. + +**Round-trip safely:** `GET`, drop `status` and `_links`, keep `kind` / `apiVersion` / `metadata` / `spec`, +change the one field, `PUT`. Required on `metadata`: `name`, `ownedByWorkspace`, `uuid`. Required on `spec`: +`availability`, `contributingWorkspaces`, `displayName`, `locationRef`, `quotaDefinitions`. **Include +`spec.config`** — it is optional in the schema, so omitting it risks wiping the platform's configuration. + +Three guards fire in sequence, each a `400` with a precise message: + +1. `'spec.availability.restriction' must be 'PRIVATE' when marketplaceStatus is 'UNPUBLISHED'` +2. `'spec.availability.restrictedToWorkspaces' must contain exactly the owner when restriction is 'PRIVATE'` +3. `Cannot change spec.availability.restriction to 'PRIVATE' when the platform instance was published before.` + +**Together these make `UNPUBLISHED` unreachable for any platform that has ever been published.** A +published platform can only be `PUBLIC` (empty allowed-workspaces list) or `RESTRICTED` (non-empty). So +"disable a platform" is not an operation the API offers — deactivate its landing zones instead. + +Going the other way, to public, is also ordered. `setAllowedWorkspaces()` rejects an empty list that does +not contain the owner until `wasOncePublished` is true, so the sequence is: publish first, which leaves the +platform visibly `RESTRICTED`, then clear the list to become `PUBLIC`. + +**Platform deletion permanently consumes the identifier.** The delete is soft, but the identifier check +does not exclude deleted rows, so the name can never be reused. Recovering from that has meant deleting a +row from the database. + +## Auditing where building block code actually lives + +A `kit/`-style directory that nothing in a repository references can still be live, because a BBD reaches +it by Git repository path. To answer the question properly, read the sources off the instance: + +```sh +# per definition +GET /api/meshobjects/meshbuildingblockdefinitionversions?buildingBlockDefinitionUuid= +# then read each version's +.spec.implementation.terraform.repositoryUrl and .spec.implementation.terraform.repositoryPath +``` + +The field is `spec.implementation`, not `spec.source`. Write the fallback +`(.spec.implementation // .spec.source)` if you want the query to survive either shape, because a plain +`.spec.source` returns null and reads as "this definition has no source". + +Two things to get right: + +- **Check versions, not definitions.** A definition's source moves over its lifetime. One observed + definition pointed at three different paths across v1–v37, ending on a hub module. +- **Check what live blocks are pinned to.** A released version stays orderable, so a path is only truly + dead when no active block runs a version that uses it. Map each block's + `spec.buildingBlockDefinitionVersionRef.uuid` back to a version number. + +A display name is a weak signal but a useful hint — a definition renamed to something like +"(Deprecated, don't use!)" is telling you where to look first. diff --git a/.agents/references/tenant-migration-case-study.md b/.agents/references/tenant-migration-case-study.md new file mode 100644 index 00000000..cd8a5ecb --- /dev/null +++ b/.agents/references/tenant-migration-case-study.md @@ -0,0 +1,149 @@ +# Case study: migrating nine tenants off a custom STACKIT platform + +Evidence rather than instruction: the procedure itself is in `tenant-migration-runbook.md`, and this file +records what actually happened, because the failures are more instructive than the procedure. Nine meshTenants moved +from a hand-built custom platform (`stackit.sovereign`, owned by one workspace) to a platform deployed by +the STACKIT landing zone reference architecture (`likvid-stackit.global`), on a demo meshStack instance. + +Outcome: seven tenants migrated with their cloud project ids intact, two deleted as never-replicated, all +seven building blocks `SUCCEEDED`, all seven projects moved into the new platform's folder, no duplicate +project created at any point. + +## The inventory, and why the classes mattered + +| Class | Count | Procedure difference | +|---|---|---| +| Building-block-backed | 4 | Purge the old block before deleting the tenant | +| Replicator-created | 3 | No block to purge; seed had to be built from the live project | +| Never replicated | 2 | Deleted — `platformTenantId` was null, nothing to migrate | +| Also held in Terraform | 2 (subset of the above) | Needed `state rm` + `import` reconciliation afterwards | + +The four building-block-backed projects were exactly the four sitting at the organization root, because the +old BBD set `parent_container_id` to the organization. The three in folders predated the block and were made +by the replicator. That correlation was the fastest way to classify them. + +Ordering mattered: the three replicator-created tenants went first because nothing depended on them, which +meant the procedure was exercised twice before it met the project running a live Kubernetes cluster. + +## Failure 1: pre-existing role assignments + +The first migration failed on the apply with: + +``` +Error: Error while checking for duplicate role assignments +found a duplicate role assignment +``` + +The seed carried the project but no role assignments, so Terraform tried to create all six grants the +meshProject called for. Two already existed, put there by the old replicator. The duplicate error failed +the **whole apply**, so the project was renamed and moved but no grants were created and the run never +reached its output-collection step. + +The fix looked obvious — put every existing grant in the seed — and was wrong in the other direction. A +grant in the seed that the run's configuration does not contain gets **destroyed**. On the project hosting +the Kubernetes cluster that would have stripped `owner` from the cluster's own service accounts. + +So the set has to be computed: read the meshProject's user bindings, map them through the BBD's +`role_mapping`, and adopt only the intersection with what exists in the cloud. Service accounts and legacy +roles fall outside that set and stay unmanaged. On one project this left an out-of-band `reader` grant +untouched, which is the correct outcome — the block never asked to manage it. + +Reading the bindings needed the non-obvious endpoint, because `meshusers` is 403 without a `USER_*` +permission. The role-name mapping (`Project Admin → admin`, `Project User → user`, +`Project Reader → reader`) was confirmed against a real run's `users` input rather than assumed. + +## Failure 2: labels the provider cannot clear + +The second migration failed differently: + +``` +Error: Provider produced inconsistent result after apply +… .labels: was null, but now cty.MapVal(map[string]cty.Value{"project":…, "workspace":…}) +``` + +The hub module computes `labels = length(var.labels) > 0 ? var.labels : null` and the BBD passes `{}`, so +every adoption plans `labels -> null`. STACKIT ignores that instead of clearing the labels, and the +provider then fails the apply. Two of the three replicator-created projects carried the replicator's own +`project` / `workspace` labels and hit it. + +Clearing the labels fixed those two. The third project was worse: it carried `billingReference=""`, and + +``` +DELETE /v2/projects//labels?keys=billingReference +→ 409 The label [billingReference] is protected and can't be changed or deleted +``` + +even for the organization owner. `stackit project update --label` can only add labels, and a `PATCH` with +`labels: {}` is a merge that silently keeps everything. What works is an explicit null: + +```sh +curl -X PATCH -H "Authorization: Bearer $(stackit auth get-access-token)" \ + -H 'Content-Type: application/json' -d '{"labels":{"billingReference":null}}' \ + "https://resource-manager.api.stackit.cloud/v2/projects/" +``` + +Three things this taught, in order of usefulness: + +**A seed that misreports reality does not work.** Recording `labels: null` in the pushed state looks like it should +suppress the diff, but the runner refreshes before it plans, and the refresh puts reality back. + +**A failed first run has a consequence beyond the red status.** The apply dies before the +output-collection step, so the new meshTenant never gets a `platform_tenant_id`. Anything reading that +attribute downstream — in this case a `stackit_project_id` output feeding four other units — would have +broken. That is why this tenant could not simply be pushed through with a known-failing block. + +**Diagnose the origin before assuming a configuration bug.** The label was not set by any repository: +`billingReference` appeared nowhere in the foundation, the hub or the sibling foundation, and all 44 +projects belonging to the other foundation's install had no labels at all. The old building block did not +set it either — its `labels` input was `{}`, the same as sibling projects that came out clean, and it +created the project 21 seconds after itself with no labels. A scan of all 107 projects in the organization +found the key on exactly two, both the same platform project and its predecessor, both updated a month +after creation. It was a manual Portal edit, and STACKIT exposes the field as a first-class *Billing +Reference* project setting whose storage happens to be a protected label. The Portal renders "absent" and +"present but empty" identically, so the UI cannot distinguish the broken state from the healthy one. + +## What went right, and why + +**The adopt harness.** After the first failure, seed construction moved from rewriting the old block's +state to importing the live project into a byte-identical copy of the new module and planning it. That +turned every subsequent migration into a reviewed change: the harness printed the plan and refused to emit +a seed unless it showed no replacements and no destroys. Every later run came back +`0 added, 1 changed, 0 destroyed`, and the one change was computed timestamps. + +**The race was never lost** across seven migrations. The documented fifteen-second window against a +sub-second script is a wide enough margin that the timing is not the risk; the lookup logic is. + +**Container moves are genuinely in-place.** Terraform planned `parent_container_id` as an update, the +project id survived, and the meshTenant's `platform_tenant_id` — which *is* the project id — stayed valid. +`owner_email` is create-only in STACKIT, so the provider recorded the new value while the actual owner did +not change; that is a silent state-versus-reality divergence worth knowing about but harmless here. + +**Adoption avoided a permanent obstruction.** STACKIT deletions sit in `DELETING` for weeks — observed at 21 and +134 days on this organization — and block the parent folder's deletion with a `409`. Creating replacements +and deleting the originals would have left seven tombstones inside the new platform's folder, making it +undeletable for good. Adopting sidesteps the problem instead of paying for it seven times. + +## Retiring the old platform + +Once empty, the intent was to unpublish it. That is impossible: `UNPUBLISHED` requires +`restriction = PRIVATE`, and `PRIVATE` is permanently forbidden once a platform has been published. The +three `400` messages that establish this are in the cookbook. + +What worked instead was deactivating its three landing zones and deleting the old mandatory BBD. Since a +meshTenant cannot exist without a landing zone, that makes the platform unorderable regardless of its +availability. The platform row was deliberately left `ACTIVE`: deleting it permanently consumes the +identifier, and it was still the `SUCCESSFUL` metering comparison against the new platform's `FAILED` +metering — a useful control while that question was open. + +## Two corrections the migration forced on the plan + +Both were assumptions that had been written down as facts, and both were caught only by reading the live +instance: + +- A landing zone the plan claimed existed did not, because the deployment never set the input that creates + it. That changed a downstream design decision (a select would show one option, not two). +- The claim that platform availability "cannot be set in code at all" was wrong. The API accepts it; the + hub module's `ignore_changes` on it is a convention, not a limitation. + +The lesson is narrow and worth stating: in migration work, verify the current state against the API before +planning around it, especially for anything a document asserts about a deployed system. diff --git a/.agents/references/tenant-migration-runbook.md b/.agents/references/tenant-migration-runbook.md new file mode 100644 index 00000000..3ff8dd5a --- /dev/null +++ b/.agents/references/tenant-migration-runbook.md @@ -0,0 +1,164 @@ +# Tenant migration runbook + +The per-tenant procedure for moving a meshTenant to another meshPlatform while the cloud resource behind +it stays where it is and keeps its id. Read this together with `.agents/skills/tenant-migration/SKILL.md`, +which carries the workflow, the tenant classes and the risk method. + +## Before the first tenant + +- An API key with the tfstate permissions listed in `building-block-state-doctoring.md`, plus read access + to tenants, blocks, projects and project user bindings. +- A working adopt harness — a copy of the *new* module's `buildingblock/` that imports the live cloud + resource into a local state file. Prove it on a throwaway resource first. +- The runner's OpenTofu version, read off a successful run's state, because the seed's + `terraform_version` must not exceed it. + +## The procedure + +Steps 3 and 4 are one scripted unit, not two manual ones — see *The race*. + +1. **Purge the old building block.** + `DELETE /api/meshobjects/meshbuildingblocks//purge` answers `202`, and within about + ten seconds the block reads `forcePurge = true` and `lifecycle.state = DELETED`. No destroy runs and + the cloud resource is untouched. Confirm `forcePurge` before going on — this is what makes step 2 safe. + Skip this step for a tenant that has no block. + +2. **Delete the old meshTenant.** Also `202`, and also non-destructive *once the block is purged*. Without + the purge, this runs the block's teardown and destroys the cloud resource. + +3. **Create the new meshTenant with no `platform_tenant_id`.** Setting it is what makes meshStack call the + adoption an import. Leave it unset, `localId` stays null, and the guard returns early. meshStack then + creates the mandatory building block itself. + +4. **Seed the new block's state before its runner starts.** `POST` the prepared state to the block's state + address. See `building-block-state-doctoring.md`. + +5. **Let the run finish and check what it did.** One in-place update on the cloud resource, no creates of + the resource itself. Creates of role assignments are normal — see *Access parity*. + +## Building the seed state + +Two sources, and the second is better. + +**From the old block's state**, when the tenant has one: `GET` the old block's state and rewrite it into +the new module's resource shape. This works when the two modules' resource addresses agree, which they +usually do if both come from the same hub module lineage. Watch for renamed resources — a `for_each` +resource renamed between module versions needs its instances re-keyed and `each: "map"` set. + +**From the live cloud resource**, which is more robust and works for every class: stand up a throwaway +directory containing a copy of the *new* module's `buildingblock/`, `tofu import` the real resource into +it, plan, and use the resulting state as the seed. This reads reality instead of trusting an old state, +and the plan doubles as the safety check. Copy the buildingblock directory **with its non-`.tf` files** — +a missing `SUMMARY.md.tftpl` breaks the plan. + +Whichever source, normalise before pushing: + +- **`terraform_version` must not exceed the runner's.** OpenTofu refuses to read a state written by a + newer version. Read the runner's version off a successful run's state and rewrite the field to match. +- **`serial`** to 1 and **`outputs`** to `{}` — the run recomputes them. +- Drop resource entries with no instances. + +## Access parity: the trap that fails whole runs + +The seed's role assignments are the ones Terraform adopts. Getting the set wrong in either direction is +expensive: + +- **Too few** — a grant the run wants but the seed lacks comes back as a duplicate error from the cloud + API, and that fails the **entire apply**, not just that grant. +- **Too many** — a grant in the seed that the run's configuration does not contain is **destroyed**, + silently removing somebody's access. On a platform project this can strip the service accounts that + operate the platform. + +So compute the set rather than guessing it: read the meshProject's user bindings, map them through the +BBD's `role_mapping` input, and adopt only the intersection with what already exists in the cloud. +Everything else — service accounts, legacy roles, grants held by people who are not project members — +stays unmanaged and therefore untouched. + +## Labels can block the adoption outright + +If the module computes `labels = length(var.labels) > 0 ? var.labels : null` and the BBD passes `{}`, then +every adoption of a resource that already has labels plans `labels -> null`. When the cloud provider +ignores that instead of clearing them, the provider fails the apply: + +``` +Error: Provider produced inconsistent result after apply +… .labels: was null, but now cty.MapVal(map[string]cty.Value{…}) +``` + +Check the resource's labels before adopting it. Clearing them is usually enough. A **protected** label is +worse: the dedicated label-delete endpoint refuses it, and only an explicit `null` in the update payload +removes it. Recording `labels: null` in the seed does not help, because the runner refreshes before it +plans and the refresh puts reality back. + +## The race + +Between creating the tenant and pushing the state, the runner may start on its own. Measured window: +`PENDING` to `IN_PROGRESS` in about fifteen seconds, against a script that needs under half a second. The +margin is wide, so the race is only lost when the script fails to *find* the new block at all. + +Two ways to lose it, both from the lookup rather than the timing: + +- **A `jq` filter that rebinds `.`.** `select(($b|split(" "))|index(.)|not)` pipes `.` into the split + array, so the filter never matches. Use `[uuids] - $b | .[0] // empty`, and unit-test it against both a + hit and a miss before running it live. +- **A polling loop with no `sleep`.** Two hundred back-to-back requests finish long before meshStack has + created the block. + +**Recovery when the race is lost.** The run creates a second cloud resource and saves its own state over +the pushed one. Nothing is destroyed and the original resource is untouched: + +1. Let the run finish. Purging mid-apply just leaves the duplicate unowned. +2. Read the duplicate's id from the block's `status.outputs`, then purge the block and delete the tenant. +3. **Move the duplicate to the organization root before deleting it.** Deleting it in place leaves a + tombstone inside the target folder that never clears and blocks that folder's deletion for good. +4. Rebuild the seed and re-run. The old tenant is already gone, so skip the purge step. + +## Recovery when the first run fails + +More common than losing the race, and equally recoverable. A failed apply still saves its state and the +cloud resource keeps its id. + +1. **Read the run log.** The run's `_links.downloadLogs` needs one specific `Accept` value and returns + 406 for everything else, including `*/*`. See the cookbook. +2. Fix the cause. It is almost always the access-parity set or a label. +3. Purge the new block, then delete the new tenant. Confirm `forcePurge` before deleting. +4. Rebuild the seed and re-run the migration unchanged. Its delete of the old tenant is then a harmless + `404`, because that tenant is already gone. + +There is no way to re-trigger a failed run in place: the block exposes only `forcePurge`, `meshtenant` +and `self`, and a `runs` sub-resource does not exist. Purge and re-create is the only path. + +## Tenants held in Terraform state need reconciliation + +When some repository holds the tenant as a `meshstack_tenant` resource, the out-of-band migration leaves +that state stale, and the next plan will try to fix it by replacing the tenant. Per tenant, in this order: + +1. Run the migration, so the new tenant and its adopted block exist. +2. `state rm` the old tenant address, then `import` the new tenant **at the same address**. The import id + is the tenant uuid, or the legacy `workspace.project.platform.location` composite. +3. Only now change `platform_ref` and `landing_zone_ref` in the configuration. +4. Re-plan. It must come back clean. **Anything showing a replace means stop.** + +Doing step 3 before step 2 is what produces the replace. And check what reads the tenant's +`platform_tenant_id`: adoption preserves it, so those consumers keep resolving, but confirm by reading the +output back rather than assuming. + +## Background: why import is not an option + +Three separate blockers, and each one is worth knowing because each rules out a different shortcut. + +**meshStack refuses the import.** `MeshTenantService.verifyNoImportIntoCustomPlatformImplementedUsingBuildingBlocks` +treats `meshTenant.localId != null` on create as an import and throws `CustomPlatformImportUnsupportedException` +when the platform's category is `CUSTOM` and the landing zone has mandatory building block definitions. +The guard's own comment names the problem: the building block that is supposed to create the tenant would +have to run the equivalent of a `terraform import` to construct its initial state. That is exactly what +this runbook does — by hand, outside meshStack, where the guard cannot see it. + +**A mandatory BBD with a required input blocks tenant creation entirely.** An earlier guard, +`verifyMandatoryBuildingBlocksCreatableOrThrow`, rejects a mandatory BBD carrying an input with neither +an automatic assignment nor a default: `requires Building Blocks that cannot be created due to missing +inputs: :`. A BBD whose inputs are all `STATIC`, `PROJECT_IDENTIFIER` or `USER_PERMISSIONS` +passes, which is what lets step 3 create a tenant through the plain API with no inputs at all. + +**Identifiers cannot be reproduced.** `metadata.name` *is* the identifier, so renaming an object is a +replace. Importing an old landing zone under a new name only makes the next plan destroy and recreate it. diff --git a/.agents/skills/tenant-migration/SKILL.md b/.agents/skills/tenant-migration/SKILL.md new file mode 100644 index 00000000..c145afa4 --- /dev/null +++ b/.agents/skills/tenant-migration/SKILL.md @@ -0,0 +1,200 @@ +--- +name: tenant-migration +description: > + Move meshTenants to a different meshPlatform without destroying the cloud resources behind them, by + adopting each resource into the new building block's Terraform state. Use when asked to migrate, + consolidate or retire meshTenants, especially off a custom platform whose landing zones carry + mandatory building block definitions — the case where meshStack rejects a create carrying + `spec.localId` with `CustomPlatformImportUnsupportedException` and Terraform reports + `spec.platform_ref` as `RequiresReplace`. Covers tenant classes, the per-tenant runbook, building + block state doctoring, access parity, Terraform state reconciliation, and retiring the emptied + platform. +--- + +# Tenant Migration Skill + +Moving a meshTenant from one meshPlatform to another looks like an import and is not one. meshStack +refuses the import when the target platform is `CUSTOM` and its landing zone carries a mandatory +building block definition. Terraform cannot do it either: `spec.platform_ref` is `RequiresReplace`, +replacing a meshTenant destroys its building block, and that block's teardown destroys the live cloud +resource. + +The way through is **adoption**. Create the new meshTenant plainly, with no `platform_tenant_id`, so +meshStack sees an ordinary create and provisions the mandatory building block itself. Then write the +link to the existing cloud resource into the new block's **Terraform state**, before its runner starts. +The block adopts the resource instead of creating a replacement. + +--- + +## Two rules that prevent the expensive mistakes + +- **Never let Terraform replace a meshTenant.** Replacing it destroys its building block and with it + the live cloud resource. A plan that shows a meshTenant being replaced or destroyed means stop and + re-read the state. +- **Never delete a cloud resource in place.** Move it to the organization root first. On STACKIT a + deleted project sits in `DELETING` for weeks and permanently blocks its parent folder's deletion. + +--- + +## The scripts that did this once + +`scripts/` holds the tooling that migrated seven STACKIT tenants: `adopt-seed.sh` builds the seed state +from the live cloud resource, `migrate.sh` runs one tenant end to end, and `adopt/` is the plan harness. +Read [`scripts/README.md`](scripts/README.md) before using them. + +They are a worked example rather than a general tool — the migration logic is portable, the platform and +role details are STACKIT-specific. Every instance-specific value comes from a `MIG_*` environment +variable with no default. For a different platform, keep the shape and replace the harness and +`role_mapping`. + +--- + +## Workflow + +1. **Inventory the tenants and classify each one.** The class decides the procedure — see *Classify the + tenants*. Listing tenants needs a per-workspace filter and the right media type; both are in + `.agents/references/meshstack-api-cookbook.md`. +2. **Build the adopt harness and prove it on a throwaway resource.** The harness is a copy of the *new* + module's `buildingblock/` with a local state file, into which the live cloud resource is imported. + See `.agents/references/building-block-state-doctoring.md`. +3. **Order the tenants cheapest first** — the one nothing depends on — so the procedure is exercised + before it meets the tenant that matters. +4. **Run the per-tenant loop**, one tenant at a time, checking the run before starting the next. +5. **Reconcile Terraform state** for every tenant held in some repository's state, before anyone re-plans + that repository. See `.agents/references/tenant-migration-runbook.md` § *Tenants held in Terraform + state*. +6. **Retire the old platform** once no tenants remain on it — see *Retiring the old platform*. + +--- + +## Classify the tenants + +Tenants that look alike migrate differently, so establish the class before planning any work. + +| Class | How to spot it | What changes | +|---|---|---| +| **Building-block-backed** | A block instance exists for the tenant on the old mandatory BBD | Purge the block before deleting the tenant, or the teardown destroys the cloud resource | +| **Replicator-created** | No block instance; the resource predates the BBD | No block to purge; the seed state must be built from the live resource | +| **Never replicated** | `spec.localId` / `platformTenantId` is null | Nothing to migrate — delete the tenant | +| **Held in IaC** | A `meshstack_tenant` resource in some repository's state points at it | Needs state reconciliation on top of the migration, or the next plan replaces the tenant | + +The last class is the dangerous one, because the damage arrives later, out of a plan someone else runs. + +--- + +## The per-tenant loop + +`$MT` is a meshStack bearer token; the cookbook shows how to mint one. Steps 3 and 4 are a single +scripted unit — the runner may start on its own between them. + +1. **Purge the old building block.** Skip for a tenant that has no block. + + ```sh + curl -sS -X DELETE -H "Authorization: Bearer $MT" \ + "$MESHSTACK/api/meshobjects/meshbuildingblocks/$OLD_BB_UUID/purge" + ``` + + The answer is `202`. Within about ten seconds the block reads `forcePurge = true` and + `lifecycle.state = DELETED`, and no destroy run happens. Confirm `forcePurge` before step 2 — the + purge is what makes the delete safe. + +2. **Delete the old meshTenant.** Also `202`, and also non-destructive *once the block is purged*. + Without the purge, this runs the block's teardown and destroys the cloud resource. + +3. **Create the new meshTenant with no `platform_tenant_id`.** Setting it is what makes meshStack treat + the adoption as an import. Left unset, `spec.localId` stays null, the guard returns early, and + meshStack creates the mandatory building block itself. + +4. **Seed the new block's state before its runner starts.** Find the new block by diffing the uuid list + of the mandatory BBD's blocks against the list taken before step 3, then push the seed: + + ```sh + # block_uuids: a JSON array of the uuids under meshbuildingblocks?definitionUuid=$BBD_UUID + before=$(block_uuids) # taken before step 3 + new=$(block_uuids | jq -r --argjson b "$before" '. - $b | .[0] // empty') + curl -sS -X POST -H "Authorization: Bearer $MT" --data-binary @seed.tfstate \ + "$MESHSTACK/api/terraform/state/workspace/$WS/buildingBlock/$new" + ``` + + Poll for `$new` with a `sleep` in the loop. Both known ways of losing the race are lookup bugs, not + timing — see the runbook § *The race*. + +5. **Let the run finish and check what it did.** Expect one in-place update on the cloud resource and no + create of the resource itself. Creates of role assignments are normal, and getting that set wrong in + either direction fails the run — see the runbook § *Access parity*. Read the new tenant's + `platform_tenant_id` back: a run that fails before its output-collection step leaves it unset, which + breaks every downstream consumer of that attribute. + +--- + +## Risk method + +These four habits are what keep the procedure recoverable. They generalise past this migration. + +**Experiment on throwaway resources, and snapshot even then.** The throwaway proves the mechanism; the +snapshot is what allows a retry after the first attempt teaches something. Before mutating an object +through an API, `GET` it and keep the response — that file is the rollback. Before deleting an +attribute, record its value in the comment above the destructive command, so the restore command is +written down next to it. + +**Assert on the plan, not on the intention.** Refuse to emit a seed or continue a run unless the plan +shows no destroys and no replacements. Put the check in the script, not in the operator's head: + +```sh +tofu plan -out=tfplan +tofu show -json tfplan \ + | jq -e '[.resource_changes[]?.change.actions[]] | index("delete") == null' > /dev/null \ + || { echo "plan destroys or replaces a resource — refusing to proceed"; exit 1; } +``` + +A replacement appears as `["delete","create"]` or `["create","delete"]`, so a single check for `delete` +covers both. Then pass the expected cloud resource id into every script and abort when the resource +found does not match it. + +**Prefer an operation the provider models as an in-place update** over one it models as a replacement, +and confirm which it is by planning rather than by reading documentation. + +**Weigh the blast radius, not the wording of the action.** Deleting a landing zone reads as more +destructive than editing a platform's availability, but the landing zone delete is a reversible +deactivation, while the availability change is guarded by rules that can leave the platform in a state +it cannot return from. Check what each operation does before ranking them. + +--- + +## Retiring the old platform + +Do this only once no tenants remain on it. + +**Deactivating its landing zones is the effective retirement.** A meshTenant cannot exist without a +landing zone, so a deactivated landing zone makes the platform unorderable whatever its availability +says. `DELETE /api/meshobjects/meshlandingzones/` is a disable rather than a removal: the +object stays readable with `lifecycle.state = DEACTIVATED` and existing tenants are untouched. The +identifier is global, not scoped per platform, so verify uniqueness before deleting by bare identifier. + +**Do not expect to unpublish the platform.** Once a platform has been published, `UNPUBLISHED` is +unreachable — the cookbook lists the three `400` guards that establish this. Deleting the platform +permanently consumes its identifier even though the delete is soft, so keep the row unless the +identifier is certainly never wanted again. + +--- + +## Telling live code from dead code + +A `kit/`-style directory that no repository references may still be live, because a building block +definition reaches it by Git repository path rather than by a module source. Grepping the repository +cannot answer the question. Ask the instance instead, and read **versions**, not definitions: a +definition's source moves over its lifetime, so the current version may point somewhere entirely +different from the version an old block is pinned to. The cookbook § *Auditing where building block code +actually lives* has the two endpoints. + +--- + +## Key references + +| Topic | Reference | +|---|---| +| Per-tenant procedure, seed construction, traps, recovery, IaC reconciliation | `.agents/references/tenant-migration-runbook.md` | +| Reaching and rewriting a running block's Terraform state; the adopt harness | `.agents/references/building-block-state-doctoring.md` | +| Endpoints, media types, validation rules, traps that read as empty results | `.agents/references/meshstack-api-cookbook.md` | +| Worked migration of nine tenants, including both failures | `.agents/references/tenant-migration-case-study.md` | +| STACKIT backplane identity | `.agents/references/stackit-backplane.md` | diff --git a/.agents/skills/tenant-migration/scripts/README.md b/.agents/skills/tenant-migration/scripts/README.md new file mode 100644 index 00000000..352c2e1d --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/README.md @@ -0,0 +1,126 @@ +# Tenant migration tooling + +The scripts that ran the STACKIT migration this skill is written from: they move a meshTenant from one +meshPlatform to another **without destroying the cloud resource behind it**. Seven tenants were migrated +with them, off `stackit.sovereign` onto `likvid-stackit.global`. + +Read [the runbook](../../../references/tenant-migration-runbook.md) first. It carries the reasoning; +this file only covers how to run the scripts. + +**They are a worked example, not a general tool.** The migration logic is portable, but the platform +and role details are STACKIT-specific: the `adopt/` harness is a copy of the STACKIT project starterkit, +and `role_mapping` maps meshStack project roles onto STACKIT `owner` / `editor` / `reader`. For another +platform, keep the shape and replace those parts. + +| File | Purpose | +|---|---| +| `env.sh` | Credentials and API constants. Source it, do not execute it. | +| `adopt-seed.sh` | Builds the seed state from the **live** cloud resource. Preferred. | +| `migrate.sh` | Migrates one tenant end to end. | +| `reseed.jq` | Rewrites an old block's Terraform state into the new module's resource shape. Superseded by `adopt-seed.sh`. | +| `adopt/` | Standalone plan harness — a byte-identical copy of the hub module the runner executes. | + +## Configure the instance first + +Nothing is hardcoded to an instance, and nothing has a default — the wrong value here migrates the +wrong tenants: + +```sh +export MIG_IAC_REPO=~/git/likvid-bank/likvid-cloudfoundation # its setup-env.sh loads the API key +export MIG_MESH_URL=https://federation..meshcloud.io +export MIG_CLIENT_ID= +export MIG_PARENT_CONTAINER_ID= +export MIG_SERVICE_ACCOUNT_EMAIL= +export MIG_NEW_BBD_UUID= +``` + +`env.sh` derives the SSO host from `MIG_MESH_URL` by swapping `federation.` for `sso.`; set +`MIG_SSO_URL` if your instance does not follow that pattern. + +## Running one migration + +Two steps. Build the seed, then migrate: + +```sh +source scripts/env.sh +scripts/adopt-seed.sh +scripts/migrate.sh - +``` + +`adopt-seed.sh` imports the live project and its adoptable role assignments, prints the plan, and writes +`/tmp/mig-seed.json`. It aborts unless the plan adds and destroys nothing on the project itself — +creates of role assignments are expected and fine. `migrate.sh` sources `env.sh` itself; `-` as its +third argument means "the seed is already prepared". + +This works whether or not the tenant has an old building block. When it has one, purge it first and +confirm the purge before letting `migrate.sh` delete the tenant: + +```sh +curl -X DELETE -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks//purge" +``` + +The older single-step form still works and uses `reseed.jq` instead, purging the old block itself: + +```sh +scripts/migrate.sh +``` + +Prefer `adopt-seed.sh`. It reads reality instead of trusting the old state, and it catches the two +things that fail a run: a pre-existing role assignment missing from the seed, and a project label the +module wants to clear. Both are in the runbook, under *Access parity* and *Labels can block the +adoption outright*. + +The project-id argument is a safety interlock, not a convenience: both scripts abort if what they find +holds a different project id than the one given. + +## Check the labels first + +```sh +stackit project describe -o json | jq .labels +``` + +Anything non-empty fails the run, because the module plans `labels -> null` and STACKIT does not clear +them. Remove them with an explicit null — `stackit project update --label` can only add, and the +`DELETE .../labels` endpoint refuses protected keys such as `billingReference` with a 409: + +```sh +curl -X PATCH -H "Authorization: Bearer $(stackit auth get-access-token)" \ + -H 'Content-Type: application/json' -d '{"labels":{"":null}}' \ + "https://resource-manager.api.stackit.cloud/v2/projects/" +``` + +This is a provider bug, reported upstream on +[stackitcloud/terraform-provider-stackit#1381](https://github.com/stackitcloud/terraform-provider-stackit/issues/1381) +and still open as of `0.112.0`. `labels` is `Optional` and not `Computed`, so a configuration that omits +it plans `labels -> null`, STACKIT does not clear them, and the apply fails on the inconsistent result. +For a protected label such as `billingReference` there is no workaround through the labels endpoint at +all — only the `PATCH` above. + +## What it does + +1. Reads the old block's state and verifies the project id. +2. Purges the old building block — `DELETE .../purge`, which runs **no destroy** and leaves the + cloud project intact. +3. Deletes the old meshTenant. +4. Creates the new meshTenant with **no** `platform_tenant_id`. Setting it would make meshStack treat + the adoption as an import, which custom platforms reject. +5. Seeds the new block's state before its runner starts, then waits for the run. + +## The race + +Between steps 4 and 5 the runner may start on its own — the window is roughly fifteen seconds. If it +wins, it creates a *duplicate* cloud project. Recovery is known and lossless; see *The race* in the +runbook. The one rule that matters: **move a doomed project to the organization root before deleting +it**, because STACKIT deletions sit in `DELETING` for weeks and block the parent folder. + +The race was never lost across the seven migrations. Every failure was a plan problem instead, and those +are recoverable too — purge the new block, delete the new tenant, fix the seed, re-run. See *Recovery +when the first run fails* in the runbook, including how to read a run's log, which needs one specific +`Accept` header and 406s on everything else. + +## If nothing comes back + +An expired Vault session produces a 401, and `jq` renders that as an empty list — so "no tenants +found" usually means "not authenticated". `env.sh` checks for this and fails loudly. To fix it, run +`source ./setup-env.sh` in a real terminal so the interactive OIDC login can complete. diff --git a/.agents/skills/tenant-migration/scripts/adopt-seed.sh b/.agents/skills/tenant-migration/scripts/adopt-seed.sh new file mode 100755 index 00000000..99d24858 --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt-seed.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# adopt-seed.sh +# +# Builds /tmp/mig-seed.json for a tenant that has no old building block, so `migrate.sh - +# ` has a seed state to push. Imports the live STACKIT project into the `adopt/` harness — +# a byte-identical copy of the hub module the runner executes — then normalises the result. +# +# It also adopts every project role assignment that already exists in STACKIT *and* that the run will +# manage. Replicator-created projects already carry `owner` / `editor` / `reader` grants, and a grant +# the run wants but the seed omits fails the whole run with "found a duplicate role assignment". +# +# The run's grants come from the meshProject's user bindings through `role_mapping`, so the set is +# computed here the same way instead of guessed. Grants outside that set — the legacy `project.*` and +# `ufw.*` roles, and `owner` / `editor` / `reader` given to somebody who is not a meshProject member — +# are deliberately left out, so the run never manages them and never removes them. +# +# It plans before it writes the seed and aborts if anything is replaced or destroyed. Either would cost +# the live STACKIT project or somebody's access. Creates are expected: they are the grants the +# meshProject calls for that STACKIT does not have yet. +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WS=$1; PROJ=$2; CONTAINER=$3; PID=$4 + +# Where new projects go and who the module manages them as. Both are instance-specific, so they come +# from the environment — a wrong parent container id creates the project in the wrong folder. +: "${MIG_PARENT_CONTAINER_ID:?set MIG_PARENT_CONTAINER_ID to the landing-zone folder container id}" +: "${MIG_SERVICE_ACCOUNT_EMAIL:?set MIG_SERVICE_ACCOUNT_EMAIL to the platform service account}" + +# The runner executes OpenTofu 1.11.0, and OpenTofu refuses to read a state written by a newer +# version. The local binary is newer, so its version has to be rewritten out of the seed. +RUNNER_TOFU_VERSION=1.11.0 + +# The block's own role_mapping, and the meshStack role display names that produce its keys. Confirmed +# against the `users` input of a real run: Project Admin -> admin, Project User -> user, +# Project Reader -> reader. +ROLE_MAPPING='{"admin":["owner"],"reader":["reader"],"user":["editor"]}' +MESH_ROLES='{"Project Admin":"admin","Project User":"user","Project Reader":"reader"}' + +cd "$HERE/adopt" +unset STACKIT_SERVICE_ACCOUNT_KEY_PATH +export STACKIT_SERVICE_ACCOUNT_KEY="$STACKIT_ORG_SERVICE_ACCOUNT_KEY" + +# The meshProject's user bindings, which is what meshStack turns into the block's `users` input. +curl -sf -H "Authorization: Bearer $MT" \ + -H "Accept: application/vnd.meshcloud.api.meshprojectuserbinding.v3.hal+json" \ + "$MESH/api/meshobjects/meshprojectbindings/userbindings?projectIdentifier=$PROJ&workspaceIdentifier=$WS" \ + > /tmp/adopt-bindings.json +skipped=$(jq -r --argjson r "$MESH_ROLES" \ + '[._embedded.meshProjectUserBindings[]?.roleRef.name | select($r[.] == null)] | unique | join(", ")' \ + /tmp/adopt-bindings.json) +[ -n "$skipped" ] && echo " NOTE: meshStack roles with no STACKIT mapping, ignored: $skipped" + +jq -n --arg name "$PROJ" --argjson mapping "$ROLE_MAPPING" --argjson meshroles "$MESH_ROLES" \ + --arg parent "$MIG_PARENT_CONTAINER_ID" --arg sa "$MIG_SERVICE_ACCOUNT_EMAIL" \ + --slurpfile b /tmp/adopt-bindings.json '{ + parent_container_id: $parent, + project_name: $name, + service_account_email: $sa, + labels: {}, + role_mapping: $mapping, + users: ($b[0]._embedded.meshProjectUserBindings // [] + | map({subject: .subject.name, role: $meshroles[.roleRef.name]}) | map(select(.role != null)) + | group_by(.subject) | map({ + meshIdentifier: .[0].subject, username: .[0].subject, firstName: "-", lastName: "-", + email: .[0].subject, euid: .[0].subject, roles: ([.[].role] | unique) + })) +}' > /tmp/adopt.tfvars.json +echo " meshProject members: $(jq -r '[.users[].email]|join(" ")' /tmp/adopt.tfvars.json)" + +TFARGS=(-no-color -var-file=/tmp/adopt.tfvars.json -var "stackit_service_account_key=$STACKIT_SERVICE_ACCOUNT_KEY") + +rm -f terraform.tfstate terraform.tfstate.backup +tofu import "${TFARGS[@]}" stackit_resourcemanager_project.project "$CONTAINER" >/dev/null +got=$(jq -r '.resources[]|select(.type=="stackit_resourcemanager_project")|.instances[0].attributes.project_id' terraform.tfstate) +[ "$got" = "$PID" ] || { echo "ABORT: imported $got, expected $PID"; exit 1; } + +# Grants that already exist in STACKIT and that the run will manage. Everything else stays unmanaged. +stackit project member list --project-id "$PID" -o json > /tmp/adopt-members.json +jq -r --argjson mapping "$ROLE_MAPPING" --slurpfile v /tmp/adopt.tfvars.json ' + ([ $v[0].users[] | .email as $s | .roles[] | $mapping[.][] | "\($s):\(.)" ] | unique) as $wanted + | [ .[] | "\(.subject):\(.role)" ] | unique | map(select(IN($wanted[]))) | .[]' \ + /tmp/adopt-members.json > /tmp/adopt-adopt-keys.txt + +# The role assignment's import id is ",,". +while IFS= read -r key; do + [ -n "$key" ] || continue + tofu import "${TFARGS[@]}" \ + "stackit_authorization_project_role_assignment.role_assignments[\"$key\"]" \ + "$PID,${key##*:},${key%:*}" >/dev/null + echo " adopted $key" +done < /tmp/adopt-adopt-keys.txt + +tofu plan "${TFARGS[@]}" > /tmp/adopt-plan.txt 2>&1 +if grep -qE 'must be replaced|forces replacement' /tmp/adopt-plan.txt; then + echo "ABORT: plan replaces a resource. See /tmp/adopt-plan.txt"; exit 1 +fi +if grep -q '^No changes\.' /tmp/adopt-plan.txt; then + echo " plan: no changes" +elif grep -qE '^Plan: [0-9]+ to add, [0-9]+ to change, 0 to destroy\.$' /tmp/adopt-plan.txt; then + echo " plan: $(grep -E '^Plan: ' /tmp/adopt-plan.txt)" + grep -E '^ # ' /tmp/adopt-plan.txt | sed 's/^/ /' +else + echo "ABORT: plan destroys something. See /tmp/adopt-plan.txt"; tail -30 /tmp/adopt-plan.txt; exit 1 +fi + +jq --arg v "$RUNNER_TOFU_VERSION" \ + '.terraform_version = $v | .serial = 1 | .outputs = {} | .resources |= map(select((.instances|length) > 0))' \ + terraform.tfstate > /tmp/mig-seed.json +echo " seed: $(jq -r '[.resources[]|"\(.name)(\(.instances|length))"]|join(" ")' /tmp/mig-seed.json), tofu $(jq -r .terraform_version /tmp/mig-seed.json)" diff --git a/.agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl b/.agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl new file mode 100644 index 00000000..ca939c5b --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl @@ -0,0 +1,11 @@ +# Project: **${project_name}** + +## Details + +| Property | Value | +|----------|-------| +| **Project ID** | `${project_id}` | +| **Container ID** | `${container_id}` | +| **Portal** | [Open in STACKIT Portal](${project_url}) | + +${membership_summary} diff --git a/.agents/skills/tenant-migration/scripts/adopt/main.tf b/.agents/skills/tenant-migration/scripts/adopt/main.tf new file mode 100644 index 00000000..de99b36a --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/main.tf @@ -0,0 +1,44 @@ +locals { + # Determine the parent container ID based on environment + selected_parent_container_id = var.environment != null ? lookup(var.parent_container_ids, var.environment, var.parent_container_id) : var.parent_container_id + + users_with_stackit_roles = [ + for user in var.users : { + email = user.email + roles = distinct(flatten([ + for meshstack_role in user.roles : lookup(var.role_mapping, meshstack_role, []) + ])) + } + ] + + user_role_assignments = { + for assignment in flatten([ + for user in local.users_with_stackit_roles : [ + for stackit_role in user.roles : { + key = "${user.email}:${stackit_role}" + subject = user.email + stackit_role = stackit_role + } + ] + ]) : assignment.key => assignment + } +} + +resource "stackit_resourcemanager_project" "project" { + parent_container_id = local.selected_parent_container_id + name = var.project_name + owner_email = var.service_account_email + + # Only set labels if there are actually labels to set + labels = length(var.labels) > 0 ? var.labels : null +} + +# User role assignments (experimental IAM feature) +resource "stackit_authorization_project_role_assignment" "role_assignments" { + for_each = local.user_role_assignments + + resource_id = stackit_resourcemanager_project.project.project_id + role = each.value.stackit_role + subject = each.value.subject +} + diff --git a/.agents/skills/tenant-migration/scripts/adopt/outputs.tf b/.agents/skills/tenant-migration/scripts/adopt/outputs.tf new file mode 100644 index 00000000..213ba0c5 --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/outputs.tf @@ -0,0 +1,30 @@ +output "project_id" { + value = stackit_resourcemanager_project.project.project_id + description = "The UUID of the created StackIt project." +} + +output "container_id" { + value = stackit_resourcemanager_project.project.container_id + description = "The user-friendly container ID of the created StackIt project." +} + +output "project_name" { + value = stackit_resourcemanager_project.project.name + description = "The name of the created StackIt project." +} + +output "project_url" { + value = "https://portal.stackit.cloud/projects/${stackit_resourcemanager_project.project.project_id}" + description = "The deep link URL to access the project in the StackIt portal." +} + +output "summary" { + description = "Summary of the created project and STACKIT organization membership onboarding for assigned project users." + value = templatefile("${path.module}/SUMMARY.md.tftpl", { + project_name = stackit_resourcemanager_project.project.name + project_id = stackit_resourcemanager_project.project.project_id + container_id = stackit_resourcemanager_project.project.container_id + project_url = "https://portal.stackit.cloud/projects/${stackit_resourcemanager_project.project.project_id}" + membership_summary = fileexists("${path.module}/stackit_organization_membership_summary.md") ? file("${path.module}/stackit_organization_membership_summary.md") : "STACKIT organization membership summary was not generated." + }) +} diff --git a/.agents/skills/tenant-migration/scripts/adopt/provider.tf b/.agents/skills/tenant-migration/scripts/adopt/provider.tf new file mode 100644 index 00000000..20c1ed6f --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/provider.tf @@ -0,0 +1,10 @@ +# local-only: the real building block authenticates via WIF; here we use the org service account key +provider "stackit" { + service_account_key = var.stackit_service_account_key + experiments = ["iam"] +} + +variable "stackit_service_account_key" { + type = string + sensitive = true +} diff --git a/.agents/skills/tenant-migration/scripts/adopt/variables.tf b/.agents/skills/tenant-migration/scripts/adopt/variables.tf new file mode 100644 index 00000000..b6cab1ee --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/variables.tf @@ -0,0 +1,61 @@ +variable "parent_container_id" { + type = string + nullable = false + description = "The parent container ID (organization or folder) where the project will be created." +} + +variable "environment" { + type = string + default = null + description = "The environment type (production, staging, development). If not set, uses parent_container_id directly." +} + +variable "parent_container_ids" { + type = object({ + production = optional(string) + staging = optional(string) + development = optional(string) + }) + default = {} + description = "Parent container IDs for different environments. If environment is set, the corresponding container ID will be used." +} + +variable "project_name" { + type = string + nullable = false + description = "The name of the StackIt project to create." +} + +variable "service_account_email" { + type = string + nullable = false + description = "Email of the STACKIT service account for WIF-based authentication and project ownership." +} + +variable "labels" { + type = map(string) + nullable = false + description = "Labels to apply to the project. Includes the `networkArea` label when the building block definition is wired to a network area." +} + +variable "users" { + description = "List of users from the authoritative system. Each user's `roles` are meshStack roles that are mapped to STACKIT project roles via `role_mapping`." + type = list(object({ + meshIdentifier = string + username = string + firstName = string + lastName = string + email = string + euid = string + roles = list(string) + })) + nullable = false +} + +variable "role_mapping" { + type = map(list(string)) + description = "Maps meshStack roles from `users[*].roles` to STACKIT project roles. Values can be built-in STACKIT roles or custom STACKIT role names. Unknown meshStack roles are ignored." + + nullable = false +} + diff --git a/.agents/skills/tenant-migration/scripts/adopt/versions.tf b/.agents/skills/tenant-migration/scripts/adopt/versions.tf new file mode 100644 index 00000000..364654db --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/versions.tf @@ -0,0 +1,13 @@ +terraform { + required_version = ">= 1.11.0" + required_providers { + stackit = { + source = "stackitcloud/stackit" + version = ">= 0.98.0" + } + meshstack = { + source = "meshcloud/meshstack" + version = ">= 0.21.0" + } + } +} diff --git a/.agents/skills/tenant-migration/scripts/env.sh b/.agents/skills/tenant-migration/scripts/env.sh new file mode 100644 index 00000000..b8282d6b --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/env.sh @@ -0,0 +1,49 @@ +# Credentials and API constants for the tenant migration. Source it, do not execute it. +# +# source scripts/env.sh || return 1 +# +# Set these three for your instance before sourcing. There are no defaults on purpose: the wrong +# instance here migrates the wrong tenants. +# +# MIG_IAC_REPO the IaC runtime repo whose setup-env.sh loads the meshStack API key from Vault +# MIG_MESH_URL the meshStack API base url +# MIG_CLIENT_ID the API key's client id, paired with the key that setup-env.sh exports +# +: "${MIG_IAC_REPO:?set MIG_IAC_REPO to the IaC runtime repo, e.g. ~/git//-cloudfoundation}" +: "${MIG_MESH_URL:?set MIG_MESH_URL, e.g. https://federation..meshcloud.io}" +: "${MIG_CLIENT_ID:?set MIG_CLIENT_ID to the meshStack API key client id}" + +# It cds to the IaC repo first on purpose: setup-env.sh only works from the repo root, and agent shells +# reset the working directory between commands, which makes a bare `source ./setup-env.sh` fail silently. +cd "$MIG_IAC_REPO" || return 1 +source ./setup-env.sh >/dev/null 2>&1 + +# setup-env.sh falls back to an interactive `vault login -method=oidc`, which cannot complete in a +# non-interactive shell — it then carries on with no secrets at all. Catch that here, because the +# alternative is a 401 that jq renders as an empty list, i.e. "no tenants exist". +if [ -z "$MESHSTACK_API_KEY_CLOUDFOUNDATION" ]; then + echo "FATAL: vault secrets not loaded. Run 'source ./setup-env.sh' in a terminal to do the OIDC login." >&2 + return 1 +fi + +export MESH="$MIG_MESH_URL" + +# /api/login only issues a redirect; the token has to come from keycloak directly. The sso host mirrors +# the api host: federation..meshcloud.io -> sso..meshcloud.io. +MIG_SSO_URL=${MIG_SSO_URL:-$(printf '%s' "$MESH" | sed 's#//federation\.#//sso.#')} +export MT=$(curl -s -u "${MIG_CLIENT_ID}:${MESHSTACK_API_KEY_CLOUDFOUNDATION}" \ + -d 'grant_type=client_credentials' \ + "${MIG_SSO_URL}/auth/realms/meshfed/protocol/openid-connect/token" \ + | jq -r '.access_token // empty') +[ -z "$MT" ] && { echo "FATAL: token request failed" >&2; return 1; } + +# Every meshObject type needs its own versioned Accept header. Plain v4 (no -preview) returns 406. +export BBACC="Accept: application/vnd.meshcloud.api.meshbuildingblock.v2-preview.hal+json" +export TN3="Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json" +export TN4="Accept: application/vnd.meshcloud.api.meshtenant.v4-preview.hal+json" +export PL2="Accept: application/vnd.meshcloud.api.meshplatform.v2.hal+json" + +# ~/.terraformrc may carry a dev override with an older meshstack provider than the one that wrote the +# state, which makes tofu refuse to read it. +[ -f /tmp/tofu-no-override.tfrc ] || printf 'provider_installation {\n direct {}\n}\n' > /tmp/tofu-no-override.tfrc +export TF_CLI_CONFIG_FILE=/tmp/tofu-no-override.tfrc diff --git a/.agents/skills/tenant-migration/scripts/migrate.sh b/.agents/skills/tenant-migration/scripts/migrate.sh new file mode 100755 index 00000000..81f9bfbf --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/migrate.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# migrate.sh +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$HERE/env.sh" || exit 1 +set -u +WS=$1; PROJ=$2; OLDBB=$3; PID=$4 + +# The building block definition the new tenant's landing zone makes mandatory. Instance-specific. +: "${MIG_NEW_BBD_UUID:?set MIG_NEW_BBD_UUID to the building block definition of the target landing zone}" +NEWDEF=$MIG_NEW_BBD_UUID +say(){ echo " $*"; } + +echo "=== $WS/$PROJ -> likvid-stackit.global ===" + +# 1. capture and disarm the old block +if [ "$OLDBB" != "-" ]; then + curl -s -H "Authorization: Bearer $MT" \ + "$MESH/api/terraform/state/workspace/$WS/buildingBlock/$OLDBB" > /tmp/mig-old.json + got=$(jq -r '.resources[]|select(.type=="stackit_resourcemanager_project")|.instances[0].attributes.project_id' /tmp/mig-old.json) + [ "$got" = "$PID" ] || { echo "ABORT: state holds $got, expected $PID"; exit 1; } + jq -f "$HERE/reseed.jq" /tmp/mig-old.json > /tmp/mig-seed.json + say "seed state: $(jq -r '[.resources[]|"\(.name)(\(.instances|length))"]|join(" ")' /tmp/mig-seed.json)" + say "purge old block -> $(curl -s -X DELETE -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks/$OLDBB/purge")" + sleep 8 +else + [ -s /tmp/mig-seed.json ] || { echo "ABORT: no /tmp/mig-seed.json prepared"; exit 1; } + say "using externally prepared seed state" +fi + +# 2. delete the old tenant +say "delete old tenant -> $(curl -s -X DELETE -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $MT" \ + -H "Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json" \ + "$MESH/api/meshobjects/meshtenants/$WS.$PROJ.stackit.sovereign")" +sleep 10 + +# 3. create the new tenant and seed its block's state before the runner starts +before=$(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks?definitionUuid=$NEWDEF&size=200" \ + | jq -r '._embedded.meshBuildingBlocks[]?.metadata.uuid' | sort | tr '\n' ' ') +printf '{"apiVersion":"v3","kind":"meshTenant","metadata":{"ownedByWorkspace":"%s","ownedByProject":"%s","platformIdentifier":"likvid-stackit.global"},"spec":{"landingZoneIdentifier":"likvid-stackit-default"}}' \ + "$WS" "$PROJ" > /tmp/mig-ten.json +code=$(curl -s -o /tmp/mig-ten-out.json -w '%{http_code}' -X POST -H "Authorization: Bearer $MT" \ + -H "Content-Type: application/vnd.meshcloud.api.meshtenant.v3.hal+json" \ + -H "Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json" \ + --data-binary @/tmp/mig-ten.json "$MESH/api/meshobjects/meshtenants") +say "create new tenant -> $code" +[ "$code" = 201 ] || { head -c 300 /tmp/mig-ten-out.json; exit 1; } + +BB="" +for i in $(seq 1 60); do + sleep 0.5 + BB=$(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks?definitionUuid=$NEWDEF&size=200" \ + | jq -r --argjson b "$(printf '%s' "$before" | jq -R 'split(" ")|map(select(length>0))')" \ + '[._embedded.meshBuildingBlocks[]?.metadata.uuid] - $b | .[0] // empty') + [ -n "$BB" ] && break +done +[ -n "$BB" ] || { echo "ABORT: new block not found"; exit 1; } +say "seed state into $BB -> $(curl -s -o /dev/null -w '%{http_code}' -X POST -H "Authorization: Bearer $MT" \ + -H "Content-Type: application/json" --data-binary @/tmp/mig-seed.json \ + "$MESH/api/terraform/state/workspace/$WS/buildingBlock/$BB")" +echo "$BB" > /tmp/mig-bb + +# 4. wait, then verify +for i in $(seq 1 40); do + s=$(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" "$MESH/api/meshobjects/meshbuildingblocks/$BB" | jq -r '.status.status') + case "$s" in SUCCEEDED|FAILED) break;; esac + sleep 15 +done +say "run -> $s" +say "block project_id = $(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" "$MESH/api/meshobjects/meshbuildingblocks/$BB" | jq -r '.status.outputs.project_id.value')" +say "tenant localId = $(curl -s -H "Authorization: Bearer $MT" -H 'Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json' \ + "$MESH/api/meshobjects/meshtenants/$WS.$PROJ.likvid-stackit.global" | jq -r '.spec.localId')" +say "expected = $PID" +stackit project describe "$PID" 2>&1 | grep -E 'NAME|STATE|PARENT' | sed 's/^/ /' diff --git a/.agents/skills/tenant-migration/scripts/reseed.jq b/.agents/skills/tenant-migration/scripts/reseed.jq new file mode 100644 index 00000000..df82e3e1 --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/reseed.jq @@ -0,0 +1,26 @@ +# Rewrite an old STACKIT Project block state into the new module's shape. +# Keeps the project resource untouched; merges every role-assignment resource +# into the new module's single `role_assignments` map, re-keyed ":". +def assignments: + [ .resources[] | select(.type == "stackit_authorization_project_role_assignment") ]; +{ + version: .version, + terraform_version: .terraform_version, + serial: 1, + lineage: .lineage, + outputs: {}, + check_results: null, + resources: ( + [ .resources[] | select(.type == "stackit_resourcemanager_project") ] + + (if (assignments | length) == 0 then [] else + [ { mode: "managed", + type: "stackit_authorization_project_role_assignment", + name: "role_assignments", + provider: (assignments[0].provider), + each: "map", + instances: [ assignments[].instances[] + | .index_key = (.attributes.subject + ":" + .attributes.role) ] + } ] + end) + ) +} diff --git a/AGENTS.md b/AGENTS.md index 5c36f59d..8888ac5e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -386,6 +386,19 @@ See [.agents/skills/e2e-test/SKILL.md](.agents/skills/e2e-test/SKILL.md) (the `e --- +## Tenant Migration + +Moving meshTenants between meshPlatforms — for example off a hand-built custom platform onto one deployed +by a reference architecture — cannot use meshStack's import path when the landing zone carries a mandatory +building block definition. The tenants must instead adopt their existing cloud resources through the +building block's Terraform state. + +See [.agents/skills/tenant-migration/SKILL.md](.agents/skills/tenant-migration/SKILL.md) (the +`tenant-migration` skill) for the workflow, the tenant classes, the per-tenant runbook, building block +state doctoring and the meshStack API cookbook that the migration relies on. + +--- + ## Checklist for New Modules - [ ] `backplane/` (optional) and `buildingblock/` with all required files