From 3783a41333dfde88b133c8dedd3f76bf6681aa6e Mon Sep 17 00:00:00 2001 From: Andreas Grub Date: Wed, 19 Aug 2026 10:23:49 +0200 Subject: [PATCH 1/4] docs(agents): dump tenant-migration findings from the STACKIT platform migration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Raw capture of a platform engineering session that moved nine meshTenants off a hand-built custom STACKIT platform onto one deployed by the landing zone reference architecture, without destroying the projects behind them. This is the unrefined material, written down while it is still fresh: - skills/tenant-migration/SKILL.md — first draft of the reusable skill - references/tenant-migration-runbook.md — the per-tenant procedure - references/building-block-state-doctoring.md — reaching and rewriting a running block's Terraform state - references/meshstack-api-cookbook.md — endpoints, media types, validation rules and the traps that read as empty results - references/tenant-migration-case-study.md — the worked migration, including both failures and what they taught The next commit turns this into a skill that reads like an instruction rather than a session log. Co-Authored-By: Claude Opus 5 (1M context) --- .../building-block-state-doctoring.md | 83 ++++++++++ .agents/references/meshstack-api-cookbook.md | 131 +++++++++++++++ .../references/tenant-migration-case-study.md | 148 +++++++++++++++++ .../references/tenant-migration-runbook.md | 154 ++++++++++++++++++ .agents/skills/tenant-migration/SKILL.md | 100 ++++++++++++ 5 files changed, 616 insertions(+) create mode 100644 .agents/references/building-block-state-doctoring.md create mode 100644 .agents/references/meshstack-api-cookbook.md create mode 100644 .agents/references/tenant-migration-case-study.md create mode 100644 .agents/references/tenant-migration-runbook.md create mode 100644 .agents/skills/tenant-migration/SKILL.md diff --git a/.agents/references/building-block-state-doctoring.md b/.agents/references/building-block-state-doctoring.md new file mode 100644 index 00000000..44c12644 --- /dev/null +++ b/.agents/references/building-block-state-doctoring.md @@ -0,0 +1,83 @@ +# Building block state doctoring + +A building block run keeps its Terraform state in meshStack's own HTTP backend, and that state is +reachable from outside the run. This is what makes tenant migration, adoption and run repair possible. + +## The endpoint + +``` +/api/terraform/state/workspace//buildingBlock/ +``` + +`GET` reads it, `POST` with the state as the body writes it. The runner sets only `address` on the `http` +backend and passes credentials as `TF_HTTP_USERNAME=x` / `TF_HTTP_PASSWORD=`; +`TfStateRunTokenBasicAuthFilter` rewrites `Basic(x:)` into `Bearer `, so a normal API-key bearer +token works either way. No lock address is configured, so `-lock=false` is correct. + +The API key needs `ADM_TFSTATE_LIST`, `ADM_TFSTATE_SAVE` and `ADM_TFSTATE_DELETE`. + +## A local harness against a live block's state + +A throwaway directory with the same module source, a matching provider configuration and an `http` +backend pointing at the run's state can `tofu import`, `state rm` and `state mv` against it like any other +remote state. Full plan and apply cycles work too. + +It needs: + +- the module's `buildingblock/` directory copied **with its non-`.tf` files** — a missing template such as + `SUMMARY.md.tftpl` breaks the plan +- the BBD's inputs as tfvars +- `MESHSTACK_ENDPOINT` / `MESHSTACK_API_KEY` / `MESHSTACK_API_SECRET` in the environment, because the + buildingblock does not configure the meshstack provider itself + +## The adopt harness + +The safest way to build a seed state: a copy of the *new* module's `buildingblock/`, with a **local** state +file rather than the HTTP backend. Import the live cloud resource into it, plan, then use the resulting +state as the seed. + +Why this beats rewriting an old block's state: + +- it reads the resource's real attributes from the cloud API rather than trusting a stale record +- the plan is a free safety check, and it can be asserted on before anything is written +- it works for tenants that never had a building block at all + +Keep the harness a byte-identical copy of the module the runner executes. If it drifts, its plan stops +predicting the run's plan, which is the only reason the harness is worth having. + +**Assert before you write.** Refuse to emit a seed unless the plan shows no replacements and no destroys. +Creates of subordinate resources such as role assignments are expected. A single grep on the plan output +is enough, and it belongs in the script rather than in your head. + +## Normalising a state before pushing it + +| Field | What to do | Why | +|---|---|---| +| `terraform_version` | Set to the runner's version, never higher | OpenTofu refuses to read a state written by a newer version. Read the runner's version off a successful run's state. | +| `serial` | 1 | The run rewrites it. | +| `outputs` | `{}` | The run recomputes them; stale values are misleading if it fails early. | +| `lineage` | Leave as generated | The runner has no prior local state to conflict with. | +| resources with no instances | Drop them | Harmless but noisy. | +| `each` | `"map"` on any resource with string-keyed instances | State v4 needs it, and a hand-built state easily omits it. | + +## Import ids are provider-specific and worth checking + +Two examples, both from STACKIT, both non-obvious: + +- `stackit_resourcemanager_project` imports by **container id**, not project id — the `id` attribute in + state is the container id. +- `stackit_authorization_project_role_assignment` imports by the triple `,,`. + +Read the resource's `id` attribute in an existing state to work out the format rather than guessing. + +## Tokens and long applies + +**Mint the token immediately before a long apply.** A meshStack token that expires mid-apply gives `HTTP +401` on state save: the resources exist and the state does not record them. Recovery is +`tofu state push errored.tfstate`, which OpenTofu writes for exactly this case. + +## What state doctoring cannot fix + +It moves and rewrites *Terraform's* record of the world. It does not help with meshStack objects whose +blockers are about names and identity — a landing zone that needs a different `metadata.name`, or a +platform whose location is wrong. Those are replaces, and no state edit makes them otherwise. diff --git a/.agents/references/meshstack-api-cookbook.md b/.agents/references/meshstack-api-cookbook.md new file mode 100644 index 00000000..eb753f55 --- /dev/null +++ b/.agents/references/meshstack-api-cookbook.md @@ -0,0 +1,131 @@ +# meshStack API cookbook for migration work + +Every meshObject type needs its own versioned `Accept` header, and the wrong one fails in a way that looks +like an empty result rather than an error. The public spec is at +`https://docs.meshcloud.io/api/meshstack-openapi-docs.json` — download it and query it with `jq` rather +than guessing, because it documents required headers and validation rules that are not obvious from the +object shapes. + +## Getting a token + +`/api/login` only issues a redirect. Take the token from Keycloak directly: + +```sh +MT=$(curl -s -u ":" -d 'grant_type=client_credentials' \ + "https://sso./auth/realms/meshfed/protocol/openid-connect/token" | jq -r .access_token) +``` + +## Media types that matter + +| Object | `Accept` | Notes | +|---|---|---| +| meshTenant | `meshtenant.v3.hal+json` | Has `spec.localId` — the cloud resource id — but no uuid. | +| meshTenant | `meshtenant.v4-preview.hal+json` | The **only** place tenant uuids are exposed. Plain `v4` returns 406. | +| meshBuildingBlock | `meshbuildingblock.v2-preview.hal+json` | | +| meshBuildingBlockRun | `meshbuildingblockrun.v1-preview.hal+json` | Also the only accepted type on the logs sub-resource. | +| meshBuildingBlockDefinition | `meshbuildingblockdefinition.v1-preview.hal+json` | | +| meshBuildingBlockDefinitionVersion | `meshbuildingblockdefinitionversion.v1-preview.hal+json` | | +| meshLandingZone | `meshlandingzone.v1.hal+json` | Not `v1-preview`. | +| meshPlatform | `meshplatform.v2.hal+json` | `v1-preview` renders every custom platform's config as `type: "unsupported"`, which is misleading. `v3`+ returns 406. | +| meshProject | `meshproject.v2.hal+json` | Tags are under `spec.tags`, **not** `metadata.tags`. | +| meshProjectUserBinding | `meshprojectuserbinding.v3.hal+json` | | + +## Traps, each of which cost real time + +**A 401 looks like an empty result.** `jq` on an error body yields `null` or an empty list, so a stale +token reads as "no tenants exist". Always check the HTTP status before believing an empty list. + +**`meshtenants` returns nothing without a filter.** Use `?workspaceIdentifier=`. There is no +instance-wide listing, so an inventory means iterating workspaces. + +**`meshlandingzones?platformIdentifier=…` ignores the filter** and returns every landing zone on the +instance. Filter client-side, and beware that the `v1` list does not expose the owning platform at all — +fetch each zone individually if you need it. + +**Blocks of a definition** come from `meshbuildingblocks?definitionUuid=`. Do **not** filter the +full block list on `spec.buildingBlockDefinitionVersionRef.uuid` — that field holds the *version* uuid, not +the definition's, so it never matches. + +**Run logs need one exact `Accept`.** The run's `_links.downloadLogs` returns 406 for +`application/octet-stream`, `text/plain`, `application/zip` and even `*/*`. Only +`meshbuildingblockrun.v1-preview.hal+json` works, and it returns JSON with a `steps` array carrying +`displayName`, `status`, `systemMessage` and `userMessage`. The run uuid is `status.latestRunUuid` on the +block. + +**Project members are not readable via `meshusers`** without a `USER_*` permission — it returns 403. Use +the bindings instead, and note the path is `meshprojectbindings/userbindings`, not +`meshprojectuserbindings`, which 404s: + +``` +GET /api/meshobjects/meshprojectbindings/userbindings?workspaceIdentifier=&projectIdentifier=

+Accept: application/vnd.meshcloud.api.meshprojectuserbinding.v3.hal+json +``` + +Roles come back as display names — `Project Admin`, `Project User`, `Project Reader` — which map to the +lowercase identifiers a building block's `users` input carries: `admin`, `user`, `reader`. Confirm the +mapping against a real run's `users` input rather than assuming it. + +**`meshbuildingblockdefinitionversions` requires `buildingBlockDefinitionUuid`.** There is no unfiltered +listing, so auditing sources across an instance means one request per definition. + +## Landing zones + +`DELETE /api/meshobjects/meshlandingzones/` is documented as a **disable**, not a removal: +*"Deleting a meshLandingZone will disable it, preventing new tenants from being created on this landing +zone. Existing tenants will not be affected."* The object stays readable afterwards with +`lifecycle.state = DEACTIVATED`. + +**The identifier is global, not scoped per platform.** Verify uniqueness before deleting by bare +identifier — on one instance, 141 landing zones had zero duplicate names, but many carried the same +platform word in their name while belonging to different platforms. + +## Platform availability + +`PUT /api/meshobjects/meshplatforms/` with the `meshplatform.v2` media type accepts +`spec.availability`, so availability *can* be set through the API even where the panel is the usual route. + +**Round-trip safely:** `GET`, drop `status` and `_links`, keep `kind` / `apiVersion` / `metadata` / `spec`, +change the one field, `PUT`. Required on `metadata`: `name`, `ownedByWorkspace`, `uuid`. Required on `spec`: +`availability`, `contributingWorkspaces`, `displayName`, `locationRef`, `quotaDefinitions`. **Include +`spec.config`** — it is optional in the schema, so omitting it risks wiping the platform's configuration. + +Three guards fire in sequence, each a `400` with a precise message: + +1. `'spec.availability.restriction' must be 'PRIVATE' when marketplaceStatus is 'UNPUBLISHED'` +2. `'spec.availability.restrictedToWorkspaces' must contain exactly the owner when restriction is 'PRIVATE'` +3. `Cannot change spec.availability.restriction to 'PRIVATE' when the platform instance was published before.` + +**Together these make `UNPUBLISHED` unreachable for any platform that has ever been published.** A +published platform can only be `PUBLIC` (empty allowed-workspaces list) or `RESTRICTED` (non-empty). So +"disable a platform" is not an operation the API offers — deactivate its landing zones instead. + +Going the other way, to public, is also ordered. `setAllowedWorkspaces()` rejects an empty list that does +not contain the owner until `wasOncePublished` is true, so the sequence is: publish first, which leaves the +platform visibly `RESTRICTED`, then clear the list to become `PUBLIC`. + +**Platform deletion burns the identifier.** The delete is soft, but the identifier check does not exclude +deleted rows, so the name can never be reused. Recovering from that has meant deleting a row from the +database. + +## Auditing where building block code actually lives + +A `kit/`-style directory that nothing in a repository references can still be live, because a BBD reaches +it by Git repository path. To answer the question properly, read the sources off the instance: + +```sh +# per definition +GET /api/meshobjects/meshbuildingblockdefinitionversions?buildingBlockDefinitionUuid= +# then read each version's +.spec.source.terraform.repositoryUrl and .spec.source.terraform.repositoryPath +``` + +Two things to get right: + +- **Check versions, not definitions.** A definition's source moves over its lifetime. One observed + definition pointed at three different paths across v1–v37, ending on a hub module. +- **Check what live blocks are pinned to.** A released version stays orderable, so a path is only truly + dead when no active block runs a version that uses it. Map each block's + `spec.buildingBlockDefinitionVersionRef.uuid` back to a version number. + +A display name is a weak signal but a useful hint — a definition renamed to something like +"(Deprecated, don't use!)" is telling you where to look first. diff --git a/.agents/references/tenant-migration-case-study.md b/.agents/references/tenant-migration-case-study.md new file mode 100644 index 00000000..eff250ed --- /dev/null +++ b/.agents/references/tenant-migration-case-study.md @@ -0,0 +1,148 @@ +# Case study: migrating nine tenants off a custom STACKIT platform + +A worked example, kept because the failures are more instructive than the procedure. Nine meshTenants moved +from a hand-built custom platform (`stackit.sovereign`, owned by one workspace) to a platform deployed by +the STACKIT landing zone reference architecture (`likvid-stackit.global`), on a demo meshStack instance. + +Outcome: seven tenants migrated with their cloud project ids intact, two deleted as never-replicated, all +seven building blocks `SUCCEEDED`, all seven projects moved into the new platform's folder, no duplicate +project created at any point. + +## The inventory, and why the classes mattered + +| Class | Count | Procedure difference | +|---|---|---| +| Building-block-backed | 4 | Purge the old block before deleting the tenant | +| Replicator-created | 3 | No block to purge; seed had to be built from the live project | +| Never replicated | 2 | Deleted — `platformTenantId` was null, nothing to migrate | +| Also held in Terraform | 2 (subset of the above) | Needed `state rm` + `import` reconciliation afterwards | + +The four building-block-backed projects were exactly the four sitting at the organization root, because the +old BBD set `parent_container_id` to the organization. The three in folders predated the block and were made +by the replicator. That correlation was the fastest way to classify them. + +Ordering mattered: the three replicator-created tenants went first because nothing depended on them, which +meant the procedure was exercised twice before it met the project running a live Kubernetes cluster. + +## Failure 1: pre-existing role assignments + +The first migration failed on the apply with: + +``` +Error: Error while checking for duplicate role assignments +found a duplicate role assignment +``` + +The seed carried the project but no role assignments, so Terraform tried to create all six grants the +meshProject called for. Two already existed, put there by the old replicator. The duplicate error failed +the **whole apply**, so the project was renamed and moved but no grants were created and the run never +reached its output-collection step. + +The fix looked obvious — put every existing grant in the seed — and was wrong in the other direction. A +grant in the seed that the run's configuration does not contain gets **destroyed**. On the project hosting +the Kubernetes cluster that would have stripped `owner` from the cluster's own service accounts. + +So the set has to be computed: read the meshProject's user bindings, map them through the BBD's +`role_mapping`, and adopt only the intersection with what exists in the cloud. Service accounts and legacy +roles fall outside that set and stay unmanaged. On one project this left an out-of-band `reader` grant +untouched, which is the correct outcome — the block never asked to manage it. + +Reading the bindings needed the non-obvious endpoint, because `meshusers` is 403 without a `USER_*` +permission. The role-name mapping (`Project Admin → admin`, `Project User → user`, +`Project Reader → reader`) was confirmed against a real run's `users` input rather than assumed. + +## Failure 2: labels the provider cannot clear + +The second migration failed differently: + +``` +Error: Provider produced inconsistent result after apply +… .labels: was null, but now cty.MapVal(map[string]cty.Value{"project":…, "workspace":…}) +``` + +The hub module computes `labels = length(var.labels) > 0 ? var.labels : null` and the BBD passes `{}`, so +every adoption plans `labels -> null`. STACKIT ignores that instead of clearing the labels, and the +provider then fails the apply. Two of the three replicator-created projects carried the replicator's own +`project` / `workspace` labels and hit it. + +Clearing the labels fixed those two. The third project was worse: it carried `billingReference=""`, and + +``` +DELETE /v2/projects//labels?keys=billingReference +→ 409 The label [billingReference] is protected and can't be changed or deleted +``` + +even for the organization owner. `stackit project update --label` can only add labels, and a `PATCH` with +`labels: {}` is a merge that silently keeps everything. What works is an explicit null: + +```sh +curl -X PATCH -H "Authorization: Bearer $(stackit auth get-access-token)" \ + -H 'Content-Type: application/json' -d '{"labels":{"billingReference":null}}' \ + "https://resource-manager.api.stackit.cloud/v2/projects/" +``` + +Three things this taught, in order of usefulness: + +**Lying in the seed does not work.** Recording `labels: null` in the pushed state looks like it should +suppress the diff, but the runner refreshes before it plans, and the refresh puts reality back. + +**A failed first run has a consequence beyond the red status.** The apply dies before the +output-collection step, so the new meshTenant never gets a `platform_tenant_id`. Anything reading that +attribute downstream — in this case a `stackit_project_id` output feeding four other units — would have +broken. That is why this tenant could not simply be pushed through with a known-failing block. + +**Diagnose the origin before assuming a configuration bug.** The label was not set by any repository: +`billingReference` appeared nowhere in the foundation, the hub or the sibling foundation, and all 44 +projects belonging to the other foundation's install had no labels at all. The old building block did not +set it either — its `labels` input was `{}`, the same as sibling projects that came out clean, and it +created the project 21 seconds after itself with no labels. A scan of all 107 projects in the organization +found the key on exactly two, both the same platform project and its predecessor, both updated a month +after creation. It was a manual Portal edit, and STACKIT exposes the field as a first-class *Billing +Reference* project setting whose storage happens to be a protected label. The Portal renders "absent" and +"present but empty" identically, so the UI cannot distinguish the broken state from the healthy one. + +## What went right, and why + +**The adopt harness.** After the first failure, seed construction moved from rewriting the old block's +state to importing the live project into a byte-identical copy of the new module and planning it. That +turned every subsequent migration into a reviewed change: the harness printed the plan and refused to emit +a seed unless it showed no replacements and no destroys. Every later run came back +`0 added, 1 changed, 0 destroyed`, and the one change was computed timestamps. + +**The race was never lost** across seven migrations. The documented fifteen-second window against a +sub-second script is a wide enough margin that the timing is not the risk; the lookup logic is. + +**Container moves are genuinely in-place.** Terraform planned `parent_container_id` as an update, the +project id survived, and the meshTenant's `platform_tenant_id` — which *is* the project id — stayed valid. +`owner_email` is create-only in STACKIT, so the provider recorded the new value while the actual owner did +not change; that is a silent state-versus-reality divergence worth knowing about but harmless here. + +**Adoption avoided a permanent mess.** STACKIT deletions sit in `DELETING` for weeks — observed at 21 and +134 days on this organization — and block the parent folder's deletion with a `409`. Creating replacements +and deleting the originals would have left seven tombstones inside the new platform's folder, making it +undeletable for good. Adopting sidesteps the problem instead of paying for it seven times. + +## Retiring the old platform + +Once empty, the intent was to unpublish it. That is impossible: `UNPUBLISHED` requires +`restriction = PRIVATE`, and `PRIVATE` is permanently forbidden once a platform has been published. The +three `400` messages that establish this are in the cookbook. + +What worked instead was deactivating its three landing zones and deleting the old mandatory BBD. Since a +meshTenant cannot exist without a landing zone, that makes the platform unorderable regardless of its +availability. The platform row was deliberately left `ACTIVE`: deleting it burns the identifier +permanently, and it was still the `SUCCESSFUL` metering comparison against the new platform's `FAILED` +metering — a useful control while that question was open. + +## Two corrections the migration forced on the plan + +Both were assumptions that had been written down as facts, and both were caught only by reading the live +instance: + +- A landing zone the plan claimed existed did not, because the deployment never set the input that creates + it. That changed a downstream design decision (a select would show one option, not two). +- The claim that platform availability "cannot be set in code at all" was wrong. The API accepts it; the + hub module's `ignore_changes` on it is a convention, not a limitation. + +The lesson is narrow and worth stating: in migration work, verify the current state against the API before +planning around it, especially for anything a document asserts about a deployed system. diff --git a/.agents/references/tenant-migration-runbook.md b/.agents/references/tenant-migration-runbook.md new file mode 100644 index 00000000..ac2207d3 --- /dev/null +++ b/.agents/references/tenant-migration-runbook.md @@ -0,0 +1,154 @@ +# Tenant migration runbook + +The per-tenant procedure for moving a meshTenant to another meshPlatform while the cloud resource behind +it stays where it is and keeps its id. + +## Why import is not an option + +Three separate blockers, and each one is worth knowing because each rules out a different shortcut. + +**meshStack refuses the import.** `MeshTenantService.verifyNoImportIntoCustomPlatformImplementedUsingBuildingBlocks` +treats `meshTenant.localId != null` on create as an import and throws `CustomPlatformImportUnsupportedException` +when the platform's category is `CUSTOM` and the landing zone has mandatory building block definitions. +The guard's own comment names the problem: the building block that is supposed to create the tenant would +have to run the equivalent of a `terraform import` to construct its initial state. That is exactly what +this runbook does — by hand, outside meshStack, where the guard cannot see it. + +**A mandatory BBD with a required input blocks tenant creation entirely.** An earlier guard, +`verifyMandatoryBuildingBlocksCreatableOrThrow`, rejects a mandatory BBD carrying an input with neither +an automatic assignment nor a default: `requires Building Blocks that cannot be created due to missing +inputs: :`. A BBD whose inputs are all `STATIC`, `PROJECT_IDENTIFIER` or `USER_PERMISSIONS` +passes, which is what lets step 3 create a tenant through the plain API with no inputs at all. + +**Identifiers cannot be reproduced.** `metadata.name` *is* the identifier, so renaming an object is a +replace. Importing an old landing zone under a new name only makes the next plan destroy and recreate it. + +## The procedure + +Steps 3 and 4 are one scripted unit, not two manual ones — see *The race*. + +1. **Disarm the old building block.** + `DELETE /api/meshobjects/meshbuildingblocks//purge` answers `202`, and within about + ten seconds the block reads `forcePurge = true` and `lifecycle.state = DELETED`. No destroy runs and + the cloud resource is untouched. Confirm `forcePurge` before going on — this is what makes step 2 safe. + Skip this step for a tenant that has no block. + +2. **Delete the old meshTenant.** Also `202`, and also non-destructive *once the block is purged*. Without + the purge, this runs the block's teardown and destroys the cloud resource. + +3. **Create the new meshTenant with no `platform_tenant_id`.** Setting it is what makes meshStack call the + adoption an import. Leave it unset, `localId` stays null, and the guard returns early. meshStack then + creates the mandatory building block itself. + +4. **Seed the new block's state before its runner starts.** `POST` the prepared state to the block's state + address. See `building-block-state-doctoring.md`. + +5. **Let the run finish and check what it did.** One in-place update on the cloud resource, no creates of + the resource itself. Creates of role assignments are normal — see below. + +## Building the seed state + +Two sources, and the second is better. + +**From the old block's state**, when the tenant has one: `GET` the old block's state and rewrite it into +the new module's resource shape. This works when the two modules' resource addresses agree, which they +usually do if both come from the same hub module lineage. Watch for renamed resources — a `for_each` +resource renamed between module versions needs its instances re-keyed and `each: "map"` set. + +**From the live cloud resource**, which is more robust and works for every class: stand up a throwaway +directory containing a copy of the *new* module's `buildingblock/`, `tofu import` the real resource into +it, plan, and use the resulting state as the seed. This reads reality instead of trusting an old state, +and the plan doubles as the safety check. Copy the buildingblock directory **with its non-`.tf` files** — +a missing `SUMMARY.md.tftpl` breaks the plan. + +Whichever source, normalise before pushing: + +- **`terraform_version` must not exceed the runner's.** OpenTofu refuses to read a state written by a + newer version. Read the runner's version off a successful run's state and rewrite the field to match. +- **`serial`** to 1 and **`outputs`** to `{}` — the run recomputes them. +- Drop resource entries with no instances. + +## Access parity: the trap that fails whole runs + +The seed's role assignments are the ones Terraform adopts. Get the set wrong in either direction and it +costs you: + +- **Too few** — a grant the run wants but the seed lacks comes back as a duplicate error from the cloud + API, and that fails the **entire apply**, not just that grant. +- **Too many** — a grant in the seed that the run's configuration does not contain is **destroyed**, + silently removing somebody's access. On a platform project this can strip the service accounts that + operate the platform. + +So compute the set rather than guessing it: read the meshProject's user bindings, map them through the +BBD's `role_mapping` input, and adopt only the intersection with what already exists in the cloud. +Everything else — service accounts, legacy roles, grants held by people who are not project members — +stays unmanaged and therefore untouched. + +## The race + +Between creating the tenant and pushing the state, the runner may start on its own. Measured window: +`PENDING` to `IN_PROGRESS` in about fifteen seconds, against a script that needs under half a second. The +margin is wide, so the race is only lost when the script fails to *find* the new block at all. + +Two ways to lose it, both from the lookup rather than the timing: + +- **A `jq` filter that rebinds `.`.** `select(($b|split(" "))|index(.)|not)` pipes `.` into the split + array, so the filter never matches. Use `[uuids] - $b | .[0] // empty`, and unit-test it against both a + hit and a miss before running it live. +- **A polling loop with no `sleep`.** Two hundred back-to-back requests finish long before meshStack has + created the block. + +**Recovery when the race is lost.** The run creates a second cloud resource and saves its own state over +your push. Nothing is destroyed and the original resource is untouched: + +1. Let the run finish. Purging mid-apply just leaves the duplicate unowned. +2. Read the duplicate's id from the block's `status.outputs`, then purge the block and delete the tenant. +3. **Move the duplicate to the organization root before deleting it.** Deleting it in place leaves a + tombstone inside the target folder that never clears and blocks that folder's deletion for good. +4. Rebuild the seed and re-run. The old tenant is already gone, so skip the disarm step. + +## Recovery when the first run fails + +More common than losing the race, and equally recoverable. A failed apply still saves its state and the +cloud resource keeps its id. + +1. **Read the run log.** The run's `_links.downloadLogs` needs one specific `Accept` value and returns + 406 for everything else, including `*/*`. See the cookbook. +2. Fix the cause. It is almost always the access-parity set or a label. +3. Purge the new block, then delete the new tenant. Confirm `forcePurge` before deleting. +4. Rebuild the seed and re-run the migration unchanged. Its delete of the old tenant is then a harmless + `404`, because that tenant is already gone. + +There is no way to re-trigger a failed run in place: the block exposes only `forcePurge`, `meshtenant` +and `self`, and a `runs` sub-resource does not exist. Purge and re-create is the only path. + +## Labels can block the adoption outright + +If the module computes `labels = length(var.labels) > 0 ? var.labels : null` and the BBD passes `{}`, then +every adoption of a resource that already has labels plans `labels -> null`. When the cloud provider +ignores that instead of clearing them, the provider fails the apply: + +``` +Error: Provider produced inconsistent result after apply +… .labels: was null, but now cty.MapVal(map[string]cty.Value{…}) +``` + +Check the resource's labels before adopting it. Clearing them is usually enough. A **protected** label is +worse: the dedicated label-delete endpoint refuses it, and only an explicit `null` in the update payload +removes it. Lying about labels in the seed does not work, because the runner refreshes before it plans and +the refresh puts reality back. + +## Tenants held in Terraform state need reconciliation + +When some repository holds the tenant as a `meshstack_tenant` resource, the out-of-band migration leaves +that state stale, and the next plan will try to fix it by replacing the tenant. Per tenant, in this order: + +1. Run the migration, so the new tenant and its adopted block exist. +2. `state rm` the old tenant address, then `import` the new tenant **at the same address**. The import id + is the tenant uuid, or the legacy `workspace.project.platform.location` composite. +3. Only now change `platform_ref` and `landing_zone_ref` in the configuration. +4. Re-plan. It must come back clean. **Anything showing a replace means stop.** + +Doing step 3 before step 2 is what produces the replace. And check what reads the tenant's +`platform_tenant_id`: adoption preserves it, so those consumers keep resolving, but confirm by reading the +output back rather than assuming. diff --git a/.agents/skills/tenant-migration/SKILL.md b/.agents/skills/tenant-migration/SKILL.md new file mode 100644 index 00000000..a6fb9aaf --- /dev/null +++ b/.agents/skills/tenant-migration/SKILL.md @@ -0,0 +1,100 @@ +--- +name: tenant-migration +description: > + Move meshTenants between meshPlatforms without destroying the cloud resources behind them. Use when + asked to migrate, consolidate or retire tenants on a custom platform whose landing zones carry + mandatory building block definitions — the case where meshStack's own import path is refused. Covers + inventory, the per-tenant runbook, building block state doctoring, Terraform state reconciliation, + retiring the old platform, and the traps that fail a run. +--- + +# Tenant Migration Skill + +Moving a meshTenant from one meshPlatform to another looks like an import and is not one. On a custom +platform whose landing zone carries a mandatory building block definition, meshStack refuses the import +outright, and Terraform cannot do it either because `spec.platform_ref` is `RequiresReplace` — replacing +a meshTenant destroys its building block, and the building block's teardown destroys the live cloud +resource. + +The way through is to create the new tenant plainly and make the link to the existing cloud resource in +**Terraform state** instead of through the meshStack API. The building block then adopts the resource +rather than creating a replacement. + +Read these references before starting: + +- `.agents/references/tenant-migration-runbook.md` — the per-tenant procedure and its ordering +- `.agents/references/building-block-state-doctoring.md` — reaching and rewriting a block's state +- `.agents/references/meshstack-api-cookbook.md` — endpoints, media types and the traps +- `.agents/references/tenant-migration-case-study.md` — a worked migration of nine tenants + +--- + +## Two rules that prevent the expensive mistakes + +- **Never let Terraform replace a meshTenant.** Replacing it destroys its building block and with it the + live cloud resource. If a plan shows a tenant being replaced or destroyed, stop and re-read the state. +- **Never delete a cloud resource in place.** Move it to the organization root first. On STACKIT a + deleted project sits in `DELETING` for weeks and blocks its parent folder's deletion permanently. + +## Method + +1. **Inventory first, and classify.** Tenants that look alike migrate differently. See *Classify the + tenants* below. +2. **Prove the procedure on a throwaway resource** before touching anything real, and snapshot whatever + you are about to mutate so you can put it back. +3. **Assert on the plan, not on your intention.** Refuse to proceed unless the plan shows zero destroys + and no replacements. Automate that check so it cannot be skipped when you are tired. +4. **Interlock on identity.** Pass the expected cloud resource id into every script and abort when what + you find does not match. +5. **Migrate the cheapest tenant first** — the one nothing depends on — so the procedure is exercised + before it meets the tenant that matters. + +## Classify the tenants + +The class decides the procedure, so establish it before planning any work: + +| Class | How to spot it | What changes | +|---|---|---| +| **Building-block-backed** | A block instance exists for the tenant on the old mandatory BBD | Purge the block before deleting the tenant, or the teardown destroys the cloud resource | +| **Replicator-created** | No block instance; the resource predates the BBD | No block to purge; the seed state must be built from the live resource | +| **Never replicated** | `spec.localId` / `platformTenantId` is null | Nothing to migrate — delete the tenant | +| **Held in IaC** | A `meshstack_tenant` resource in some repo's state points at it | Needs state reconciliation on top of the migration, or the next plan replaces it | + +The last class is the dangerous one, because the damage arrives later, from a plan someone else runs. + +## Risk method + +**Experiment on throwaway resources, but snapshot even then.** A throwaway proves the mechanism; the +snapshot is what lets you retry after the first attempt teaches you something. Concretely: + +- Before mutating an object through an API, `GET` it and keep the response. That file is your rollback. +- Before deleting an attribute, record its value in the command's own comment, so the restore command is + written down next to the destructive one. +- Prefer an operation the provider models as an in-place update over one it models as a replacement, and + confirm which it is by planning rather than by reading documentation. + +**Weigh the blast radius, not the action.** Deleting a landing zone reads as more destructive than +editing a platform's availability, but the landing zone delete is a reversible deactivation while the +availability change is guarded by rules that can leave the platform in a state you cannot return from. +Check what each operation actually does before ranking them. + +## Retiring the old platform + +Do this only once no tenants remain on it. + +**Deactivating its landing zones is the effective retirement.** A meshTenant cannot exist without a +landing zone, so a deactivated landing zone makes the platform unorderable whatever its availability +says. `DELETE /api/meshobjects/meshlandingzones/` is a disable, not a removal: the object +stays readable with `lifecycle.state = DEACTIVATED` and existing tenants are untouched. + +**Do not expect to unpublish the platform.** Once a platform has been published, `UNPUBLISHED` is +unreachable — see the cookbook. And deleting the platform burns its identifier permanently even though +the delete is soft, so keep the row unless you are certain the identifier is never wanted again. + +## Telling live code from dead code + +A `kit/`-style directory that no repository references may still be live, because a building block +definition reaches it by Git repository path rather than by a module source. Grepping the repository +cannot answer the question. Ask the instance instead, and check **versions**, not definitions: a +definition's source moves over its lifetime, so the current version may point somewhere entirely +different from the version an old block is pinned to. See the cookbook for the two endpoints. From 92c9ca1fd713f7f25b48465b87d89e1f4b5e9410 Mon Sep 17 00:00:00 2001 From: Andreas Grub Date: Wed, 19 Aug 2026 10:30:45 +0200 Subject: [PATCH 2/4] docs(agents): refine the tenant-migration dump into a usable skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The raw capture read as a session log: the SKILL.md restated the runbook, the method was buried below the narrative, and the tool-like material was hard to reach from the place an agent starts. Restructure it as a decision layer over the references: - SKILL.md now opens with the reason adoption is the only path, the two rules that prevent an unrecoverable mistake, a six-step workflow, the tenant class table, and a per-tenant loop that gives the actual purge, lookup and state-push commands instead of describing them. Detail that belongs to one step moved into the reference it belongs to. - The risk method — throwaway plus snapshot, assert on the plan, prefer in-place over replacement, weigh the blast radius — is a named section with an executable plan gate, so it cannot be skipped by an operator who is tired. - The runbook leads with the procedure and closes with the three guards that rule out an import, since the reader arriving from SKILL.md already knows why. - State doctoring leads with the adopt harness, the recommended path, and spells out both harness setups as commands. - The cookbook states the `application/vnd.meshcloud.api.` prefix the media type table only implied, and shows one complete request. - The case study stays as evidence and is labelled as such. - AGENTS.md points at the skill, matching how it indexes `module` and `e2e-test`. Every technical claim is unchanged; the description field now names the concrete situation (`CustomPlatformImportUnsupportedException`, `RequiresReplace` on `spec.platform_ref`) so only an agent facing this problem loads the skill. Co-Authored-By: Claude Opus 5 (1M context) --- .../building-block-state-doctoring.md | 60 ++++-- .agents/references/meshstack-api-cookbook.md | 21 +- .../references/tenant-migration-case-study.md | 11 +- .../references/tenant-migration-runbook.md | 92 +++++---- .agents/skills/tenant-migration/SKILL.md | 195 +++++++++++++----- AGENTS.md | 13 ++ 6 files changed, 269 insertions(+), 123 deletions(-) diff --git a/.agents/references/building-block-state-doctoring.md b/.agents/references/building-block-state-doctoring.md index 44c12644..748c88d0 100644 --- a/.agents/references/building-block-state-doctoring.md +++ b/.agents/references/building-block-state-doctoring.md @@ -3,6 +3,9 @@ A building block run keeps its Terraform state in meshStack's own HTTP backend, and that state is reachable from outside the run. This is what makes tenant migration, adoption and run repair possible. +Read this when a building block must take over a resource it did not create, when a run has to be +repaired after a failed apply, or when a seed state has to be built for a tenant migration. + ## The endpoint ``` @@ -16,26 +19,22 @@ token works either way. No lock address is configured, so `-lock=false` is corre The API key needs `ADM_TFSTATE_LIST`, `ADM_TFSTATE_SAVE` and `ADM_TFSTATE_DELETE`. -## A local harness against a live block's state - -A throwaway directory with the same module source, a matching provider configuration and an `http` -backend pointing at the run's state can `tofu import`, `state rm` and `state mv` against it like any other -remote state. Full plan and apply cycles work too. - -It needs: - -- the module's `buildingblock/` directory copied **with its non-`.tf` files** — a missing template such as - `SUMMARY.md.tftpl` breaks the plan -- the BBD's inputs as tfvars -- `MESHSTACK_ENDPOINT` / `MESHSTACK_API_KEY` / `MESHSTACK_API_SECRET` in the environment, because the - buildingblock does not configure the meshstack provider itself - ## The adopt harness The safest way to build a seed state: a copy of the *new* module's `buildingblock/`, with a **local** state file rather than the HTTP backend. Import the live cloud resource into it, plan, then use the resulting state as the seed. +```sh +mkdir -p /tmp/adopt && cp -r modules///buildingblock/. /tmp/adopt/ +cd /tmp/adopt +tofu init +tofu import -var-file=bbd-inputs.tfvars '' '' +tofu plan -out=tfplan -var-file=bbd-inputs.tfvars +# gate the seed on the plan — see the risk method in .agents/skills/tenant-migration/SKILL.md +cp terraform.tfstate seed.tfstate +``` + Why this beats rewriting an old block's state: - it reads the resource's real attributes from the cloud API rather than trusting a stale record @@ -43,11 +42,36 @@ Why this beats rewriting an old block's state: - it works for tenants that never had a building block at all Keep the harness a byte-identical copy of the module the runner executes. If it drifts, its plan stops -predicting the run's plan, which is the only reason the harness is worth having. +predicting the run's plan, which is the only reason the harness is worth having. In particular, copy the +non-`.tf` files too — a missing template such as `SUMMARY.md.tftpl` breaks the plan. + +**Assert before writing.** Refuse to emit a seed unless the plan shows no replacements and no destroys. +Creates of subordinate resources such as role assignments are expected. The check belongs in the script +rather than in the operator's head; the SKILL.md risk method has the exact gate. + +## A harness against a live block's state + +The same throwaway directory can point at a running block's state instead of a local file, which allows +`tofu import`, `state rm` and `state mv` against it like any other remote state. Full plan and apply +cycles work too. + +```hcl +terraform { + backend "http" { + address = "/api/terraform/state/workspace//buildingBlock/" + } +} +``` + +```sh +export TF_HTTP_USERNAME=x TF_HTTP_PASSWORD="$MT" +tofu init +tofu state list -lock=false +``` -**Assert before you write.** Refuse to emit a seed unless the plan shows no replacements and no destroys. -Creates of subordinate resources such as role assignments are expected. A single grep on the plan output -is enough, and it belongs in the script rather than in your head. +It also needs the BBD's inputs as tfvars, and `MESHSTACK_ENDPOINT` / `MESHSTACK_API_KEY` / +`MESHSTACK_API_SECRET` in the environment, because the buildingblock does not configure the meshstack +provider itself. ## Normalising a state before pushing it diff --git a/.agents/references/meshstack-api-cookbook.md b/.agents/references/meshstack-api-cookbook.md index eb753f55..c4e508e9 100644 --- a/.agents/references/meshstack-api-cookbook.md +++ b/.agents/references/meshstack-api-cookbook.md @@ -17,6 +17,8 @@ MT=$(curl -s -u ":" -d 'grant_type=client_credentials' \ ## Media types that matter +The full header value is `application/vnd.meshcloud.api.`; the table lists the `` part. + | Object | `Accept` | Notes | |---|---|---| | meshTenant | `meshtenant.v3.hal+json` | Has `spec.localId` — the cloud resource id — but no uuid. | @@ -30,10 +32,19 @@ MT=$(curl -s -u ":" -d 'grant_type=client_credentials' \ | meshProject | `meshproject.v2.hal+json` | Tags are under `spec.tags`, **not** `metadata.tags`. | | meshProjectUserBinding | `meshprojectuserbinding.v3.hal+json` | | -## Traps, each of which cost real time +A request therefore looks like this: + +```sh +curl -sS --fail-with-body -H "Authorization: Bearer $MT" \ + -H 'Accept: application/vnd.meshcloud.api.meshtenant.v4-preview.hal+json' \ + "$MESHSTACK/api/meshobjects/meshtenants?workspaceIdentifier=$WS" +``` + +## Traps that cost real time **A 401 looks like an empty result.** `jq` on an error body yields `null` or an empty list, so a stale -token reads as "no tenants exist". Always check the HTTP status before believing an empty list. +token reads as "no tenants exist". Always check the HTTP status before believing an empty list — +`--fail-with-body` is enough. **`meshtenants` returns nothing without a filter.** Use `?workspaceIdentifier=`. There is no instance-wide listing, so an inventory means iterating workspaces. @@ -103,9 +114,9 @@ Going the other way, to public, is also ordered. `setAllowedWorkspaces()` reject not contain the owner until `wasOncePublished` is true, so the sequence is: publish first, which leaves the platform visibly `RESTRICTED`, then clear the list to become `PUBLIC`. -**Platform deletion burns the identifier.** The delete is soft, but the identifier check does not exclude -deleted rows, so the name can never be reused. Recovering from that has meant deleting a row from the -database. +**Platform deletion permanently consumes the identifier.** The delete is soft, but the identifier check +does not exclude deleted rows, so the name can never be reused. Recovering from that has meant deleting a +row from the database. ## Auditing where building block code actually lives diff --git a/.agents/references/tenant-migration-case-study.md b/.agents/references/tenant-migration-case-study.md index eff250ed..cd8a5ecb 100644 --- a/.agents/references/tenant-migration-case-study.md +++ b/.agents/references/tenant-migration-case-study.md @@ -1,6 +1,7 @@ # Case study: migrating nine tenants off a custom STACKIT platform -A worked example, kept because the failures are more instructive than the procedure. Nine meshTenants moved +Evidence rather than instruction: the procedure itself is in `tenant-migration-runbook.md`, and this file +records what actually happened, because the failures are more instructive than the procedure. Nine meshTenants moved from a hand-built custom platform (`stackit.sovereign`, owned by one workspace) to a platform deployed by the STACKIT landing zone reference architecture (`likvid-stackit.global`), on a demo meshStack instance. @@ -83,7 +84,7 @@ curl -X PATCH -H "Authorization: Bearer $(stackit auth get-access-token)" \ Three things this taught, in order of usefulness: -**Lying in the seed does not work.** Recording `labels: null` in the pushed state looks like it should +**A seed that misreports reality does not work.** Recording `labels: null` in the pushed state looks like it should suppress the diff, but the runner refreshes before it plans, and the refresh puts reality back. **A failed first run has a consequence beyond the red status.** The apply dies before the @@ -117,7 +118,7 @@ project id survived, and the meshTenant's `platform_tenant_id` — which *is* th `owner_email` is create-only in STACKIT, so the provider recorded the new value while the actual owner did not change; that is a silent state-versus-reality divergence worth knowing about but harmless here. -**Adoption avoided a permanent mess.** STACKIT deletions sit in `DELETING` for weeks — observed at 21 and +**Adoption avoided a permanent obstruction.** STACKIT deletions sit in `DELETING` for weeks — observed at 21 and 134 days on this organization — and block the parent folder's deletion with a `409`. Creating replacements and deleting the originals would have left seven tombstones inside the new platform's folder, making it undeletable for good. Adopting sidesteps the problem instead of paying for it seven times. @@ -130,8 +131,8 @@ three `400` messages that establish this are in the cookbook. What worked instead was deactivating its three landing zones and deleting the old mandatory BBD. Since a meshTenant cannot exist without a landing zone, that makes the platform unorderable regardless of its -availability. The platform row was deliberately left `ACTIVE`: deleting it burns the identifier -permanently, and it was still the `SUCCESSFUL` metering comparison against the new platform's `FAILED` +availability. The platform row was deliberately left `ACTIVE`: deleting it permanently consumes the +identifier, and it was still the `SUCCESSFUL` metering comparison against the new platform's `FAILED` metering — a useful control while that question was open. ## Two corrections the migration forced on the plan diff --git a/.agents/references/tenant-migration-runbook.md b/.agents/references/tenant-migration-runbook.md index ac2207d3..3ff8dd5a 100644 --- a/.agents/references/tenant-migration-runbook.md +++ b/.agents/references/tenant-migration-runbook.md @@ -1,33 +1,23 @@ # Tenant migration runbook The per-tenant procedure for moving a meshTenant to another meshPlatform while the cloud resource behind -it stays where it is and keeps its id. +it stays where it is and keeps its id. Read this together with `.agents/skills/tenant-migration/SKILL.md`, +which carries the workflow, the tenant classes and the risk method. -## Why import is not an option +## Before the first tenant -Three separate blockers, and each one is worth knowing because each rules out a different shortcut. - -**meshStack refuses the import.** `MeshTenantService.verifyNoImportIntoCustomPlatformImplementedUsingBuildingBlocks` -treats `meshTenant.localId != null` on create as an import and throws `CustomPlatformImportUnsupportedException` -when the platform's category is `CUSTOM` and the landing zone has mandatory building block definitions. -The guard's own comment names the problem: the building block that is supposed to create the tenant would -have to run the equivalent of a `terraform import` to construct its initial state. That is exactly what -this runbook does — by hand, outside meshStack, where the guard cannot see it. - -**A mandatory BBD with a required input blocks tenant creation entirely.** An earlier guard, -`verifyMandatoryBuildingBlocksCreatableOrThrow`, rejects a mandatory BBD carrying an input with neither -an automatic assignment nor a default: `requires Building Blocks that cannot be created due to missing -inputs: :`. A BBD whose inputs are all `STATIC`, `PROJECT_IDENTIFIER` or `USER_PERMISSIONS` -passes, which is what lets step 3 create a tenant through the plain API with no inputs at all. - -**Identifiers cannot be reproduced.** `metadata.name` *is* the identifier, so renaming an object is a -replace. Importing an old landing zone under a new name only makes the next plan destroy and recreate it. +- An API key with the tfstate permissions listed in `building-block-state-doctoring.md`, plus read access + to tenants, blocks, projects and project user bindings. +- A working adopt harness — a copy of the *new* module's `buildingblock/` that imports the live cloud + resource into a local state file. Prove it on a throwaway resource first. +- The runner's OpenTofu version, read off a successful run's state, because the seed's + `terraform_version` must not exceed it. ## The procedure Steps 3 and 4 are one scripted unit, not two manual ones — see *The race*. -1. **Disarm the old building block.** +1. **Purge the old building block.** `DELETE /api/meshobjects/meshbuildingblocks//purge` answers `202`, and within about ten seconds the block reads `forcePurge = true` and `lifecycle.state = DELETED`. No destroy runs and the cloud resource is untouched. Confirm `forcePurge` before going on — this is what makes step 2 safe. @@ -44,7 +34,7 @@ Steps 3 and 4 are one scripted unit, not two manual ones — see *The race*. address. See `building-block-state-doctoring.md`. 5. **Let the run finish and check what it did.** One in-place update on the cloud resource, no creates of - the resource itself. Creates of role assignments are normal — see below. + the resource itself. Creates of role assignments are normal — see *Access parity*. ## Building the seed state @@ -70,8 +60,8 @@ Whichever source, normalise before pushing: ## Access parity: the trap that fails whole runs -The seed's role assignments are the ones Terraform adopts. Get the set wrong in either direction and it -costs you: +The seed's role assignments are the ones Terraform adopts. Getting the set wrong in either direction is +expensive: - **Too few** — a grant the run wants but the seed lacks comes back as a duplicate error from the cloud API, and that fails the **entire apply**, not just that grant. @@ -84,6 +74,22 @@ BBD's `role_mapping` input, and adopt only the intersection with what already ex Everything else — service accounts, legacy roles, grants held by people who are not project members — stays unmanaged and therefore untouched. +## Labels can block the adoption outright + +If the module computes `labels = length(var.labels) > 0 ? var.labels : null` and the BBD passes `{}`, then +every adoption of a resource that already has labels plans `labels -> null`. When the cloud provider +ignores that instead of clearing them, the provider fails the apply: + +``` +Error: Provider produced inconsistent result after apply +… .labels: was null, but now cty.MapVal(map[string]cty.Value{…}) +``` + +Check the resource's labels before adopting it. Clearing them is usually enough. A **protected** label is +worse: the dedicated label-delete endpoint refuses it, and only an explicit `null` in the update payload +removes it. Recording `labels: null` in the seed does not help, because the runner refreshes before it +plans and the refresh puts reality back. + ## The race Between creating the tenant and pushing the state, the runner may start on its own. Measured window: @@ -99,13 +105,13 @@ Two ways to lose it, both from the lookup rather than the timing: created the block. **Recovery when the race is lost.** The run creates a second cloud resource and saves its own state over -your push. Nothing is destroyed and the original resource is untouched: +the pushed one. Nothing is destroyed and the original resource is untouched: 1. Let the run finish. Purging mid-apply just leaves the duplicate unowned. 2. Read the duplicate's id from the block's `status.outputs`, then purge the block and delete the tenant. 3. **Move the duplicate to the organization root before deleting it.** Deleting it in place leaves a tombstone inside the target folder that never clears and blocks that folder's deletion for good. -4. Rebuild the seed and re-run. The old tenant is already gone, so skip the disarm step. +4. Rebuild the seed and re-run. The old tenant is already gone, so skip the purge step. ## Recovery when the first run fails @@ -122,22 +128,6 @@ cloud resource keeps its id. There is no way to re-trigger a failed run in place: the block exposes only `forcePurge`, `meshtenant` and `self`, and a `runs` sub-resource does not exist. Purge and re-create is the only path. -## Labels can block the adoption outright - -If the module computes `labels = length(var.labels) > 0 ? var.labels : null` and the BBD passes `{}`, then -every adoption of a resource that already has labels plans `labels -> null`. When the cloud provider -ignores that instead of clearing them, the provider fails the apply: - -``` -Error: Provider produced inconsistent result after apply -… .labels: was null, but now cty.MapVal(map[string]cty.Value{…}) -``` - -Check the resource's labels before adopting it. Clearing them is usually enough. A **protected** label is -worse: the dedicated label-delete endpoint refuses it, and only an explicit `null` in the update payload -removes it. Lying about labels in the seed does not work, because the runner refreshes before it plans and -the refresh puts reality back. - ## Tenants held in Terraform state need reconciliation When some repository holds the tenant as a `meshstack_tenant` resource, the out-of-band migration leaves @@ -152,3 +142,23 @@ that state stale, and the next plan will try to fix it by replacing the tenant. Doing step 3 before step 2 is what produces the replace. And check what reads the tenant's `platform_tenant_id`: adoption preserves it, so those consumers keep resolving, but confirm by reading the output back rather than assuming. + +## Background: why import is not an option + +Three separate blockers, and each one is worth knowing because each rules out a different shortcut. + +**meshStack refuses the import.** `MeshTenantService.verifyNoImportIntoCustomPlatformImplementedUsingBuildingBlocks` +treats `meshTenant.localId != null` on create as an import and throws `CustomPlatformImportUnsupportedException` +when the platform's category is `CUSTOM` and the landing zone has mandatory building block definitions. +The guard's own comment names the problem: the building block that is supposed to create the tenant would +have to run the equivalent of a `terraform import` to construct its initial state. That is exactly what +this runbook does — by hand, outside meshStack, where the guard cannot see it. + +**A mandatory BBD with a required input blocks tenant creation entirely.** An earlier guard, +`verifyMandatoryBuildingBlocksCreatableOrThrow`, rejects a mandatory BBD carrying an input with neither +an automatic assignment nor a default: `requires Building Blocks that cannot be created due to missing +inputs: :`. A BBD whose inputs are all `STATIC`, `PROJECT_IDENTIFIER` or `USER_PERMISSIONS` +passes, which is what lets step 3 create a tenant through the plain API with no inputs at all. + +**Identifiers cannot be reproduced.** `metadata.name` *is* the identifier, so renaming an object is a +replace. Importing an old landing zone under a new name only makes the next plan destroy and recreate it. diff --git a/.agents/skills/tenant-migration/SKILL.md b/.agents/skills/tenant-migration/SKILL.md index a6fb9aaf..ac06e975 100644 --- a/.agents/skills/tenant-migration/SKILL.md +++ b/.agents/skills/tenant-migration/SKILL.md @@ -1,82 +1,152 @@ --- name: tenant-migration description: > - Move meshTenants between meshPlatforms without destroying the cloud resources behind them. Use when - asked to migrate, consolidate or retire tenants on a custom platform whose landing zones carry - mandatory building block definitions — the case where meshStack's own import path is refused. Covers - inventory, the per-tenant runbook, building block state doctoring, Terraform state reconciliation, - retiring the old platform, and the traps that fail a run. + Move meshTenants to a different meshPlatform without destroying the cloud resources behind them, by + adopting each resource into the new building block's Terraform state. Use when asked to migrate, + consolidate or retire meshTenants, especially off a custom platform whose landing zones carry + mandatory building block definitions — the case where meshStack rejects a create carrying + `spec.localId` with `CustomPlatformImportUnsupportedException` and Terraform reports + `spec.platform_ref` as `RequiresReplace`. Covers tenant classes, the per-tenant runbook, building + block state doctoring, access parity, Terraform state reconciliation, and retiring the emptied + platform. --- # Tenant Migration Skill -Moving a meshTenant from one meshPlatform to another looks like an import and is not one. On a custom -platform whose landing zone carries a mandatory building block definition, meshStack refuses the import -outright, and Terraform cannot do it either because `spec.platform_ref` is `RequiresReplace` — replacing -a meshTenant destroys its building block, and the building block's teardown destroys the live cloud +Moving a meshTenant from one meshPlatform to another looks like an import and is not one. meshStack +refuses the import when the target platform is `CUSTOM` and its landing zone carries a mandatory +building block definition. Terraform cannot do it either: `spec.platform_ref` is `RequiresReplace`, +replacing a meshTenant destroys its building block, and that block's teardown destroys the live cloud resource. -The way through is to create the new tenant plainly and make the link to the existing cloud resource in -**Terraform state** instead of through the meshStack API. The building block then adopts the resource -rather than creating a replacement. - -Read these references before starting: - -- `.agents/references/tenant-migration-runbook.md` — the per-tenant procedure and its ordering -- `.agents/references/building-block-state-doctoring.md` — reaching and rewriting a block's state -- `.agents/references/meshstack-api-cookbook.md` — endpoints, media types and the traps -- `.agents/references/tenant-migration-case-study.md` — a worked migration of nine tenants +The way through is **adoption**. Create the new meshTenant plainly, with no `platform_tenant_id`, so +meshStack sees an ordinary create and provisions the mandatory building block itself. Then write the +link to the existing cloud resource into the new block's **Terraform state**, before its runner starts. +The block adopts the resource instead of creating a replacement. --- ## Two rules that prevent the expensive mistakes -- **Never let Terraform replace a meshTenant.** Replacing it destroys its building block and with it the - live cloud resource. If a plan shows a tenant being replaced or destroyed, stop and re-read the state. +- **Never let Terraform replace a meshTenant.** Replacing it destroys its building block and with it + the live cloud resource. A plan that shows a meshTenant being replaced or destroyed means stop and + re-read the state. - **Never delete a cloud resource in place.** Move it to the organization root first. On STACKIT a - deleted project sits in `DELETING` for weeks and blocks its parent folder's deletion permanently. - -## Method - -1. **Inventory first, and classify.** Tenants that look alike migrate differently. See *Classify the - tenants* below. -2. **Prove the procedure on a throwaway resource** before touching anything real, and snapshot whatever - you are about to mutate so you can put it back. -3. **Assert on the plan, not on your intention.** Refuse to proceed unless the plan shows zero destroys - and no replacements. Automate that check so it cannot be skipped when you are tired. -4. **Interlock on identity.** Pass the expected cloud resource id into every script and abort when what - you find does not match. -5. **Migrate the cheapest tenant first** — the one nothing depends on — so the procedure is exercised + deleted project sits in `DELETING` for weeks and permanently blocks its parent folder's deletion. + +--- + +## Workflow + +1. **Inventory the tenants and classify each one.** The class decides the procedure — see *Classify the + tenants*. Listing tenants needs a per-workspace filter and the right media type; both are in + `.agents/references/meshstack-api-cookbook.md`. +2. **Build the adopt harness and prove it on a throwaway resource.** The harness is a copy of the *new* + module's `buildingblock/` with a local state file, into which the live cloud resource is imported. + See `.agents/references/building-block-state-doctoring.md`. +3. **Order the tenants cheapest first** — the one nothing depends on — so the procedure is exercised before it meets the tenant that matters. +4. **Run the per-tenant loop**, one tenant at a time, checking the run before starting the next. +5. **Reconcile Terraform state** for every tenant held in some repository's state, before anyone re-plans + that repository. See `.agents/references/tenant-migration-runbook.md` § *Tenants held in Terraform + state*. +6. **Retire the old platform** once no tenants remain on it — see *Retiring the old platform*. + +--- ## Classify the tenants -The class decides the procedure, so establish it before planning any work: +Tenants that look alike migrate differently, so establish the class before planning any work. | Class | How to spot it | What changes | |---|---|---| | **Building-block-backed** | A block instance exists for the tenant on the old mandatory BBD | Purge the block before deleting the tenant, or the teardown destroys the cloud resource | | **Replicator-created** | No block instance; the resource predates the BBD | No block to purge; the seed state must be built from the live resource | | **Never replicated** | `spec.localId` / `platformTenantId` is null | Nothing to migrate — delete the tenant | -| **Held in IaC** | A `meshstack_tenant` resource in some repo's state points at it | Needs state reconciliation on top of the migration, or the next plan replaces it | +| **Held in IaC** | A `meshstack_tenant` resource in some repository's state points at it | Needs state reconciliation on top of the migration, or the next plan replaces the tenant | + +The last class is the dangerous one, because the damage arrives later, out of a plan someone else runs. + +--- + +## The per-tenant loop + +`$MT` is a meshStack bearer token; the cookbook shows how to mint one. Steps 3 and 4 are a single +scripted unit — the runner may start on its own between them. + +1. **Purge the old building block.** Skip for a tenant that has no block. + + ```sh + curl -sS -X DELETE -H "Authorization: Bearer $MT" \ + "$MESHSTACK/api/meshobjects/meshbuildingblocks/$OLD_BB_UUID/purge" + ``` + + The answer is `202`. Within about ten seconds the block reads `forcePurge = true` and + `lifecycle.state = DELETED`, and no destroy run happens. Confirm `forcePurge` before step 2 — the + purge is what makes the delete safe. + +2. **Delete the old meshTenant.** Also `202`, and also non-destructive *once the block is purged*. + Without the purge, this runs the block's teardown and destroys the cloud resource. + +3. **Create the new meshTenant with no `platform_tenant_id`.** Setting it is what makes meshStack treat + the adoption as an import. Left unset, `spec.localId` stays null, the guard returns early, and + meshStack creates the mandatory building block itself. -The last class is the dangerous one, because the damage arrives later, from a plan someone else runs. +4. **Seed the new block's state before its runner starts.** Find the new block by diffing the uuid list + of the mandatory BBD's blocks against the list taken before step 3, then push the seed: + + ```sh + # block_uuids: a JSON array of the uuids under meshbuildingblocks?definitionUuid=$BBD_UUID + before=$(block_uuids) # taken before step 3 + new=$(block_uuids | jq -r --argjson b "$before" '. - $b | .[0] // empty') + curl -sS -X POST -H "Authorization: Bearer $MT" --data-binary @seed.tfstate \ + "$MESHSTACK/api/terraform/state/workspace/$WS/buildingBlock/$new" + ``` + + Poll for `$new` with a `sleep` in the loop. Both known ways of losing the race are lookup bugs, not + timing — see the runbook § *The race*. + +5. **Let the run finish and check what it did.** Expect one in-place update on the cloud resource and no + create of the resource itself. Creates of role assignments are normal, and getting that set wrong in + either direction fails the run — see the runbook § *Access parity*. Read the new tenant's + `platform_tenant_id` back: a run that fails before its output-collection step leaves it unset, which + breaks every downstream consumer of that attribute. + +--- ## Risk method -**Experiment on throwaway resources, but snapshot even then.** A throwaway proves the mechanism; the -snapshot is what lets you retry after the first attempt teaches you something. Concretely: +These four habits are what keep the procedure recoverable. They generalise past this migration. + +**Experiment on throwaway resources, and snapshot even then.** The throwaway proves the mechanism; the +snapshot is what allows a retry after the first attempt teaches something. Before mutating an object +through an API, `GET` it and keep the response — that file is the rollback. Before deleting an +attribute, record its value in the comment above the destructive command, so the restore command is +written down next to it. + +**Assert on the plan, not on the intention.** Refuse to emit a seed or continue a run unless the plan +shows no destroys and no replacements. Put the check in the script, not in the operator's head: + +```sh +tofu plan -out=tfplan +tofu show -json tfplan \ + | jq -e '[.resource_changes[]?.change.actions[]] | index("delete") == null' > /dev/null \ + || { echo "plan destroys or replaces a resource — refusing to proceed"; exit 1; } +``` + +A replacement appears as `["delete","create"]` or `["create","delete"]`, so a single check for `delete` +covers both. Then pass the expected cloud resource id into every script and abort when the resource +found does not match it. -- Before mutating an object through an API, `GET` it and keep the response. That file is your rollback. -- Before deleting an attribute, record its value in the command's own comment, so the restore command is - written down next to the destructive one. -- Prefer an operation the provider models as an in-place update over one it models as a replacement, and - confirm which it is by planning rather than by reading documentation. +**Prefer an operation the provider models as an in-place update** over one it models as a replacement, +and confirm which it is by planning rather than by reading documentation. -**Weigh the blast radius, not the action.** Deleting a landing zone reads as more destructive than -editing a platform's availability, but the landing zone delete is a reversible deactivation while the -availability change is guarded by rules that can leave the platform in a state you cannot return from. -Check what each operation actually does before ranking them. +**Weigh the blast radius, not the wording of the action.** Deleting a landing zone reads as more +destructive than editing a platform's availability, but the landing zone delete is a reversible +deactivation, while the availability change is guarded by rules that can leave the platform in a state +it cannot return from. Check what each operation does before ranking them. + +--- ## Retiring the old platform @@ -84,17 +154,34 @@ Do this only once no tenants remain on it. **Deactivating its landing zones is the effective retirement.** A meshTenant cannot exist without a landing zone, so a deactivated landing zone makes the platform unorderable whatever its availability -says. `DELETE /api/meshobjects/meshlandingzones/` is a disable, not a removal: the object -stays readable with `lifecycle.state = DEACTIVATED` and existing tenants are untouched. +says. `DELETE /api/meshobjects/meshlandingzones/` is a disable rather than a removal: the +object stays readable with `lifecycle.state = DEACTIVATED` and existing tenants are untouched. The +identifier is global, not scoped per platform, so verify uniqueness before deleting by bare identifier. **Do not expect to unpublish the platform.** Once a platform has been published, `UNPUBLISHED` is -unreachable — see the cookbook. And deleting the platform burns its identifier permanently even though -the delete is soft, so keep the row unless you are certain the identifier is never wanted again. +unreachable — the cookbook lists the three `400` guards that establish this. Deleting the platform +permanently consumes its identifier even though the delete is soft, so keep the row unless the +identifier is certainly never wanted again. + +--- ## Telling live code from dead code A `kit/`-style directory that no repository references may still be live, because a building block definition reaches it by Git repository path rather than by a module source. Grepping the repository -cannot answer the question. Ask the instance instead, and check **versions**, not definitions: a +cannot answer the question. Ask the instance instead, and read **versions**, not definitions: a definition's source moves over its lifetime, so the current version may point somewhere entirely -different from the version an old block is pinned to. See the cookbook for the two endpoints. +different from the version an old block is pinned to. The cookbook § *Auditing where building block code +actually lives* has the two endpoints. + +--- + +## Key references + +| Topic | Reference | +|---|---| +| Per-tenant procedure, seed construction, traps, recovery, IaC reconciliation | `.agents/references/tenant-migration-runbook.md` | +| Reaching and rewriting a running block's Terraform state; the adopt harness | `.agents/references/building-block-state-doctoring.md` | +| Endpoints, media types, validation rules, traps that read as empty results | `.agents/references/meshstack-api-cookbook.md` | +| Worked migration of nine tenants, including both failures | `.agents/references/tenant-migration-case-study.md` | +| STACKIT backplane identity | `.agents/references/stackit-backplane.md` | diff --git a/AGENTS.md b/AGENTS.md index 5c36f59d..8888ac5e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -386,6 +386,19 @@ See [.agents/skills/e2e-test/SKILL.md](.agents/skills/e2e-test/SKILL.md) (the `e --- +## Tenant Migration + +Moving meshTenants between meshPlatforms — for example off a hand-built custom platform onto one deployed +by a reference architecture — cannot use meshStack's import path when the landing zone carries a mandatory +building block definition. The tenants must instead adopt their existing cloud resources through the +building block's Terraform state. + +See [.agents/skills/tenant-migration/SKILL.md](.agents/skills/tenant-migration/SKILL.md) (the +`tenant-migration` skill) for the workflow, the tenant classes, the per-tenant runbook, building block +state doctoring and the meshStack API cookbook that the migration relies on. + +--- + ## Checklist for New Modules - [ ] `backplane/` (optional) and `buildingblock/` with all required files From b39b9b12b7547a6cb00a288c202ee854418776de Mon Sep 17 00:00:00 2001 From: Andreas Grub Date: Wed, 19 Aug 2026 11:38:36 +0200 Subject: [PATCH 3/4] docs(agents): correct the field holding a definition version's source The cookbook told readers to look at `spec.source.terraform.*` on a building block definition version. The field is `spec.implementation.terraform.*`, and the difference is not benign: `spec.source` returns null, which reads as "this definition has no source" rather than as a wrong query. Found by re-running the audit against a different set of definitions, where every source came back empty until the field name was corrected. Co-Authored-By: Claude Opus 5 (1M context) --- .agents/references/meshstack-api-cookbook.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.agents/references/meshstack-api-cookbook.md b/.agents/references/meshstack-api-cookbook.md index c4e508e9..7062aad1 100644 --- a/.agents/references/meshstack-api-cookbook.md +++ b/.agents/references/meshstack-api-cookbook.md @@ -127,9 +127,13 @@ it by Git repository path. To answer the question properly, read the sources off # per definition GET /api/meshobjects/meshbuildingblockdefinitionversions?buildingBlockDefinitionUuid= # then read each version's -.spec.source.terraform.repositoryUrl and .spec.source.terraform.repositoryPath +.spec.implementation.terraform.repositoryUrl and .spec.implementation.terraform.repositoryPath ``` +The field is `spec.implementation`, not `spec.source`. Write the fallback +`(.spec.implementation // .spec.source)` if you want the query to survive either shape, because a plain +`.spec.source` returns null and reads as "this definition has no source". + Two things to get right: - **Check versions, not definitions.** A definition's source moves over its lifetime. One observed From 93e7378c5edf12a9c65cab925af5146d94f8d560 Mon Sep 17 00:00:00 2001 From: Andreas Grub Date: Mon, 24 Aug 2026 20:48:43 +0200 Subject: [PATCH 4/4] docs(agents): add the scripts that ran the STACKIT tenant migration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill described the migration in prose; the tooling that actually performed it lived only in a scratch folder outside any repo. It moves here, next to the runbook it implements, before that folder is deleted. Seven tenants were migrated with these scripts. `adopt-seed.sh` builds the seed state from the live cloud resource, `migrate.sh` runs one tenant end to end, and `adopt/` is the plan harness — a copy of the STACKIT project starterkit. Every instance-specific value now comes from a `MIG_*` environment variable with no default, so nothing here names an instance, a container id or a service account. The originals hardcoded them, which is fine in a scratch folder and not in a public repo. A wrong value migrates the wrong tenants, so a missing variable aborts rather than falling back. Otherwise the scripts are unchanged: `reseed.jq` and `adopt/` are byte-identical to what ran the migrations. Co-Authored-By: Claude Opus 5 (1M context) --- .agents/skills/tenant-migration/SKILL.md | 13 ++ .../skills/tenant-migration/scripts/README.md | 126 ++++++++++++++++++ .../tenant-migration/scripts/adopt-seed.sh | 109 +++++++++++++++ .../scripts/adopt/SUMMARY.md.tftpl | 11 ++ .../tenant-migration/scripts/adopt/main.tf | 44 ++++++ .../tenant-migration/scripts/adopt/outputs.tf | 30 +++++ .../scripts/adopt/provider.tf | 10 ++ .../scripts/adopt/variables.tf | 61 +++++++++ .../scripts/adopt/versions.tf | 13 ++ .../skills/tenant-migration/scripts/env.sh | 49 +++++++ .../tenant-migration/scripts/migrate.sh | 76 +++++++++++ .../skills/tenant-migration/scripts/reseed.jq | 26 ++++ 12 files changed, 568 insertions(+) create mode 100644 .agents/skills/tenant-migration/scripts/README.md create mode 100755 .agents/skills/tenant-migration/scripts/adopt-seed.sh create mode 100644 .agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl create mode 100644 .agents/skills/tenant-migration/scripts/adopt/main.tf create mode 100644 .agents/skills/tenant-migration/scripts/adopt/outputs.tf create mode 100644 .agents/skills/tenant-migration/scripts/adopt/provider.tf create mode 100644 .agents/skills/tenant-migration/scripts/adopt/variables.tf create mode 100644 .agents/skills/tenant-migration/scripts/adopt/versions.tf create mode 100644 .agents/skills/tenant-migration/scripts/env.sh create mode 100755 .agents/skills/tenant-migration/scripts/migrate.sh create mode 100644 .agents/skills/tenant-migration/scripts/reseed.jq diff --git a/.agents/skills/tenant-migration/SKILL.md b/.agents/skills/tenant-migration/SKILL.md index ac06e975..c145afa4 100644 --- a/.agents/skills/tenant-migration/SKILL.md +++ b/.agents/skills/tenant-migration/SKILL.md @@ -36,6 +36,19 @@ The block adopts the resource instead of creating a replacement. --- +## The scripts that did this once + +`scripts/` holds the tooling that migrated seven STACKIT tenants: `adopt-seed.sh` builds the seed state +from the live cloud resource, `migrate.sh` runs one tenant end to end, and `adopt/` is the plan harness. +Read [`scripts/README.md`](scripts/README.md) before using them. + +They are a worked example rather than a general tool — the migration logic is portable, the platform and +role details are STACKIT-specific. Every instance-specific value comes from a `MIG_*` environment +variable with no default. For a different platform, keep the shape and replace the harness and +`role_mapping`. + +--- + ## Workflow 1. **Inventory the tenants and classify each one.** The class decides the procedure — see *Classify the diff --git a/.agents/skills/tenant-migration/scripts/README.md b/.agents/skills/tenant-migration/scripts/README.md new file mode 100644 index 00000000..352c2e1d --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/README.md @@ -0,0 +1,126 @@ +# Tenant migration tooling + +The scripts that ran the STACKIT migration this skill is written from: they move a meshTenant from one +meshPlatform to another **without destroying the cloud resource behind it**. Seven tenants were migrated +with them, off `stackit.sovereign` onto `likvid-stackit.global`. + +Read [the runbook](../../../references/tenant-migration-runbook.md) first. It carries the reasoning; +this file only covers how to run the scripts. + +**They are a worked example, not a general tool.** The migration logic is portable, but the platform +and role details are STACKIT-specific: the `adopt/` harness is a copy of the STACKIT project starterkit, +and `role_mapping` maps meshStack project roles onto STACKIT `owner` / `editor` / `reader`. For another +platform, keep the shape and replace those parts. + +| File | Purpose | +|---|---| +| `env.sh` | Credentials and API constants. Source it, do not execute it. | +| `adopt-seed.sh` | Builds the seed state from the **live** cloud resource. Preferred. | +| `migrate.sh` | Migrates one tenant end to end. | +| `reseed.jq` | Rewrites an old block's Terraform state into the new module's resource shape. Superseded by `adopt-seed.sh`. | +| `adopt/` | Standalone plan harness — a byte-identical copy of the hub module the runner executes. | + +## Configure the instance first + +Nothing is hardcoded to an instance, and nothing has a default — the wrong value here migrates the +wrong tenants: + +```sh +export MIG_IAC_REPO=~/git/likvid-bank/likvid-cloudfoundation # its setup-env.sh loads the API key +export MIG_MESH_URL=https://federation..meshcloud.io +export MIG_CLIENT_ID= +export MIG_PARENT_CONTAINER_ID= +export MIG_SERVICE_ACCOUNT_EMAIL= +export MIG_NEW_BBD_UUID= +``` + +`env.sh` derives the SSO host from `MIG_MESH_URL` by swapping `federation.` for `sso.`; set +`MIG_SSO_URL` if your instance does not follow that pattern. + +## Running one migration + +Two steps. Build the seed, then migrate: + +```sh +source scripts/env.sh +scripts/adopt-seed.sh +scripts/migrate.sh - +``` + +`adopt-seed.sh` imports the live project and its adoptable role assignments, prints the plan, and writes +`/tmp/mig-seed.json`. It aborts unless the plan adds and destroys nothing on the project itself — +creates of role assignments are expected and fine. `migrate.sh` sources `env.sh` itself; `-` as its +third argument means "the seed is already prepared". + +This works whether or not the tenant has an old building block. When it has one, purge it first and +confirm the purge before letting `migrate.sh` delete the tenant: + +```sh +curl -X DELETE -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks//purge" +``` + +The older single-step form still works and uses `reseed.jq` instead, purging the old block itself: + +```sh +scripts/migrate.sh +``` + +Prefer `adopt-seed.sh`. It reads reality instead of trusting the old state, and it catches the two +things that fail a run: a pre-existing role assignment missing from the seed, and a project label the +module wants to clear. Both are in the runbook, under *Access parity* and *Labels can block the +adoption outright*. + +The project-id argument is a safety interlock, not a convenience: both scripts abort if what they find +holds a different project id than the one given. + +## Check the labels first + +```sh +stackit project describe -o json | jq .labels +``` + +Anything non-empty fails the run, because the module plans `labels -> null` and STACKIT does not clear +them. Remove them with an explicit null — `stackit project update --label` can only add, and the +`DELETE .../labels` endpoint refuses protected keys such as `billingReference` with a 409: + +```sh +curl -X PATCH -H "Authorization: Bearer $(stackit auth get-access-token)" \ + -H 'Content-Type: application/json' -d '{"labels":{"":null}}' \ + "https://resource-manager.api.stackit.cloud/v2/projects/" +``` + +This is a provider bug, reported upstream on +[stackitcloud/terraform-provider-stackit#1381](https://github.com/stackitcloud/terraform-provider-stackit/issues/1381) +and still open as of `0.112.0`. `labels` is `Optional` and not `Computed`, so a configuration that omits +it plans `labels -> null`, STACKIT does not clear them, and the apply fails on the inconsistent result. +For a protected label such as `billingReference` there is no workaround through the labels endpoint at +all — only the `PATCH` above. + +## What it does + +1. Reads the old block's state and verifies the project id. +2. Purges the old building block — `DELETE .../purge`, which runs **no destroy** and leaves the + cloud project intact. +3. Deletes the old meshTenant. +4. Creates the new meshTenant with **no** `platform_tenant_id`. Setting it would make meshStack treat + the adoption as an import, which custom platforms reject. +5. Seeds the new block's state before its runner starts, then waits for the run. + +## The race + +Between steps 4 and 5 the runner may start on its own — the window is roughly fifteen seconds. If it +wins, it creates a *duplicate* cloud project. Recovery is known and lossless; see *The race* in the +runbook. The one rule that matters: **move a doomed project to the organization root before deleting +it**, because STACKIT deletions sit in `DELETING` for weeks and block the parent folder. + +The race was never lost across the seven migrations. Every failure was a plan problem instead, and those +are recoverable too — purge the new block, delete the new tenant, fix the seed, re-run. See *Recovery +when the first run fails* in the runbook, including how to read a run's log, which needs one specific +`Accept` header and 406s on everything else. + +## If nothing comes back + +An expired Vault session produces a 401, and `jq` renders that as an empty list — so "no tenants +found" usually means "not authenticated". `env.sh` checks for this and fails loudly. To fix it, run +`source ./setup-env.sh` in a real terminal so the interactive OIDC login can complete. diff --git a/.agents/skills/tenant-migration/scripts/adopt-seed.sh b/.agents/skills/tenant-migration/scripts/adopt-seed.sh new file mode 100755 index 00000000..99d24858 --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt-seed.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# adopt-seed.sh +# +# Builds /tmp/mig-seed.json for a tenant that has no old building block, so `migrate.sh - +# ` has a seed state to push. Imports the live STACKIT project into the `adopt/` harness — +# a byte-identical copy of the hub module the runner executes — then normalises the result. +# +# It also adopts every project role assignment that already exists in STACKIT *and* that the run will +# manage. Replicator-created projects already carry `owner` / `editor` / `reader` grants, and a grant +# the run wants but the seed omits fails the whole run with "found a duplicate role assignment". +# +# The run's grants come from the meshProject's user bindings through `role_mapping`, so the set is +# computed here the same way instead of guessed. Grants outside that set — the legacy `project.*` and +# `ufw.*` roles, and `owner` / `editor` / `reader` given to somebody who is not a meshProject member — +# are deliberately left out, so the run never manages them and never removes them. +# +# It plans before it writes the seed and aborts if anything is replaced or destroyed. Either would cost +# the live STACKIT project or somebody's access. Creates are expected: they are the grants the +# meshProject calls for that STACKIT does not have yet. +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WS=$1; PROJ=$2; CONTAINER=$3; PID=$4 + +# Where new projects go and who the module manages them as. Both are instance-specific, so they come +# from the environment — a wrong parent container id creates the project in the wrong folder. +: "${MIG_PARENT_CONTAINER_ID:?set MIG_PARENT_CONTAINER_ID to the landing-zone folder container id}" +: "${MIG_SERVICE_ACCOUNT_EMAIL:?set MIG_SERVICE_ACCOUNT_EMAIL to the platform service account}" + +# The runner executes OpenTofu 1.11.0, and OpenTofu refuses to read a state written by a newer +# version. The local binary is newer, so its version has to be rewritten out of the seed. +RUNNER_TOFU_VERSION=1.11.0 + +# The block's own role_mapping, and the meshStack role display names that produce its keys. Confirmed +# against the `users` input of a real run: Project Admin -> admin, Project User -> user, +# Project Reader -> reader. +ROLE_MAPPING='{"admin":["owner"],"reader":["reader"],"user":["editor"]}' +MESH_ROLES='{"Project Admin":"admin","Project User":"user","Project Reader":"reader"}' + +cd "$HERE/adopt" +unset STACKIT_SERVICE_ACCOUNT_KEY_PATH +export STACKIT_SERVICE_ACCOUNT_KEY="$STACKIT_ORG_SERVICE_ACCOUNT_KEY" + +# The meshProject's user bindings, which is what meshStack turns into the block's `users` input. +curl -sf -H "Authorization: Bearer $MT" \ + -H "Accept: application/vnd.meshcloud.api.meshprojectuserbinding.v3.hal+json" \ + "$MESH/api/meshobjects/meshprojectbindings/userbindings?projectIdentifier=$PROJ&workspaceIdentifier=$WS" \ + > /tmp/adopt-bindings.json +skipped=$(jq -r --argjson r "$MESH_ROLES" \ + '[._embedded.meshProjectUserBindings[]?.roleRef.name | select($r[.] == null)] | unique | join(", ")' \ + /tmp/adopt-bindings.json) +[ -n "$skipped" ] && echo " NOTE: meshStack roles with no STACKIT mapping, ignored: $skipped" + +jq -n --arg name "$PROJ" --argjson mapping "$ROLE_MAPPING" --argjson meshroles "$MESH_ROLES" \ + --arg parent "$MIG_PARENT_CONTAINER_ID" --arg sa "$MIG_SERVICE_ACCOUNT_EMAIL" \ + --slurpfile b /tmp/adopt-bindings.json '{ + parent_container_id: $parent, + project_name: $name, + service_account_email: $sa, + labels: {}, + role_mapping: $mapping, + users: ($b[0]._embedded.meshProjectUserBindings // [] + | map({subject: .subject.name, role: $meshroles[.roleRef.name]}) | map(select(.role != null)) + | group_by(.subject) | map({ + meshIdentifier: .[0].subject, username: .[0].subject, firstName: "-", lastName: "-", + email: .[0].subject, euid: .[0].subject, roles: ([.[].role] | unique) + })) +}' > /tmp/adopt.tfvars.json +echo " meshProject members: $(jq -r '[.users[].email]|join(" ")' /tmp/adopt.tfvars.json)" + +TFARGS=(-no-color -var-file=/tmp/adopt.tfvars.json -var "stackit_service_account_key=$STACKIT_SERVICE_ACCOUNT_KEY") + +rm -f terraform.tfstate terraform.tfstate.backup +tofu import "${TFARGS[@]}" stackit_resourcemanager_project.project "$CONTAINER" >/dev/null +got=$(jq -r '.resources[]|select(.type=="stackit_resourcemanager_project")|.instances[0].attributes.project_id' terraform.tfstate) +[ "$got" = "$PID" ] || { echo "ABORT: imported $got, expected $PID"; exit 1; } + +# Grants that already exist in STACKIT and that the run will manage. Everything else stays unmanaged. +stackit project member list --project-id "$PID" -o json > /tmp/adopt-members.json +jq -r --argjson mapping "$ROLE_MAPPING" --slurpfile v /tmp/adopt.tfvars.json ' + ([ $v[0].users[] | .email as $s | .roles[] | $mapping[.][] | "\($s):\(.)" ] | unique) as $wanted + | [ .[] | "\(.subject):\(.role)" ] | unique | map(select(IN($wanted[]))) | .[]' \ + /tmp/adopt-members.json > /tmp/adopt-adopt-keys.txt + +# The role assignment's import id is ",,". +while IFS= read -r key; do + [ -n "$key" ] || continue + tofu import "${TFARGS[@]}" \ + "stackit_authorization_project_role_assignment.role_assignments[\"$key\"]" \ + "$PID,${key##*:},${key%:*}" >/dev/null + echo " adopted $key" +done < /tmp/adopt-adopt-keys.txt + +tofu plan "${TFARGS[@]}" > /tmp/adopt-plan.txt 2>&1 +if grep -qE 'must be replaced|forces replacement' /tmp/adopt-plan.txt; then + echo "ABORT: plan replaces a resource. See /tmp/adopt-plan.txt"; exit 1 +fi +if grep -q '^No changes\.' /tmp/adopt-plan.txt; then + echo " plan: no changes" +elif grep -qE '^Plan: [0-9]+ to add, [0-9]+ to change, 0 to destroy\.$' /tmp/adopt-plan.txt; then + echo " plan: $(grep -E '^Plan: ' /tmp/adopt-plan.txt)" + grep -E '^ # ' /tmp/adopt-plan.txt | sed 's/^/ /' +else + echo "ABORT: plan destroys something. See /tmp/adopt-plan.txt"; tail -30 /tmp/adopt-plan.txt; exit 1 +fi + +jq --arg v "$RUNNER_TOFU_VERSION" \ + '.terraform_version = $v | .serial = 1 | .outputs = {} | .resources |= map(select((.instances|length) > 0))' \ + terraform.tfstate > /tmp/mig-seed.json +echo " seed: $(jq -r '[.resources[]|"\(.name)(\(.instances|length))"]|join(" ")' /tmp/mig-seed.json), tofu $(jq -r .terraform_version /tmp/mig-seed.json)" diff --git a/.agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl b/.agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl new file mode 100644 index 00000000..ca939c5b --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/SUMMARY.md.tftpl @@ -0,0 +1,11 @@ +# Project: **${project_name}** + +## Details + +| Property | Value | +|----------|-------| +| **Project ID** | `${project_id}` | +| **Container ID** | `${container_id}` | +| **Portal** | [Open in STACKIT Portal](${project_url}) | + +${membership_summary} diff --git a/.agents/skills/tenant-migration/scripts/adopt/main.tf b/.agents/skills/tenant-migration/scripts/adopt/main.tf new file mode 100644 index 00000000..de99b36a --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/main.tf @@ -0,0 +1,44 @@ +locals { + # Determine the parent container ID based on environment + selected_parent_container_id = var.environment != null ? lookup(var.parent_container_ids, var.environment, var.parent_container_id) : var.parent_container_id + + users_with_stackit_roles = [ + for user in var.users : { + email = user.email + roles = distinct(flatten([ + for meshstack_role in user.roles : lookup(var.role_mapping, meshstack_role, []) + ])) + } + ] + + user_role_assignments = { + for assignment in flatten([ + for user in local.users_with_stackit_roles : [ + for stackit_role in user.roles : { + key = "${user.email}:${stackit_role}" + subject = user.email + stackit_role = stackit_role + } + ] + ]) : assignment.key => assignment + } +} + +resource "stackit_resourcemanager_project" "project" { + parent_container_id = local.selected_parent_container_id + name = var.project_name + owner_email = var.service_account_email + + # Only set labels if there are actually labels to set + labels = length(var.labels) > 0 ? var.labels : null +} + +# User role assignments (experimental IAM feature) +resource "stackit_authorization_project_role_assignment" "role_assignments" { + for_each = local.user_role_assignments + + resource_id = stackit_resourcemanager_project.project.project_id + role = each.value.stackit_role + subject = each.value.subject +} + diff --git a/.agents/skills/tenant-migration/scripts/adopt/outputs.tf b/.agents/skills/tenant-migration/scripts/adopt/outputs.tf new file mode 100644 index 00000000..213ba0c5 --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/outputs.tf @@ -0,0 +1,30 @@ +output "project_id" { + value = stackit_resourcemanager_project.project.project_id + description = "The UUID of the created StackIt project." +} + +output "container_id" { + value = stackit_resourcemanager_project.project.container_id + description = "The user-friendly container ID of the created StackIt project." +} + +output "project_name" { + value = stackit_resourcemanager_project.project.name + description = "The name of the created StackIt project." +} + +output "project_url" { + value = "https://portal.stackit.cloud/projects/${stackit_resourcemanager_project.project.project_id}" + description = "The deep link URL to access the project in the StackIt portal." +} + +output "summary" { + description = "Summary of the created project and STACKIT organization membership onboarding for assigned project users." + value = templatefile("${path.module}/SUMMARY.md.tftpl", { + project_name = stackit_resourcemanager_project.project.name + project_id = stackit_resourcemanager_project.project.project_id + container_id = stackit_resourcemanager_project.project.container_id + project_url = "https://portal.stackit.cloud/projects/${stackit_resourcemanager_project.project.project_id}" + membership_summary = fileexists("${path.module}/stackit_organization_membership_summary.md") ? file("${path.module}/stackit_organization_membership_summary.md") : "STACKIT organization membership summary was not generated." + }) +} diff --git a/.agents/skills/tenant-migration/scripts/adopt/provider.tf b/.agents/skills/tenant-migration/scripts/adopt/provider.tf new file mode 100644 index 00000000..20c1ed6f --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/provider.tf @@ -0,0 +1,10 @@ +# local-only: the real building block authenticates via WIF; here we use the org service account key +provider "stackit" { + service_account_key = var.stackit_service_account_key + experiments = ["iam"] +} + +variable "stackit_service_account_key" { + type = string + sensitive = true +} diff --git a/.agents/skills/tenant-migration/scripts/adopt/variables.tf b/.agents/skills/tenant-migration/scripts/adopt/variables.tf new file mode 100644 index 00000000..b6cab1ee --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/variables.tf @@ -0,0 +1,61 @@ +variable "parent_container_id" { + type = string + nullable = false + description = "The parent container ID (organization or folder) where the project will be created." +} + +variable "environment" { + type = string + default = null + description = "The environment type (production, staging, development). If not set, uses parent_container_id directly." +} + +variable "parent_container_ids" { + type = object({ + production = optional(string) + staging = optional(string) + development = optional(string) + }) + default = {} + description = "Parent container IDs for different environments. If environment is set, the corresponding container ID will be used." +} + +variable "project_name" { + type = string + nullable = false + description = "The name of the StackIt project to create." +} + +variable "service_account_email" { + type = string + nullable = false + description = "Email of the STACKIT service account for WIF-based authentication and project ownership." +} + +variable "labels" { + type = map(string) + nullable = false + description = "Labels to apply to the project. Includes the `networkArea` label when the building block definition is wired to a network area." +} + +variable "users" { + description = "List of users from the authoritative system. Each user's `roles` are meshStack roles that are mapped to STACKIT project roles via `role_mapping`." + type = list(object({ + meshIdentifier = string + username = string + firstName = string + lastName = string + email = string + euid = string + roles = list(string) + })) + nullable = false +} + +variable "role_mapping" { + type = map(list(string)) + description = "Maps meshStack roles from `users[*].roles` to STACKIT project roles. Values can be built-in STACKIT roles or custom STACKIT role names. Unknown meshStack roles are ignored." + + nullable = false +} + diff --git a/.agents/skills/tenant-migration/scripts/adopt/versions.tf b/.agents/skills/tenant-migration/scripts/adopt/versions.tf new file mode 100644 index 00000000..364654db --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/adopt/versions.tf @@ -0,0 +1,13 @@ +terraform { + required_version = ">= 1.11.0" + required_providers { + stackit = { + source = "stackitcloud/stackit" + version = ">= 0.98.0" + } + meshstack = { + source = "meshcloud/meshstack" + version = ">= 0.21.0" + } + } +} diff --git a/.agents/skills/tenant-migration/scripts/env.sh b/.agents/skills/tenant-migration/scripts/env.sh new file mode 100644 index 00000000..b8282d6b --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/env.sh @@ -0,0 +1,49 @@ +# Credentials and API constants for the tenant migration. Source it, do not execute it. +# +# source scripts/env.sh || return 1 +# +# Set these three for your instance before sourcing. There are no defaults on purpose: the wrong +# instance here migrates the wrong tenants. +# +# MIG_IAC_REPO the IaC runtime repo whose setup-env.sh loads the meshStack API key from Vault +# MIG_MESH_URL the meshStack API base url +# MIG_CLIENT_ID the API key's client id, paired with the key that setup-env.sh exports +# +: "${MIG_IAC_REPO:?set MIG_IAC_REPO to the IaC runtime repo, e.g. ~/git//-cloudfoundation}" +: "${MIG_MESH_URL:?set MIG_MESH_URL, e.g. https://federation..meshcloud.io}" +: "${MIG_CLIENT_ID:?set MIG_CLIENT_ID to the meshStack API key client id}" + +# It cds to the IaC repo first on purpose: setup-env.sh only works from the repo root, and agent shells +# reset the working directory between commands, which makes a bare `source ./setup-env.sh` fail silently. +cd "$MIG_IAC_REPO" || return 1 +source ./setup-env.sh >/dev/null 2>&1 + +# setup-env.sh falls back to an interactive `vault login -method=oidc`, which cannot complete in a +# non-interactive shell — it then carries on with no secrets at all. Catch that here, because the +# alternative is a 401 that jq renders as an empty list, i.e. "no tenants exist". +if [ -z "$MESHSTACK_API_KEY_CLOUDFOUNDATION" ]; then + echo "FATAL: vault secrets not loaded. Run 'source ./setup-env.sh' in a terminal to do the OIDC login." >&2 + return 1 +fi + +export MESH="$MIG_MESH_URL" + +# /api/login only issues a redirect; the token has to come from keycloak directly. The sso host mirrors +# the api host: federation..meshcloud.io -> sso..meshcloud.io. +MIG_SSO_URL=${MIG_SSO_URL:-$(printf '%s' "$MESH" | sed 's#//federation\.#//sso.#')} +export MT=$(curl -s -u "${MIG_CLIENT_ID}:${MESHSTACK_API_KEY_CLOUDFOUNDATION}" \ + -d 'grant_type=client_credentials' \ + "${MIG_SSO_URL}/auth/realms/meshfed/protocol/openid-connect/token" \ + | jq -r '.access_token // empty') +[ -z "$MT" ] && { echo "FATAL: token request failed" >&2; return 1; } + +# Every meshObject type needs its own versioned Accept header. Plain v4 (no -preview) returns 406. +export BBACC="Accept: application/vnd.meshcloud.api.meshbuildingblock.v2-preview.hal+json" +export TN3="Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json" +export TN4="Accept: application/vnd.meshcloud.api.meshtenant.v4-preview.hal+json" +export PL2="Accept: application/vnd.meshcloud.api.meshplatform.v2.hal+json" + +# ~/.terraformrc may carry a dev override with an older meshstack provider than the one that wrote the +# state, which makes tofu refuse to read it. +[ -f /tmp/tofu-no-override.tfrc ] || printf 'provider_installation {\n direct {}\n}\n' > /tmp/tofu-no-override.tfrc +export TF_CLI_CONFIG_FILE=/tmp/tofu-no-override.tfrc diff --git a/.agents/skills/tenant-migration/scripts/migrate.sh b/.agents/skills/tenant-migration/scripts/migrate.sh new file mode 100755 index 00000000..81f9bfbf --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/migrate.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# migrate.sh +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$HERE/env.sh" || exit 1 +set -u +WS=$1; PROJ=$2; OLDBB=$3; PID=$4 + +# The building block definition the new tenant's landing zone makes mandatory. Instance-specific. +: "${MIG_NEW_BBD_UUID:?set MIG_NEW_BBD_UUID to the building block definition of the target landing zone}" +NEWDEF=$MIG_NEW_BBD_UUID +say(){ echo " $*"; } + +echo "=== $WS/$PROJ -> likvid-stackit.global ===" + +# 1. capture and disarm the old block +if [ "$OLDBB" != "-" ]; then + curl -s -H "Authorization: Bearer $MT" \ + "$MESH/api/terraform/state/workspace/$WS/buildingBlock/$OLDBB" > /tmp/mig-old.json + got=$(jq -r '.resources[]|select(.type=="stackit_resourcemanager_project")|.instances[0].attributes.project_id' /tmp/mig-old.json) + [ "$got" = "$PID" ] || { echo "ABORT: state holds $got, expected $PID"; exit 1; } + jq -f "$HERE/reseed.jq" /tmp/mig-old.json > /tmp/mig-seed.json + say "seed state: $(jq -r '[.resources[]|"\(.name)(\(.instances|length))"]|join(" ")' /tmp/mig-seed.json)" + say "purge old block -> $(curl -s -X DELETE -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks/$OLDBB/purge")" + sleep 8 +else + [ -s /tmp/mig-seed.json ] || { echo "ABORT: no /tmp/mig-seed.json prepared"; exit 1; } + say "using externally prepared seed state" +fi + +# 2. delete the old tenant +say "delete old tenant -> $(curl -s -X DELETE -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $MT" \ + -H "Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json" \ + "$MESH/api/meshobjects/meshtenants/$WS.$PROJ.stackit.sovereign")" +sleep 10 + +# 3. create the new tenant and seed its block's state before the runner starts +before=$(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks?definitionUuid=$NEWDEF&size=200" \ + | jq -r '._embedded.meshBuildingBlocks[]?.metadata.uuid' | sort | tr '\n' ' ') +printf '{"apiVersion":"v3","kind":"meshTenant","metadata":{"ownedByWorkspace":"%s","ownedByProject":"%s","platformIdentifier":"likvid-stackit.global"},"spec":{"landingZoneIdentifier":"likvid-stackit-default"}}' \ + "$WS" "$PROJ" > /tmp/mig-ten.json +code=$(curl -s -o /tmp/mig-ten-out.json -w '%{http_code}' -X POST -H "Authorization: Bearer $MT" \ + -H "Content-Type: application/vnd.meshcloud.api.meshtenant.v3.hal+json" \ + -H "Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json" \ + --data-binary @/tmp/mig-ten.json "$MESH/api/meshobjects/meshtenants") +say "create new tenant -> $code" +[ "$code" = 201 ] || { head -c 300 /tmp/mig-ten-out.json; exit 1; } + +BB="" +for i in $(seq 1 60); do + sleep 0.5 + BB=$(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" \ + "$MESH/api/meshobjects/meshbuildingblocks?definitionUuid=$NEWDEF&size=200" \ + | jq -r --argjson b "$(printf '%s' "$before" | jq -R 'split(" ")|map(select(length>0))')" \ + '[._embedded.meshBuildingBlocks[]?.metadata.uuid] - $b | .[0] // empty') + [ -n "$BB" ] && break +done +[ -n "$BB" ] || { echo "ABORT: new block not found"; exit 1; } +say "seed state into $BB -> $(curl -s -o /dev/null -w '%{http_code}' -X POST -H "Authorization: Bearer $MT" \ + -H "Content-Type: application/json" --data-binary @/tmp/mig-seed.json \ + "$MESH/api/terraform/state/workspace/$WS/buildingBlock/$BB")" +echo "$BB" > /tmp/mig-bb + +# 4. wait, then verify +for i in $(seq 1 40); do + s=$(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" "$MESH/api/meshobjects/meshbuildingblocks/$BB" | jq -r '.status.status') + case "$s" in SUCCEEDED|FAILED) break;; esac + sleep 15 +done +say "run -> $s" +say "block project_id = $(curl -s -H "Authorization: Bearer $MT" -H "$BBACC" "$MESH/api/meshobjects/meshbuildingblocks/$BB" | jq -r '.status.outputs.project_id.value')" +say "tenant localId = $(curl -s -H "Authorization: Bearer $MT" -H 'Accept: application/vnd.meshcloud.api.meshtenant.v3.hal+json' \ + "$MESH/api/meshobjects/meshtenants/$WS.$PROJ.likvid-stackit.global" | jq -r '.spec.localId')" +say "expected = $PID" +stackit project describe "$PID" 2>&1 | grep -E 'NAME|STATE|PARENT' | sed 's/^/ /' diff --git a/.agents/skills/tenant-migration/scripts/reseed.jq b/.agents/skills/tenant-migration/scripts/reseed.jq new file mode 100644 index 00000000..df82e3e1 --- /dev/null +++ b/.agents/skills/tenant-migration/scripts/reseed.jq @@ -0,0 +1,26 @@ +# Rewrite an old STACKIT Project block state into the new module's shape. +# Keeps the project resource untouched; merges every role-assignment resource +# into the new module's single `role_assignments` map, re-keyed ":". +def assignments: + [ .resources[] | select(.type == "stackit_authorization_project_role_assignment") ]; +{ + version: .version, + terraform_version: .terraform_version, + serial: 1, + lineage: .lineage, + outputs: {}, + check_results: null, + resources: ( + [ .resources[] | select(.type == "stackit_resourcemanager_project") ] + + (if (assignments | length) == 0 then [] else + [ { mode: "managed", + type: "stackit_authorization_project_role_assignment", + name: "role_assignments", + provider: (assignments[0].provider), + each: "map", + instances: [ assignments[].instances[] + | .index_key = (.attributes.subject + ":" + .attributes.role) ] + } ] + end) + ) +}