diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5dab1d8d..95f4bc12 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,7 +33,7 @@ A small number of files are scaffolded once and then owned by you — those carr no such header, and the generator will not touch them again. The [generated boundary](docs/) documentation is the full list. -Two consequences worth knowing before opening a pull request: +Three consequences worth knowing before opening a pull request: - **Emitted output must be deterministic.** No timestamps, no tool version, no absolute paths, and no reliance on Go map iteration order in anything written @@ -41,8 +41,15 @@ Two consequences worth knowing before opening a pull request: nothing, which destroys its usefulness. - **Never commit a real credential to probe evidence.** Cassettes are committed by design. Redaction allowlists header values rather than denylisting them, and - a final pass fails the run outright if any known secret survives into the - output. Run `tfpluginframeworkgen probe redact --check` before pushing. + a final pass fails the recording outright — exit `7`, nothing written — if any + credential-shaped value survives into the output. There is nothing extra to run + before pushing; the check is part of every record run, and + `probe -mode verify` re-derives committed evidence offline. +- **Regenerate with the postcheck on.** `emit` finishes by compiling the output, + regenerating its registry docs and running `terraform fmt` + (`-skip-postcheck` exists for tight inner loops only). A PR that arrives with + stale `docs/` or unformatted fixtures is a PR whose final `emit` was skipped — + CI will say so, one gate later. diff --git a/README.md b/README.md index 4cf33c17..92fb5cd5 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,16 @@ A toolkit that programmatically generates [terraform-plugin-framework][framework] providers from an API specification **plus recorded API behaviour**, so that -state mapping is compile-checked generated code instead of runtime reflection. +state mapping is compile-checked generated code instead of runtime reflection — +and so that the first acceptance run confirms what the evidence already proved, +instead of discovering what it missed. -> **Status: Phase 6 of 7.** The pipeline works end to end: `ingest` infers a -> blueprint from a pinned OpenAPI snapshot, `probe` has recorded a live mutating -> run against a real sandbox, `merge` folds the facts in, and the emitted pilot -> compiles, tests and plans. What remains is breadth — more resources, docs, -> and a second API to prove nothing is pilot-shaped. The one unbuilt stage is -> `specs` (fetching a snapshot; committed snapshots verify and load fine). See -> [Roadmap](#roadmap). +> **Status: the pilot is real.** The pipeline runs end to end and has been walked +> across five recorded waves: 23 resource blueprints, 22 resources green against +> the live API in CI, 20 committed evidence snapshots, plus data sources, an +> ephemeral and an action. The full cross-repo chain — OpenAPI spec bump → SDK +> regeneration and release → provider re-generation → live acceptance — has been +> validated once, deliberately. See [Roadmap](#roadmap). ## Why @@ -35,118 +36,122 @@ So this toolkit **pokes the live API and records what it does**. The probe transcripts are committed as evidence, facts are re-derived from them offline in CI, and they feed the generator alongside the specification. -## Pipeline +The pilot taught one more lesson, big enough to reshape the pipeline: it is not +enough to probe fields one behaviour at a time and let acceptance tests find the +rest. Fixing failures after generation is whack-a-mole against the most expensive +oracle available. So the probe now ends by **rehearsing the exact lifecycles the +generated acceptance tests will run** — same values, same bodies, both directions +— before any provider code exists. Acceptance is confirmation, not discovery. + +## Key features + +- **A 16-probe catalogue** against a live sandbox: six read-only protocols and + ten mutating ones, ending with `write.rehearsal` — minimal→maximal and + maximal→minimal lifecycles, per-hop echo comparison, in-bounds contrast values, + and single-culprit bisection of refused bodies. +- **Evidence you can replay.** Every recorded run freezes its transcript, facts, + plan, subject and rehearsal bodies; CI re-derives the facts offline with egress + blocked and fails on any difference. +- **Static facts from the SDK itself.** Behaviour written into the SDK's struct + tags (a zero value the encoding cannot send) is derived by inspection, not + probed, and drift-gated against the pinned SDK version. +- **One fixture derivation, two renderings.** The values the probe rehearses on + the wire are the values the generator renders into `minimal.tf`/`maximal.tf` — + format-aware (`date-time`, `uuid`, `ipv4`, …), with curated hints and omissions + where no derivation can know. +- **Deterministic, committed artefacts with a drift gate on every arrow** — + snapshots, blueprints, plans, evidence, static facts, the generated tree, the + interop export. Byte-identical regeneration is a tested property. +- **Generation finishes with the tools that gate it.** `emit` runs a postcheck + battery — compile, tfplugindocs, `terraform fmt` — so a tree that would fail CI + fails at generation time. +- **A hard generated/hand-written boundary**, enforced five ways, with an escape + hatch that scaffolds once and never regenerates. +- **Safety as structure, not convention**: mutating probes sit behind a + runtime-verified sandbox gate, every create is ledgered before it is issued, + cleanup is per-probe, and cassettes pass a refuse-on-detect secret scan before + anything is written. + +## High-level architecture ``` -OpenAPI snapshot ──ingest──┐ - ├──merge──► blueprint.json ──emit──► provider Go tree -live API ────────probe──────┤ + acceptance tests - │ and fixtures -human overrides ────────────┘ +upstream URL ──specs──► OpenAPI snapshot ──ingest──┐ + ├──merge──► blueprint.json ──emit──► provider Go tree +live API ──probe (catalogue + rehearsal fixpoint)───┤ │ + tests, fixtures, +pinned SDK ──bindings -facts-out (static facts)─────┤ │ examples, docs + │ postcheck +human curation (plans, hints, presence) ────────────┘ (build · docs · fmt) ``` Each arrow writes a **committed, reviewable artefact**. CI regenerates every one of them and fails on drift, then builds and tests the result — because a -generator change can produce a clean diff and broken code. +generator change can produce a clean diff and broken code. The +[gates](docs/gates.md) page lists every job and its local reproduction. A blueprint names SDK symbols as strings, so `tfpluginframeworkgen bindings` type-checks them against the SDK the provider actually pins. That turns a wrong symbol from a pile of identical compile errors in generated code into one message naming the blueprint field to edit. -## Quick start - -Every step below works except step 0, which is a stub: snapshots are pinned by -hand today, and the committed one verifies by checksum on every load. - -```bash -# 0. pin an upstream spec snapshot -tfpluginframeworkgen specs -output-dir openapi-specs/thousandeyes - -# 1. see what the spec offers before committing to anything -tfpluginframeworkgen ingest -only Tags -list - -# 2. infer a blueprint, bound against SDK methods that provably exist -tfpluginframeworkgen ingest -only Tags -out blueprints/thousandeyes +### Repository layout -# 3. probe a sandbox, recording evidence. Every guard is required. -# The token comes from TFPFGEN_PROBE_TOKEN and never from a flag or the profile. -# Plans resolve per resource by convention: blueprints/PROVIDER/KEY.probe.plan.json. -# Scope with -resource, or omit it to record every planned resource in one wave -- -# a resource with no plan file is skipped with a stated note. -tfpluginframeworkgen probe -blueprint blueprints/thousandeyes -resource tag \ - -mode record --allow-mutations \ - -profile .tfpluginframeworkgen/sandbox/thousandeyes.json - -# 3b. re-derive the same facts from the committed transcript, with no network at all -tfpluginframeworkgen probe -blueprint blueprints/thousandeyes -mode verify - -# 4. fold the evidence in; conflicts with the spec are surfaced, never resolved silently -tfpluginframeworkgen merge -blueprint ... -facts ... -strategy annotate - -# 4b. check every SDK symbol the blueprint names actually exists -tfpluginframeworkgen bindings -blueprint blueprints/thousandeyes -module pilot/thousandeyes +| Path | Contents | +|---|---| +| `cmd/tfpluginframeworkgen/` | the one installable binary; stdlib `flag` subcommand dispatch | +| `internal/blueprint/` | the IR, its validation, and the layered-merge engine | +| `internal/ingest/` | OpenAPI → blueprint | +| `internal/specstore/` | pinned specification snapshots: list, read, checksum, pin | +| `internal/interop/` | reads and writes `terraform-plugin-codegen-spec` v0.1 | +| `internal/probe/` | the API behaviour prober | +| `internal/cassette/` | HTTP record/replay, redaction, deterministic canonicalisation | +| `internal/fixturespec/` | the one derivation of fixture values, rendered as HCL and as wire JSON | +| `internal/emit/`, `internal/render/` | blueprint → Go; all logic lives in `render` | +| `internal/sdkbind/` | type-checks blueprint bindings against the pinned SDK; derives static facts | +| `internal/manifest/` | what the last run produced, so orphaned files can be found | +| `internal/templates/` | embedded `.tmpl` files — the emitted shape, as reviewable text | +| `blueprints/` | committed blueprints, plans and static facts, one directory per provider | +| `probe-evidence/` | committed probe cassettes and derived facts | +| `openapi-specs/` | pinned, immutable specification snapshots | +| `interop-specs/` | the committed codegen-spec v0.1 export, drift-gated | +| `pilot/thousandeyes/` | a nested module: a fully generated provider, built, unit-tested and live-tested in CI | +| `docs/` | architecture, guides and the CLI reference — see below | -# 5. emit. Dry run first, always. -tfpluginframeworkgen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes -dry-run -tfpluginframeworkgen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes +## Quick start -# 6. the actual proof -cd pilot/thousandeyes && go build ./... && go test ./... && terraform plan +The loop, end to end (each step's full story is in the +[onboarding runbook](docs/onboarding-a-new-api.md)): -# 7. the CI gate -tfpluginframeworkgen verify -blueprint blueprints/thousandeyes -out pilot/thousandeyes -tfpluginframeworkgen probe -blueprint blueprints/thousandeyes -mode verify # no network +```bash +tfpluginframeworkgen specs -url https://…/api.yaml -output-dir openapi-specs/PROVIDER +tfpluginframeworkgen ingest -only THING -out blueprints/PROVIDER -plan-drafts blueprints/PROVIDER +tfpluginframeworkgen bindings -blueprint blueprints/PROVIDER -module pilot/PROVIDER +tfpluginframeworkgen probe -blueprint blueprints/PROVIDER -resource THING \ + -mode record --allow-mutations -profile .tfpluginframeworkgen/sandbox/PROVIDER.json +tfpluginframeworkgen merge -blueprint blueprints/PROVIDER -facts probe-evidence/…/facts.json +tfpluginframeworkgen emit -blueprint blueprints/PROVIDER -out pilot/PROVIDER ``` +The safe modes are the defaults: `probe` replays committed evidence unless told +to record, mutating runs demand a sandbox profile that proves itself at runtime, +and the token comes from `TFPFGEN_PROBE_TOKEN` — never a flag, never a file. + ## What is generated and what is yours The boundary is the most important thing to understand about a generated -provider. It is enforced four ways: a per-file header, the emission manifest, -`.gitattributes`, and `tfpluginframeworkgen verify`. - -| Path | Owner | Change it by | -|---|---|---| -| `internal/services/**/{resource,model,construct,state,crud,list_resource}.go` | toolkit | editing the blueprint, then `emit` | -| `internal/services/**/{modify_plan,predicate,state_upgrade}.go` | **you** | editing them; scaffolded once, `emit` never touches them again | -| `internal/services/**/{resource_acceptance_test,test_helper_test}.go`, `testdata/minimal.tf` | toolkit | re-probing, then `emit` | -| `internal/services/**/testdata/maximal.tf` | **you** | editing it; scaffolded once | -| `internal/provider/{resources,datasources,list_resources,actions}.go` | toolkit | `emit` | -| `internal/provider/{provider,interfaces}.go`, `internal/client/` | **you** | editing them — auth is always bespoke | -| `internal/services/common/{convert,crud,errors,schema}/` | **you** | editing them | - -Generated files carry exactly this header, and nothing else may: +provider. Generated files carry exactly this header, and nothing else may: ```go // Code generated by tfpluginframeworkgen from blueprints/ (sha256:…). DO NOT EDIT. ``` -There is deliberately **no** preserved-region mechanism inside a generated file: -ownership is all-or-nothing per file. A file that genuinely cannot be generated is -scaffolded once and then yours, and `emit` never touches it again. - -[`docs/generated-boundary.md`](docs/generated-boundary.md) covers how the boundary -is enforced, orphan detection, and the escape hatch. - -## Repository layout +Broadly: CRUD, models, schemas, registration, acceptance tests and **both** +fixture files are generated; authentication, the client, plan modifiers and +read-back predicates are yours — scaffolded once where declared, then never +touched again. There is deliberately **no** preserved-region mechanism inside a +generated file: ownership is all-or-nothing per file. -| Path | Contents | -|---|---| -| `cmd/tfpluginframeworkgen/` | the one installable binary; stdlib `flag` subcommand dispatch | -| `internal/blueprint/` | the IR, its validation, and the layered-merge engine | -| `internal/ingest/` | OpenAPI → blueprint | -| `internal/interop/` | reads and writes `terraform-plugin-codegen-spec` v0.1 | -| `internal/probe/` | the API behaviour prober | -| `internal/cassette/` | HTTP record/replay, redaction, deterministic canonicalisation | -| `internal/emit/`, `internal/render/` | blueprint → Go; all logic lives in `render` | -| `internal/sdkbind/` | type-checks blueprint bindings against the pinned SDK | -| `internal/manifest/` | what the last run produced, so orphaned files can be found | -| `internal/templates/` | embedded `.tmpl` files — the emitted shape, as reviewable text | -| `blueprints/` | committed blueprints, one directory per provider | -| `probe-evidence/` | committed probe cassettes and derived facts | -| `openapi-specs/` | pinned, immutable specification snapshots | -| `pilot/thousandeyes/` | a nested module: a fully generated provider, built and unit-tested in CI | -| `docs/` | architecture, CLI reference, and the new-API onboarding runbook | +[`docs/generated-boundary.md`](docs/generated-boundary.md) has the full ownership +table, the five enforcement mechanisms, and the escape hatch. ## Relationship to HashiCorp's code generation tooling @@ -161,6 +166,23 @@ interop format: `tfpluginframeworkgen interop` reads and writes v0.1 JSON, so `tfplugingen-openapi` output can be ingested and the schema slice can be handed to other tools. Everything that format cannot carry — CRUD wiring, SDK binding, observed behaviour, test scaffolding — lives in this project's own richer IR. +CI feeds the committed export to HashiCorp's real `tfplugingen-framework` on +every PR, as a conformance oracle that does not share this repository's +assumptions ([docs/interop.md](docs/interop.md)). + +## Documentation + +| Doc | What it covers | +|---|---| +| [docs/architecture.md](docs/architecture.md) | the pipeline, package map, and where logic may live | +| [docs/onboarding-a-new-api.md](docs/onboarding-a-new-api.md) | the end-to-end runbook, walked ~20 times | +| [docs/cli.md](docs/cli.md) | every subcommand, flag and exit code | +| [docs/blueprint.md](docs/blueprint.md) | the IR, field by field | +| [docs/probing.md](docs/probing.md) | the probe catalogue, sandbox gate, ledger, budgets | +| [docs/fixtures-and-rehearsal.md](docs/fixtures-and-rehearsal.md) | fixture derivation and the rehearsal fixpoint | +| [docs/generated-boundary.md](docs/generated-boundary.md) | what is generated, what is yours, how it is enforced | +| [docs/gates.md](docs/gates.md) | every CI gate and its local reproduction | +| [docs/interop.md](docs/interop.md) | the codegen-spec v0.1 bridge | ## Roadmap @@ -173,22 +195,19 @@ observed behaviour, test scaffolding — lives in this project's own richer IR. | 3 | `terraform-plugin-codegen-spec` v0.1 interop | **done** | | 4 | the prober: record, replay, gating, cleanup | **done** | | 5 | block kinds: data sources, actions, identity, list resources, arbitrary-depth nesting, generated validators, read-after-write, escape hatches | **done** | -| 6 | breadth — ~20 resources, docs, weekly spec refresh | next | -| 7 | a second API, proving nothing is pilot-shaped | | - -Phase 5 was re-scoped against a real 167-resource provider rather than the -original guess, and the reasoning is recorded in -[`docs/blueprint.md`](docs/blueprint.md) where each decision landed. Tests, -mocks and fixtures derived from probe evidence — the original phase 5 line — -moved into phase 6, since the evidence turned out to be more useful for -deciding *schema* than for generating tests. - -**Generated today:** resources, data sources, actions, resource identity, list -resources. `ephemeral` exists as a block kind — so attribute validation knows -that its attributes may not carry a `Default` or plan modifiers — but there is -no way to declare one and no template to emit it. Provider-defined functions, -state upgraders and `statestore` are not modelled at all. None of these appears -in the reference provider, which is why they are last rather than next. +| 6 | breadth: 23 resources across five waves; probe resequencing (rehearsal, static facts, generated fixtures); postcheck; spec-refresh loop; SDK chain validated end to end | **done** | +| 7 | a second API, proving nothing is pilot-shaped | next | + +Three pilot resources are deferred with their reasons recorded in the blueprints: +agent-to-agent, voice and endpoint scheduled tests all need lab hardware +(enterprise or endpoint agents) that a disposable tenant does not have. Read-only +results surfaces were never in scope. The dashboard `layout` attribute is dropped +pending a widgets model. + +**Generated today:** resources, data sources, actions, ephemerals, resource +identity, list resources. Provider-defined functions, state upgrader *bodies* +(the scaffold exists) and `statestore` are not modelled at all; none appears in +the reference provider, which is why they are last rather than next. ## Limitations @@ -208,35 +227,19 @@ in the reference provider, which is why they are last rather than next. the documented set is the wider of the two, so a stale specification surfaces as a real API error carrying the API's own message rather than as a plan failure nobody can work around. Where the prober saw a documented value refused, it stays permitted and the - refusal is named in a comment beside the validator — a value one tenant rejects may be - licence-gated rather than nonexistent. The one case that suppresses the validator - outright is direct evidence of harm: the API accepted a value from *outside* the - documented set, so a `OneOf` would reject configurations it demonstrably takes. -- **Declared bounds become validators too.** A `pattern`, a length, a size or a numeric range - in the specification becomes the framework validator its type provides — and a bound the - framework has no validator for is refused by name rather than emitted as a call to a function - that does not exist. A `pattern` Go's `regexp` cannot compile is reported and dropped: the - generated code would call `regexp.MustCompile` on it and panic at provider start. + refusal is named in a comment beside the validator. +- **The rehearsal bisects one culprit at a time.** A body refused because of two + *interacting* fields exceeds the bisection budget and is recorded as a refusal + note for a human, not silently guessed at. - **The prober cannot learn everything.** Licence-gated behaviour, cross-object constraints, RBAC, production latency, and whether a field is *semantically* a - secret all need a human. The probe plan's deny list is where that boundary is - drawn honestly. + secret all need a human. The probe plan's deny list and the blueprint's + curated hints are where that boundary is drawn honestly. - **`ingest` refuses partial resources by default.** A resource whose CRUD set is incomplete is a curation decision, not something to guess at. -- **Nesting is generated to any depth** the blueprint declares. Two things are refused, - naming the offending attribute: two nested objects that would declare the same Go - identifier, and nesting past ten levels — a runaway guard, since a schema deeper than - that is usually one whose depth is decided at runtime and so is not expressible here. -- **One attribute decision in the pilot is still an unprobed guess.** The prober - settled the server-default question: `color` really does carry one (`#A7EB10`, - corroborated), `access_type` is required by the API so cannot have one, and - `match_type` is not returned on read so a default is not observable. What it did - *not* settle is whether `legacy_id` is integral despite the specification typing - it as a `number`, so it is still generated as a `float64`. The cassette contains - 63 integral observations and no fractional one, which is suggestive but is not a - fact the prober derives — there is no numeric-integrality probe. Adding one - changes the fact protocol, so it belongs in a phase that re-records rather than - in one that replays. +- **There is no numeric-integrality probe.** A field the specification types as + `number` is generated as `float64` even when every recorded observation of it + is integral; the observations are suggestive, not a derived fact. ## Contributing diff --git a/SECURITY.md b/SECURITY.md index ab8a3420..8b41638f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,5 +1,40 @@ # Security Policy -If you discover a security issue in this repo, please submit it through the [GitHub Security Bug Bounty]() +## Reporting a vulnerability -Thanks for helping make GitHub safe for everyone. \ No newline at end of file +Please report suspected vulnerabilities privately via +[GitHub private vulnerability reporting](https://github.com/deploymenttheory/terraform-plugin-framework-codegen/security/advisories/new) +on this repository. Do not open a public issue for anything you believe is a +security problem. You should receive an acknowledgement within a few days; please +allow a reasonable window for a fix before any public disclosure. + +## What is security-sensitive in this repository + +This project probes live APIs with real credentials and commits the resulting +transcripts, so its threat surface is specific and worth stating: + +- **Bearer tokens.** Credentials are read from the environment only + (`TFPFGEN_PROBE_TOKEN`; the generated pilot provider uses + `THOUSANDEYES_BEARER_TOKEN`). No command accepts a token flag, and the sandbox + profile loader refuses a profile containing a credential-shaped value. Anything + that would move a token into a file, a flag, a log line or a committed artefact + is a vulnerability — report it. +- **Committed cassettes.** Probe transcripts under `probe-evidence/` are public + by design. Redaction allowlists rather than denylists, and a recording fails + outright (exit `7`, nothing written) if a credential-shaped value survives. + A redaction bypass — any way a secret can reach a committed cassette — is the + highest-severity bug this repository can have. +- **Sandbox profiles.** `.tfpluginframeworkgen/sandbox/` is gitignored because a + profile carries tenant identifiers. Committed artefacts must never contain + tenant names; opaque numeric identifiers are accepted. +- **Generated fixtures.** The fixture derivation refuses to invent values for + credential-shaped fields, so generated test configurations cannot contain + anything that looks like a secret. Secret-bearing attributes are generated as + sensitive or ephemeral and never written to state. + +## Scope + +The generated pilot provider (`pilot/thousandeyes/`) is not published to the +Terraform registry; issues in it are still welcome here, since its code is this +repository's output. Vulnerabilities in the ThousandEyes API or SDK belong with +their respective owners. diff --git a/cmd/tfpluginframeworkgen/commands.go b/cmd/tfpluginframeworkgen/commands.go index d702a45a..f2aec983 100644 --- a/cmd/tfpluginframeworkgen/commands.go +++ b/cmd/tfpluginframeworkgen/commands.go @@ -32,7 +32,7 @@ var commands = []command{ { name: "ingest", summary: "infer a provider blueprint from an OpenAPI snapshot", - usage: "ingest [-spec FILE] [-out DIR] [-only TAG] [-list]", + usage: "ingest [-spec-root DIR] [-snapshot NAME] [-only TAG] [-list]", run: runIngest, }, { diff --git a/docs/README.md b/docs/README.md index 7c66f97e..de76f9a8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,26 +7,28 @@ Documents are added as the phase that needs them lands, rather than up front as empty stubs — a stub describing an unbuilt design is worse than no document, because it reads as settled. -The consequence is that **outstanding** means the pipeline it documents is built -and the document is owed, which is a different thing from a phase not having -arrived yet. - -| Document | Contents | State | -|---|---|---| -| [`architecture.md`](architecture.md) | the pipeline stages, and where logic is allowed to live | **written** | -| [`cli.md`](cli.md) | command reference, flags and exit codes | **written** | -| [`blueprint.md`](blueprint.md) | the IR: every field, and what it exists to express | **written** | -| [`generated-boundary.md`](generated-boundary.md) | how generated and hand-written code stay apart, and the escape hatch | **written** | -| [`onboarding-a-new-api.md`](onboarding-a-new-api.md) | numbered runbook for taking on a new API end to end | **written** | -| [`interop.md`](interop.md) | reading and writing Provider Code Specification v0.1, and what it cannot carry | **written** | -| [`probing.md`](probing.md) | the probe catalogue, confidence levels, safety model and cleanup guarantees | **written** | -| `pilot-thousandeyes.md` | the pilot: what was generated, and the before/after against the existing provider | Phase 6 | -| `adr/` | decision records, so settled questions are not re-litigated in six months | ongoing | +| Document | Contents | +|---|---| +| [`architecture.md`](architecture.md) | the pipeline stages, and where logic is allowed to live | +| [`onboarding-a-new-api.md`](onboarding-a-new-api.md) | the numbered runbook for taking on a new API end to end | +| [`cli.md`](cli.md) | command reference, flags and exit codes | +| [`blueprint.md`](blueprint.md) | the IR: every field, and what it exists to express | +| [`probing.md`](probing.md) | the probe catalogue, confidence levels, safety model and cleanup guarantees | +| [`fixtures-and-rehearsal.md`](fixtures-and-rehearsal.md) | one fixture derivation, the rehearsal probe, and the fixpoint between them | +| [`generated-boundary.md`](generated-boundary.md) | how generated and hand-written code stay apart, and the escape hatch | +| [`gates.md`](gates.md) | every CI gate, what it proves, and its local reproduction | +| [`interop.md`](interop.md) | reading and writing Provider Code Specification v0.1, and what it cannot carry | +| [`findings/`](findings/) | investigation write-ups whose evidence shaped the toolkit — kept as history | +| [`examples/`](examples/) | starting points, e.g. the sandbox profile | + +The pilot provider documents itself: [`pilot/thousandeyes/README.md`](../pilot/thousandeyes/README.md) +covers what it proves and how to run it, and its `docs/` directory is +tfplugindocs output for the Terraform registry. ## Decisions already taken -Recorded here until `adr/` exists, because they shape everything else and are the -questions most likely to be asked again. +Recorded here because they shape everything else and are the questions most +likely to be asked again. 1. **Provider layer only.** The toolkit generates the Terraform provider. SDKs already exist and are generated elsewhere; this project binds to them. @@ -39,6 +41,17 @@ questions most likely to be asked again. Every transcript is committed, and facts are re-derived from it offline in CI. 4. **The pilot is a new provider, not a migration.** Generating onto a clean slate keeps state-upgrade work off the toolkit's critical path. +5. **Acceptance is confirmation, not discovery.** Everything an acceptance test + would discover is discovered by the probe first — the rehearsal runs the same + lifecycles with the same values before any provider code exists. A red + acceptance run means the evidence is incomplete, and the fix starts with + `probe`. +6. **Generation finishes with the tools that gate it.** `emit` runs the same + compile, docs and formatting checks CI runs, at generation time. A gate that + fires after the commit is just a slower postcheck. +7. **`deny` gates experiments, not sends.** A denied field is never probed, but a + fixture value declared for it still goes into every body that needs it — + otherwise a denied-but-required field would sink every create. ## Conventions worth knowing before reading the code diff --git a/docs/architecture.md b/docs/architecture.md index dda6d876..d59f6313 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,28 +7,30 @@ those boundaries are drawn where they are. ## The pipeline ``` -OpenAPI snapshot ──ingest──┐ - ├──merge──► blueprint.json ──emit──► provider Go tree -live API ────────probe──────┤ + tests, mocks, - │ fixtures, docs -human overrides ────────────┘ +upstream URL ──specs──► OpenAPI snapshot ──ingest──┐ + ├──merge──► blueprint.json ──emit──► provider Go tree +live API ──probe (catalogue + rehearsal fixpoint)───┤ │ + tests, fixtures, +pinned SDK ──bindings -facts-out (static facts)─────┤ │ examples, docs + │ postcheck +human curation (plans, hints, presence) ────────────┘ (build · docs · fmt) ``` Every arrow's output is a committed, reviewable artefact. That is deliberate: a pipeline whose intermediate state lives only in memory can only be reviewed by -reading its output, and its output is thousands of lines of generated Go. +reading its output, and its output is thousands of lines of generated Go. And +every arrow has a drift gate in CI — see [gates.md](gates.md). Every stage is built, and each is a subcommand of the one binary. | Stage | Package | State | |---|---|---| -| `specs` — fetch and pin an OpenAPI snapshot | `internal/specstore` | **half-built** — read, list and checksum-verify work; the fetch is a stub, so snapshots are pinned by hand | -| `ingest` — OpenAPI → blueprint | `internal/ingest/openapi` | **built** | -| `probe` — live API → behaviour facts | `internal/probe` | **built** (record and replay) | -| `merge` — fold facts into a blueprint | `internal/blueprint/merge` | **built** | -| `emit` — blueprint → provider | `internal/emit`, `internal/render`, `internal/templates` | **built** | +| `specs` — fetch and pin an OpenAPI snapshot | `internal/specstore` | **built** — the refresh loop re-fetches from the latest snapshot's recorded source | +| `ingest` — OpenAPI → blueprint + plan drafts | `internal/ingest/openapi` | **built** | +| `probe` — live API → behaviour facts | `internal/probe` | **built** (record, replay, verify, sweep; the rehearsal fixpoint) | +| `merge` — fold facts into a blueprint | `internal/blueprint/merge` | **built** (plus plan promotion) | +| `emit` — blueprint → provider, postchecked | `internal/emit`, `internal/render`, `internal/templates` | **built** | | `verify` — fail on drift | `cmd/tfpluginframeworkgen/verify.go`, `internal/manifest` | **built** | -| `bindings` — type-check bindings against the SDK | `internal/sdkbind` | **built** | +| `bindings` — type-check bindings against the SDK; derive static facts | `internal/sdkbind` | **built** | | `interop` — Provider Code Specification v0.1 | `internal/interop` | **built** (export; import writes drafts) | ## Where logic is allowed to live @@ -61,21 +63,35 @@ declarations, which the file template merely lists. cmd/tfpluginframeworkgen/ CLI. stdlib flag, one FlagSet per subcommand, no cobra. internal/ blueprint/ the IR: types, validation, canonical JSON, layered load - render/ ALL logic. Blueprint -> finished strings. + render/ ALL emit logic. Blueprint -> finished strings. templates/ embedded .tmpl. The emitted shape, as reviewable text. - emit/ plan, format, write. Owns gofumpt and overwrite refusal. + emit/ plan, format, write, postcheck. Owns gofumpt and overwrite refusal. manifest/ what the last run produced, so orphans can be found naming/ identifiers. One word-splitter, several joins. sdkbind/ type-checks bindings against the SDK actually pinned -- every kind that has one: resources, data sources, - list facets and actions. It walked resources alone until - actions landed, so three quarters went unchecked + list facets and actions. Also derives static facts + (zero-value unsendable) from the SDK's struct tags. + fixturespec/ ONE derivation of acceptance-fixture values, rendered + as HCL by render and as wire JSON by probe + ingest/openapi/ OpenAPI document -> draft blueprints + plan worksheets + specstore/ pinned OpenAPI snapshots: list, read, checksum, pin + probe/ the probe catalogue, gate, ledger, sweeper, budgets + cassette/ transcripts: record, redact, replay, freeze + interop/ codegen-spec v0.1 export and import version/ the tool version, in exactly one place ``` `naming` and `blueprint` depend on nothing else in the repo. `render` depends on -`blueprint` and `naming`. `emit` depends on `render` and `templates`. Nothing -depends on `cmd`. +`blueprint`, `naming` and `fixturespec`. `emit` depends on `render` and +`templates`. Nothing depends on `cmd`. + +Two behaviours live in `cmd` rather than in a stage package, on purpose. The +rehearsal fixpoint (`rehearse.go`) alternates `probe` and `merge` until fixture +derivation converges, and plan promotion (`promote.go`) copies plan values into +blueprint hints — both need packages from opposite ends of the pipeline, and the +probe must never import the package that interprets its output. The command layer +is the only place allowed to see both at once. ## Determinism @@ -133,4 +149,8 @@ The generator fails loudly rather than emitting something approximate. - [`blueprint.md`](blueprint.md) — the IR, field by field - [`generated-boundary.md`](generated-boundary.md) — what is generated, what is yours, and how the two are kept apart +- [`probing.md`](probing.md) — the probe catalogue, gate, ledger and budgets +- [`fixtures-and-rehearsal.md`](fixtures-and-rehearsal.md) — fixture derivation + and the rehearsal fixpoint +- [`gates.md`](gates.md) — every CI gate and its local reproduction - [`cli.md`](cli.md) — command reference diff --git a/docs/blueprint.md b/docs/blueprint.md index a120377a..0ede7d6b 100644 --- a/docs/blueprint.md +++ b/docs/blueprint.md @@ -378,6 +378,13 @@ diagnostic. Constraints live on `AttrType` rather than `Attribute` because they of the type: a collection's **element type carries its own**, and those are lifted onto the collection as `ValueStringsAre(...)` — a bound on a set's elements is not a bound on the set. +`constraints.format` carries the specification's declared string format (`date-time`, `uuid`, +`email`, …). It generates no validator — the framework has none for formats — but it drives +fixture synthesis: a probe or acceptance fixture for a `date-time` field sends a real timestamp +rather than a sentinel string an API would refuse (see +[fixtures-and-rehearsal.md](fixtures-and-rehearsal.md)). Declared numeric bounds are also what +keep the rehearsal's contrast values in range. + Three refusals, each because the framework has no validator to generate: - A bound on a kind that has none — a `pattern` on a number, a length on a collection. @@ -485,6 +492,7 @@ How one attribute crosses the boundary. | `sdkField`, `sdkGoType` | the SDK model's field and its exact declared type | | `expand` | Terraform → SDK | | `flatten` | SDK → Terraform | +| `updateExpand` | overrides `expand` for the update body only — see below | | `skipExpand` | a computed field is read, never sent | | `skipFlatten` | a write-only secret must not be flattened, or state blanks on every read | @@ -493,6 +501,21 @@ the helper returns diagnostics, so `constructResource` does, so its CRUD call si changes. That propagation is computed in `render`, so a resource with only infallible scalars still gets the simpler signatures. +An `expand` or `flatten` is a `ConvertCall`, and three small fields on it absorb +the SDK shape differences that would otherwise need bespoke converters: + +| Field | Purpose | +|---|---| +| `deref` | wrap the call in `convert.Deref`, for an SDK field declared as a value where the converter yields a pointer | +| `takesAddress` | pass the argument by address — the flatten-side mirror of `deref` | +| `cast` | wrap the finished call in a Go conversion, for an SDK field whose type is a named alias | +| `imports` | packages the call's expression needs | + +`updateExpand` exists because a generated SDK sometimes types the *same* field +differently per operation — the live case is a `string` on create and a `*string` +on update. One converter cannot satisfy both bodies, so the update body may name +its own; absent, `expand` serves both. + ## Binding The part the official specification has nothing for. The design goal is that a @@ -515,6 +538,14 @@ methods look different should be a blueprint change, not an emitter change. error return in the generated body. Getting it wrong produces code that does not compile — `tfpluginframeworkgen bindings` checks it against the method's real signature. +An argument of kind `literal` carries an `expr` — a verbatim Go expression — +plus the `imports` that expression references. It exists for constant request +options: an expansion-gated read (`teclient.WithQueryParam("expand", "filters")`) +needs the option repeated everywhere the read is generated, including the +acceptance test helper, where the SDK's client package must be aliased because +the helper's own `client` variable shadows it. A literal with no expression is +refused where it appears — there is nothing to repeat. + ## Policy `updateStyle` is **required** when a resource has an update operation, and @@ -530,6 +561,16 @@ What the API actually does, as opposed to what its document claims: `writable`, `immutable`, `serverDefault`, `returnedOnRead`, `volatile`, `requiredByApi`, and the observed value sets (`acceptedValues`, `rejectedValues`, `valuesClosed`). +The rehearsal (see [fixtures-and-rehearsal.md](fixtures-and-rehearsal.md)) added +five more: + +| Field | Meaning | +|---|---| +| `returnedOnCreate` / `returnedOnUpdate` | false for a field sent on that hop and absent or null in its response — distinct from `returnedOnRead`, because write responses and reads genuinely differ | +| `forcedValue` | the value the API substitutes no matter what is sent; the fixture derivation then sends exactly this, because rehearsing anything else rehearses a lie | +| `updateDefault` | the value assigned when an update omits the field — kept separate from `serverDefault` because the pilot's APIs default differently on create and update | +| `zeroValueUnsendable` | the SDK's wire encoding cannot express the zero value at all; derived statically from struct tags, not probed (see [probing.md](probing.md#static-facts)) | + Populated by `merge` from probe facts — the pilot's tag blueprint carries values recorded from a live run — and consumed by `render`: an observed normalisation or a `returnedOnRead: false` changes which assertions the generated acceptance test @@ -571,6 +612,51 @@ The semantics are a contract between merge (which writes variants) and emission with its condition stated — applying it unconditionally is the bug this structure exists to prevent. +## Acceptance-test curation: `accFixture`, `sweep`, `skipUnlessEnv` + +Three resource-level fields curate what the generated acceptance tests do, for +the knowledge no derivation can have. + +**`accFixture`** holds curated fixture values (the full derivation story is in +[fixtures-and-rehearsal.md](fixtures-and-rehearsal.md)): + +```jsonc +"accFixture": { + "dataBlocks": ["data \"thousandeyes_agents\" \"test\" {}"], + "values": [ + { "attr": "agent_id", "hcl": "data.thousandeyes_agents.test.agents[0].agent_id" }, + { "attr": "filters", "hcl": "[{ key = \"platform\", mode = \"in\", values = [\"windows\"] }]", + "wire": [{ "key": "platform", "mode": "in", "values": ["windows"] }] }, + { "attr": "minimum_sources_pct", "omit": true } + ] +} +``` + +- `dataBlocks` are emitted verbatim above the resource block, so a hint can + reference live tenant data instead of a literal that goes stale. +- A value's `hcl` is the fixture's right-hand side, verbatim and never salted. + `wire` is the same value in wire-typed form, which the rehearsal sends; a data + reference has no wire form, a literal does. `source` marks a hint promoted from + a probe plan (`merge -promote-plans`), which refreshes its own hints on + re-merge and never touches hand-written ones. +- `omit: true` is a **curated omission**: the attribute appears in no generated + fixture and no rehearsal body. For values that are individually valid and + jointly refused — a rule takes `minimumSources` or `minimumSourcesPct` but + never both. The attribute stays configurable; the maximal fixture just cannot + carry every combination. Recording the omission makes it a decision instead of + an accident. + +**`sweep`** overrides the prober's name-field inference (`nameField`, +`readNameField`). It lives on the resource rather than the probe plan because +`probe -mode sweep` builds its subject from the blueprint alone — a sweep is +exactly the situation where no plan may be to hand, and the field that finds +stranded objects cannot depend on one. + +**`accTest.skipUnlessEnv`** names an environment variable the generated test +requires; unset, the test skips with a message naming it. The pilot gates its +admin-scoped resources on `TFPFGEN_ACC_ADMIN` this way, so a contributor without +an admin token gets skips rather than failures. + ## Validation `Validate` reports **every** problem, not the first. Fixing a blueprint one error diff --git a/docs/cli.md b/docs/cli.md index a5fe015f..4fe071b1 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -11,17 +11,20 @@ global flag namespace for two subcommands to collide in, and `tfpluginframeworkg ## Commands +Listed in pipeline order — the order an author walks them, which is also the order +`help` prints them. + | Command | Purpose | State | |---|---|---| -| `specs` | fetch and snapshot an upstream OpenAPI document | planned | +| `specs` | fetch and snapshot an upstream OpenAPI document | **built** | | `ingest` | infer a provider blueprint from an OpenAPI snapshot | **built** *(resources only; data sources, actions and the provider block are hand-authored)* | | `blueprint` | validate, diff or list blueprints | planned *(validation itself is built and runs on every load)* | -| `probe` | exercise a resource's lifecycle; record, replay, verify or sweep | **built** *(both tiers; a mutating run needs `--allow-mutations` and a sandbox profile)* | +| `probe` | exercise a resource's lifecycle; record, replay, verify or sweep | **built** *(both tiers plus the rehearsal; a mutating run needs `--allow-mutations` and a sandbox profile)* | | `merge` | fold probe facts into a blueprint | **built** | -| `emit` | render a provider from blueprints | **built** | +| `emit` | render a provider from blueprints, then postcheck it | **built** | | `verify` | fail if the committed provider has drifted | **built** | -| `bindings` | check blueprint SDK bindings against the pinned SDK | **built** | | `scaffold` | write a blank resource from the archetype | planned *(the archetype exists; `emit` scaffolds it via `hooks`)* | +| `bindings` | check blueprint SDK bindings against the pinned SDK; derive static facts | **built** | | `interop` | export or import codegen-spec v0.1 JSON | **built** *(import is resources-only and writes drafts)* | | `version` | print the version | **built** | @@ -30,6 +33,13 @@ says so and exits non-zero. Registering the full surface up front means `help` describes the intended pipeline from the first commit, and reaching for a missing stage gets a straight answer rather than "unknown command". +Two pieces of pipeline behaviour live *inside* commands rather than as commands of +their own, and it is worth saying so because the source layout suggests otherwise: +the **rehearsal fixpoint** (`cmd/tfpluginframeworkgen/rehearse.go`) is a behaviour +of `probe -mode record`, and **plan promotion** (`cmd/tfpluginframeworkgen/promote.go`) +is a behaviour of `merge -promote-plans`. There is no `rehearse` or `promote` +subcommand. + ## Global flags Accepted by every subcommand. @@ -74,9 +84,193 @@ the errors happened to be joined in. --- +## `specs` + +Fetches the upstream OpenAPI document and pins it as a new snapshot. + +``` +tfpluginframeworkgen specs [-url URL] [-output-dir DIR] [-dry-run] +``` + +| Flag | Default | Purpose | +|---|---|---| +| `-url` | the latest snapshot's recorded source | the document to fetch | +| `-output-dir` | — | snapshot root, e.g. `openapi-specs/thousandeyes` (required) | +| `-dry-run` | `false` | fetch and report, but pin nothing | + +The refresh loop is the point: with `-url` omitted, the source comes from the +latest snapshot's own metadata, so `specs -output-dir openapi-specs/thousandeyes` +re-fetches from wherever the last pin came from. No one has to remember the URL, +because the snapshot that would go stale is the thing that records it. `-url` is +only required for the very first snapshot. + +A snapshot is a directory named `-t` holding the document +(`api.yaml`) and `metadata.json` (source URL, digest, fetch time). An upstream +document identical to the latest snapshot pins nothing and exits `0` — the weekly +refresh should be quiet when there is nothing to review. + +The fetch is bounded at two minutes: a specification is a couple of megabytes, and +anything slower is a network problem worth hearing about. + +## `ingest` + +Infers draft blueprints (and optionally probe-plan worksheets) from a pinned +OpenAPI snapshot. + +``` +tfpluginframeworkgen ingest [-spec-root DIR] [-snapshot NAME] [-only TAG] [-list] +``` + +| Flag | Default | Purpose | +|---|---|---| +| `-spec-root` | `openapi-specs/thousandeyes` | directory holding pinned snapshots | +| `-snapshot` | the newest | snapshot to read | +| `-spec` | — | read this document directly, bypassing the snapshot store | +| `-list` | `false` | list what the document offers and exit | +| `-only` | — | restrict to candidates whose tag or key contains this; comma-separate several | +| `-all` | `false` | include candidates that cannot become resources or data sources | +| `-out` | — | write inferred blueprints under this directory | +| `-plan-drafts` | — | also scaffold a `KEY.probe.plan.draft.json` worksheet per resource under this directory | +| `-provider` | `thousandeyes` | provider name, which prefixes every resource type | +| `-api-version-dir` | `v7` | version directory generated packages live under | +| `-sdk-service-root` | the ThousandEyes SDK's | import prefix the SDK's service packages live under | +| `-sdk-accessor` | `r.client.API` | expression reaching a service from the resource receiver | + +`ingest -list` is the survey: it reports every candidate the document offers and +why the ineligible ones are ineligible. The write path produces **drafts** — a +schema and a best-guess binding, which a human then curates (presence, hints, +denials, sweep configuration) before the file earns its non-draft name. The +`-plan-drafts` worksheets serve the same role for probe plans: every field listed +with a place to put fixture values and candidates, nothing invented. + +## `probe` + +Exercises a live API and writes down what it observed. + +``` +tfpluginframeworkgen probe [-mode record|replay|verify|sweep] -blueprint DIR + [-resource KEY] [-only PROBE] [-list] [-plan FILE] [-plan-dir DIR] + [--allow-mutations] [-profile FILE] [-force] [-no-rehearse] [-rederive] + [-evidence DIR] [-provider NAME] +``` + +| Flag | Purpose | +|---|---| +| `-mode` | `replay` (default), `record`, `verify` or `sweep` | +| `-blueprint` | blueprint file or directory (required) | +| `-resource` | probe one resource, by blueprint key | +| `-only` | run one probe, by name | +| `-list` | print the catalogue with its worst-case cost, and exit | +| `-plan` | probe plan: the fixtures and candidate values a probe cannot discover | +| `-plan-dir` | directory of per-resource plans, `KEY.probe.plan.json`; defaults to the blueprint directory | +| `--allow-mutations` | permit probes that create, update and delete | +| `-profile` | sandbox profile; defaults to `.tfpluginframeworkgen/sandbox/.json` | +| `-force` | record over evidence that is already committed | +| `-no-rehearse` | skip the rehearsal fixpoint after the standard mutating probes | +| `-rederive` | with `-mode replay`: rewrite `facts.json` from the committed cassette, no network | +| `-evidence` | root of the committed evidence (default `probe-evidence`) | +| `-provider` | provider name for the evidence path; defaults to the blueprint's | + +The four modes: + +- **`record`** talks to the live API and freezes everything it saw — cassette, + facts, the plan and subject as probed, and the rehearsal's derived bodies. +- **`replay`** re-runs the probes against the committed cassette. No network, no + credentials. This is the default because the safe mode should be what you get by + typing less. +- **`verify`** replays and then compares the derived facts against the committed + `facts.json`, failing on any difference — the CI drift gate for evidence. It + compares facts even when the replay reproduces an error the recording ended on, + because a reproduced failure with identical facts is a faithful replay. +- **`sweep`** deletes every leftover object the ledger still holds an intent for, + and nothing else. + +`-resource` and `-only` are separate axes and deliberately not one flag: probing one +resource with the whole catalogue and probing every resource with one protocol are +both things an operator wants. + +A mutating record ends with the **rehearsal**: `write.rehearsal` walks both +lifecycle directions (create minimal → update to maximal → downgrade → delete, and +the reverse), and the command then re-derives fixtures from the merged evidence and +re-runs it until the derived bodies stop changing. The converged bodies are frozen +as `rehearsal.json` beside the cassette. `-no-rehearse` skips this for a cheap +targeted re-record; evidence recorded that way carries no rehearsal facts. See +[fixtures-and-rehearsal.md](fixtures-and-rehearsal.md). + +`-rederive` exists for toolkit upgrades: when fact derivation itself changes, the +committed cassettes are re-read and `facts.json` files are rewritten without +touching the network, so better inference never requires re-probing a live API. + +Credentials come from the environment and nowhere else — `TFPFGEN_PROBE_ENDPOINT` and +`TFPFGEN_PROBE_TOKEN`. A flag would put the token in shell history and in the process +table; the profile is a file that gets written down, and the gate refuses one that +contains the token's value. + +`-list` needs no credentials, no cassettes and no network: + +``` +$ tfpluginframeworkgen probe -blueprint blueprints/thousandeyes -resource tag -list +``` + +A mutating run needs `-mode record`, `--allow-mutations`, and a sandbox profile that +passes every gate condition. A refusal lists all of them at once and exits `3`: + +``` +$ tfpluginframeworkgen probe -blueprint blueprints/example -resource tag \ + -mode record --allow-mutations +tfpluginframeworkgen: mutating probes refused: 5 condition(s) were not met: + - sandbox: the profile does not declare sandbox: true + - sandboxEvidence: sandboxEvidence is 2 characters, and at least 24 are required; … + - namePrefix: namePrefix "tf" is shorter than 8 characters; … + - plan: the plan declares no fixtures; … + - noSnapshotOverwrite: evidence for this plan is already committed; … +``` + +See [probing.md](probing.md) for the gate, the ledger, the sweeper and the budgets. + +## `merge` + +Folds a probe run's facts into the blueprint they describe. + +``` +tfpluginframeworkgen merge -blueprint DIR -facts FILE [-strategy annotate|apply] + [-check] [-accept-conflicts] [-promote-plans DIR] [-snapshot-id ID] + [-github-summary PATH] +``` + +| Flag | Default | Purpose | +|---|---|---| +| `-blueprint` | — | blueprint file or directory (required) | +| `-facts` | — | facts JSON to fold in (required) | +| `-strategy` | `annotate` | `annotate` writes behaviour and descriptions; `apply` may also widen presence | +| `-check` | `false` | write nothing and exit 1 if merging would change anything | +| `-accept-conflicts` | `false` | suppress the conflict exit code; conflicts are still reported and still not applied | +| `-promote-plans` | — | directory of `KEY.probe.plan.json` files whose fixture values are promoted into `accFixture` wire hints, for attributes the generator refuses to derive | +| `-snapshot-id` | the facts file's directory | identifies the evidence in the description marker | +| `-github-summary` | — | append a summary here | + +`annotate` is the default because most of what a probe learns lands in `behaviour` +and in the description marker block, neither of which changes the schema. `apply` +is for the facts that do — a field observed as server-populated widening to +`computed_optional`, for instance — and a fact whose application would *conflict* +with a curated declaration is reported and left alone under either strategy. + +`-check` is the drift gate: CI re-merges every committed facts file and fails if +any blueprint would change, so evidence and blueprint cannot silently diverge. A +facts file with no facts (a rehearsal-only or read-only snapshot) is trivially +reflected and passes. + +`-promote-plans` copies fixture values from the *plan* into `accFixture` wire +hints — but only for attributes the fixture generator refuses to derive itself, +only from the plan's first fixture (later fixtures probe variants, not the +canonical shape), and never over a hand-written hint. Static facts documents merge +through the same command with `-snapshot-id` naming the `static` channel; see +[probing.md](probing.md#static-facts). + ## `emit` -Renders a provider from blueprints. +Renders a provider from blueprints, then runs the postcheck battery over what it +wrote. ``` tfpluginframeworkgen emit -blueprint DIR -out DIR [-only NAME] [-dry-run] @@ -91,6 +285,7 @@ tfpluginframeworkgen emit -blueprint DIR -out DIR [-only NAME] [-dry-run] | `-list` | `false` | list the files that would be written and exit | | `-clean` | `false` | delete files the blueprints no longer produce | | `-force` | `false` | overwrite files that are not marked as generated | +| `-skip-postcheck` | `false` | skip the post-emit battery; for tight inner loops only | Building the plan touches nothing on disk, so `-dry-run` exercises the same code path as a real run rather than approximating it. @@ -106,6 +301,24 @@ unlisted file as an orphan. Files whose content is already identical are reported as unchanged rather than rewritten, so regenerating does not churn modification times. +### The postcheck battery + +After writing, `emit` finishes the job with the same tools that gate the result in +CI, in order: + +1. **`go build ./...`** in the output module — the tree must compile. +2. **`go generate .`** — regenerates `docs/` via tfplugindocs, but only when the + module root carries a `go:generate` directive for it. Schema changes and their + registry docs can never drift apart. +3. **`terraform fmt -recursive`** — formats the generated `.tf` fixtures, and then + *fails if it rewrote anything*, because a rewrite means the generator produced + unformatted HCL and that is a generator bug to fix, not output to keep patching. + +The battery only runs when the output root is a Go module (`go.mod` present), so +emitting into a scratch directory for inspection stays cheap. `-skip-postcheck` +exists for tight inner loops; CI and any run before a commit should keep the +battery on — a skipped battery just moves the same failures to the gates. + ## `verify` Fails if the committed provider no longer matches its blueprints. @@ -139,16 +352,18 @@ assumed. ## `bindings` Type-checks a blueprint's SDK bindings against the SDK the provider will compile -against. +against, and derives the static facts that live in the SDK's own source. ``` -tfpluginframeworkgen bindings -blueprint DIR -module DIR +tfpluginframeworkgen bindings -blueprint DIR -module DIR [-facts-out FILE | -facts-check FILE] ``` | Flag | Default | Purpose | |---|---|---| | `-blueprint` | — | blueprint file or directory (required) | | `-module` | — | a module whose `go.mod` pins the SDK (required) | +| `-facts-out` | — | write statically derived facts (zero-value unsendable) to FILE | +| `-facts-check` | — | re-derive static facts and fail when FILE differs — the drift gate for a committed static facts document | `-module` is normally the generated provider's own directory. The toolkit's module deliberately depends on no provider SDK, so it cannot be used — and resolving the @@ -165,6 +380,15 @@ tag: binding.service.accessor: "r.client.Tags" does not resolve: thousandeyes.Client has no field "Tags" (available: API, Transport) ``` +`-facts-out` scans the request models' struct tags for value-typed fields with +`omitempty` — fields whose zero value the SDK is structurally unable to send — and +writes them as `zeroValueUnsendable` facts for `merge` to fold in. `-facts-check` +is the same derivation as a drift gate: bump the SDK pin and CI tells you if the +facts document is stale. The scan also *warns* about value-typed request structs +that no attribute sets, because such a struct always serialises as `{}` on the +wire; a required struct stays a value by design, so this is a warning and not an +error. See [probing.md](probing.md#static-facts). + ## `interop` Reads and writes HashiCorp's Provider Code Specification v0.1. Two verbs; see @@ -239,64 +463,6 @@ first, collapsed: resources[tag].attributes[*].wire.{sdkField,sdkGoType,expand,flatten} (23) ``` -## `probe` - -Exercises a live API and writes down what it observed. - -``` -tfpluginframeworkgen probe [-mode record|replay|verify|sweep] -blueprint DIR - [-resource KEY] [-only PROBE] [-list] [-plan FILE] - [--allow-mutations] [-profile FILE] [-force] - [-evidence DIR] [-provider NAME] -``` - -| Flag | Purpose | -|---|---| -| `-mode` | `replay` (default), `record`, `verify` or `sweep` | -| `-blueprint` | blueprint file or directory (required) | -| `-resource` | probe one resource, by blueprint key | -| `-only` | run one probe, by name | -| `-list` | print the catalogue with its worst-case cost, and exit | -| `-plan` | probe plan: the fixtures and candidate values a probe cannot discover | -| `--allow-mutations` | permit probes that create, update and delete | -| `-profile` | sandbox profile; defaults to `.tfpluginframeworkgen/sandbox/.json` | -| `-force` | record over evidence that is already committed | -| `-evidence` | root of the committed evidence (default `probe-evidence`) | -| `-provider` | provider name for the evidence path; defaults to the blueprint's | - -`-resource` and `-only` are separate axes and deliberately not one flag: probing one -resource with the whole catalogue and probing every resource with one protocol are -both things an operator wants. - -`replay` is the default because the safe mode should be what you get by typing less. - -Credentials come from the environment and nowhere else — `TFPFGEN_PROBE_ENDPOINT` and -`TFPFGEN_PROBE_TOKEN`. A flag would put the token in shell history and in the process -table; the profile is a file that gets written down, and the gate refuses one that -contains the token's value. - -`-list` needs no credentials, no cassettes and no network: - -``` -$ tfpluginframeworkgen probe -blueprint blueprints/thousandeyes -resource tag -list -``` - -A mutating run needs `-mode record`, `--allow-mutations`, and a sandbox profile that -passes every gate condition. A refusal lists all of them at once and exits `3`: - -``` -$ tfpluginframeworkgen probe -blueprint blueprints/example -resource tag \ - -mode record --allow-mutations -tfpluginframeworkgen: mutating probes refused: 5 condition(s) were not met: - - sandbox: the profile does not declare sandbox: true - - sandboxEvidence: sandboxEvidence is 2 characters, and at least 24 are required; … - - namePrefix: namePrefix "tf" is shorter than 8 characters; … - - plan: the plan declares no fixtures; … - - noSnapshotOverwrite: evidence for this plan is already committed; … -``` - -See [probing.md](probing.md) for the gate, the ledger, the sweeper and the budgets. - ## `version` ``` @@ -305,23 +471,6 @@ tfpluginframeworkgen version [-short] --- -## Worked example - -The pilot, end to end: - -```bash -# what would be written, without writing it -tfpluginframeworkgen emit -blueprint blueprints/thousandeyes -dry-run - -# check every SDK symbol the blueprint names actually exists -tfpluginframeworkgen bindings -blueprint blueprints/thousandeyes -module pilot/thousandeyes - -# generate -tfpluginframeworkgen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes - -# the actual proof -cd pilot/thousandeyes && go build ./... && go test ./... && terraform plan - -# what CI runs -tfpluginframeworkgen verify -blueprint blueprints/thousandeyes -out pilot/thousandeyes -``` +For the full pipeline walked in order with these commands — including the record → +merge → emit loop and the CI gates each stage answers to — see +[onboarding-a-new-api.md](onboarding-a-new-api.md) and [gates.md](gates.md). diff --git a/docs/findings/tag-conditional-structure.md b/docs/findings/tag-conditional-structure.md index 6b40f0c3..8410c938 100644 --- a/docs/findings/tag-conditional-structure.md +++ b/docs/findings/tag-conditional-structure.md @@ -133,10 +133,11 @@ probing them. That is the design working: a valid dynamic body cannot omit eithe know a valid value for them is simply true. Their omission behaviour is still covered by `write.required`, which is where it belongs. -## Consequences for the blueprint, not yet applied +## Consequences for the blueprint, as diagnosed at the time -Each of these is a schema change whose evidence is above, and each changes the probe plan, so they -belong with the re-record rather than before it. +Each of these was a schema change whose evidence is above, and each changed the probe plan, so +they belonged with the re-record rather than before it. All are since applied — see the status +section below. - `type`: must become writable, with its legal value gated on `object_type`. - `assignments`: must become `computed`. It is read-only in the document and discarded in practice. diff --git a/docs/fixtures-and-rehearsal.md b/docs/fixtures-and-rehearsal.md new file mode 100644 index 00000000..83b75f39 --- /dev/null +++ b/docs/fixtures-and-rehearsal.md @@ -0,0 +1,163 @@ +# Fixtures and the rehearsal + +How the toolkit decides what values an acceptance test uses, and how it proves — +before any provider code exists — that the lifecycle those values describe is one +the live API will actually accept. + +## The problem this solves + +The pipeline originally treated acceptance testing as a discovery mechanism: emit a +provider, run its tests against the live API, and fix whatever failed — a +server-populated field here, a refused zero value there, an update that silently +reset a sibling. Every fix was real, but each one was discovered *after* +generation, one failure at a time, against the most expensive oracle available. +That is playing whack-a-mole with a live tenant. + +The correction is sequencing. Everything an acceptance test would discover is now +discovered by the probe, **before emit**, and recorded as facts with evidence. +Acceptance's job is confirmation: a red acceptance run means the evidence is +incomplete or stale, and the fix is better probing — not a hand-edit to generated +code. + +Three pieces make that hold together: + +1. **One derivation** of fixture values, shared by the probe and the generator, so + what was rehearsed is what gets rendered. +2. **The rehearsal probe**, which walks the exact lifecycles the generated tests + will run. +3. **A fixpoint** between the two: derive → rehearse → merge what was learned → + re-derive, until the bodies stop changing. + +## One derivation: `internal/fixturespec` + +`internal/fixturespec` answers, for every attribute of a resource, the question +"what value does the acceptance fixture use?" — exactly once. The probe renders +its answer as wire JSON to send; the generator renders the *same* answer as HCL in +`testdata/minimal.tf` and `testdata/maximal.tf`. Two renderings, one derivation. +Before this package existed those were two independent guesses, and every +disagreement between them was invisible until a live test failed. + +The preference ladder, most trusted first: + +| Source | Where it comes from | +|---|---| +| forced value | `behaviour.forcedValue` — the server demonstrably imposes this value, so sending anything else is rehearsing a lie | +| curated hint | `accFixture` in the blueprint — a human wrote it down, usually because the API's constraint is undiscoverable | +| documented enum | the first enum value, from the schema | +| server default | `behaviour.serverDefault`, when the field must be sent anyway | +| format synthesis | the OpenAPI `format` — see below | +| plain synthesis | a type-appropriate sentinel carrying the run's name stamp | + +Format-aware synthesis exists because a bare sentinel string is refused by any API +that validates the field, and one refused field loses the observation for the whole +body: + +| `format` | Synthesised value | +|---|---| +| `date-time` | `2027-06-01T00:00:00Z` | +| `date` | `2027-06-01` | +| `uuid` | a fixed, obviously-synthetic UUID | +| `email` | a `tfacc`-tagged address | +| `uri` / `hostname` | RFC 2606 reserved names | +| `ipv4` | `192.0.2.1` (TEST-NET-1) | +| `ipv6` | `2001:db8::1` (documentation prefix) | + +And refusal is an answer too. The derivation declines to invent values for +credential-shaped fields (a generated fixture must never contain something that +looks like a secret), for `_id`-suffixed references (an invented identifier points +at nothing), and for pattern-constrained strings it cannot satisfy. A refused +attribute is carried as an explicit `Skipped` entry with the reason, in both +renderings — the generated `maximal.tf` says *why* an attribute is absent, and the +probe knows not to send it. + +## Curated hints: `accFixture` + +When the derivation cannot know the right value, a human writes it into the +blueprint (see [blueprint.md](blueprint.md)): + +- `hcl` — the value as the fixture should render it; +- `wire` — the value as the probe should send it, when the two differ; +- `omit` — the *curated omission*: this attribute must not appear in any fixture, + recorded as a decision rather than left as an accident (the live case: a pair of + fields the API refuses jointly, where the fixture keeps one); +- `source` — provenance, so a promoted hint is distinguishable from a hand-written + one. + +Hints can also be **promoted from probe plans**: `merge -promote-plans DIR` copies +a plan's fixture values into wire hints, but only for attributes the derivation +refuses to derive itself, only from the plan's *first* fixture (later fixtures +exist to probe variants, and promoting one broke a real acceptance test), and never +over a hand-written hint. The plan stays the place where probe-only knowledge +lives; the blueprint accumulates only what generation actually needs. + +## The rehearsal: `write.rehearsal` + +The rehearsal is a mutating probe (see [probing.md](probing.md)) that runs the +derived fixtures through the exact shapes a generated acceptance test uses: + +- **Direction A**: create minimal → update to maximal → downgrade to minimal → + delete. +- **Direction B**: create maximal → downgrade to minimal → delete. + +Both directions, because the failure modes differ: a field with a server default +misbehaves when it is *added* by an update; a field the server force-sets +misbehaves when a *create* tries to set it; a downgrade is where update-resets +show themselves. + +At every hop the response is compared against what was sent, field by field. Three +refinements keep the comparison honest: + +- **Contrast**: before concluding a field is ignored on update, the probe + substitutes a *different* in-bounds value (numeric contrast respects the + declared `minimum`/`maximum` — discovered live when contrasting an HTTP version + of 2 up to an out-of-range 3) and retries an update refused with the contrast + uncontrasted, so a rejected substitute does not masquerade as suppression. +- **Bisection**: a body refused as a whole is re-tried dropping one sibling at a + time (budgeted), so the *culprit* is named instead of the whole body being + written off. Single-culprit by design; two interacting culprits exceed the + budget and are recorded as a refusal note. +- **Read-back with expansions**: every read uses the plan's declared query + expansions, because some APIs omit nested collections from a bare item read — + concluding "not returned" from an unexpanded read was a real false fact. + +## What the rehearsal learns + +Its facts are precisely the ones acceptance tests used to discover the hard way: + +| Fact | Meaning | Merge writes | +|---|---|---| +| `returnedOnCreate` / `returnedOnUpdate` (false) | sent, accepted, absent from that hop's response | description note; state-handling guidance | +| `serverForced` | the server replaces the sent value with its own, consistently | `behaviour.forcedValue` when corroborated — which then feeds the next derivation | +| `updateDefault` / `updateResets` | an update omitting the field resets it to a default | `behaviour.updateDefault`; description note | +| `interactionSuppressed` | returned normally except in the presence of a named sibling | description note naming the sibling | +| `zeroValueUnsendable` | the SDK cannot send the zero value at all | presence recommendation (static; see [probing.md](probing.md#static-facts)) | + +As with every probe, a fact carries its evidence (request indices into the +cassette) and a confidence level, and `merge` only *acts* on corroborated facts — +a single observation annotates, it does not rewrite presence. + +## The fixpoint + +Facts change fixtures: learn that the server forces a value and the right fixture +value *is* that value; learn a zero value is unsendable and the fixture must omit +it. But the rehearsal that produced those facts ran with the old fixtures — so +`probe -mode record` re-derives the bodies from the merged evidence and reruns the +rehearsal until **derivation converges** (bounded rounds; convergence is the +normal case after one). + +The loop lives in the command layer (`cmd/tfpluginframeworkgen/rehearse.go`), not +in the probe, because it needs `merge` — and the probe package must never depend +on the package that interprets its output. The converged bodies are frozen as +`rehearsal.json` in the evidence snapshot, so replay replays the fixpoint's +*outcome* rather than re-deriving from a blueprint that has since moved. + +## `maximal.tf` is generated and policed + +The last piece is ownership. Acceptance fixtures (`testdata/minimal.tf`, +`testdata/maximal.tf`) are **generated** from the same derivation, headered as +generated, and drift-gated like every other emitted file. A hand-maintained +maximal fixture was the original design, and it desynchronised from the rehearsed +bodies within one wave — the whole point of a shared derivation is defeated the +moment one of its renderings is edited by hand. What used to be hand-tuning a +fixture is now a curated `accFixture` hint, which both renderings honour. See +[generated-boundary.md](generated-boundary.md). diff --git a/docs/gates.md b/docs/gates.md new file mode 100644 index 00000000..a2930cbf --- /dev/null +++ b/docs/gates.md @@ -0,0 +1,136 @@ +# The gates + +Every artefact in the pipeline is committed, and every arrow between artefacts is +drift-gated in CI. This page lists each gate, what it proves, and the exact local +command that reproduces it — because the correct response to any red gate is to +run the pipeline stage that owns the artefact, never to hand-edit the artefact +into passing. + +The gates live in five workflows. The job names below are verbatim, so a failing +check on a PR can be matched to its section here. + +## `go | Verify generated code` (`codegen-verify.yml`) + +Five jobs, one per arrow. + +### 🔁 Regenerate and diff + +Re-emits the pilot from the committed blueprints and fails on any `git diff`. +Then builds and tests the toolkit *and* the emitted provider, and asserts the +emitted Go is a gofumpt fixed point. Proves: committed provider ⟵ committed +blueprints, with no hand edits in between. + +```bash +go run ./cmd/tfpluginframeworkgen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes +git diff --exit-code +go build ./... && go test ./... +(cd pilot/thousandeyes && go build ./... && go test ./...) +``` + +Red here means a blueprint changed without re-running `emit`, or a generated file +was edited by hand. Regenerate and commit; the hand edit belongs in the blueprint +or in a hook (see [generated-boundary.md](generated-boundary.md)). + +### 🔗 Verify SDK bindings + +Type-checks every binding in the blueprints against the SDK version the pilot's +`go.mod` pins, and re-derives the static facts document against it. + +```bash +go run ./cmd/tfpluginframeworkgen bindings \ + -blueprint blueprints/thousandeyes \ + -module pilot/thousandeyes \ + -facts-check blueprints/thousandeyes/static.facts.json +``` + +Red here usually means an SDK version bump: a method, model or struct tag moved +underneath the blueprints. Fix the bindings, then regenerate the static facts +(`-facts-out`) and re-merge them. + +### 🔀 Round-trip through tfplugingen-framework + +Exports the blueprints as codegen-spec v0.1, diffs the committed export, and +feeds it to HashiCorp's `tfplugingen-framework` to prove the export is one their +tooling accepts. See [interop.md](interop.md). + +```bash +go run ./cmd/tfpluginframeworkgen interop export \ + -blueprint blueprints/thousandeyes \ + -out interop-specs/thousandeyes/provider-code-spec.json +git diff --exit-code -- interop-specs/ +``` + +### 🔬 Re-derive probe facts offline + +Replays every committed cassette with no network and no credentials, asserts the +re-derived facts equal the committed `facts.json`, then re-merges every facts file +under `-check` to assert the blueprints already reflect the evidence. + +```bash +go run ./cmd/tfpluginframeworkgen probe -blueprint blueprints/thousandeyes -mode verify +for facts in probe-evidence/*/*/*/facts.json; do + go run ./cmd/tfpluginframeworkgen merge \ + -blueprint blueprints/thousandeyes -facts "$facts" -check -accept-conflicts +done +``` + +Red on the first half means fact derivation is no longer a pure function of the +transcript (or evidence was edited); red on the second means evidence was +recorded but never merged. See [probing.md](probing.md). + +### 🌍 Terraform validates the examples + +The only job that runs Terraform itself. Regenerates the registry docs and fails +if they drift, asserts committed HCL is a `terraform fmt` fixed point, then +builds the provider, points Terraform at it with `dev_overrides`, and validates +the committed examples against the real schema. This is the job that catches an +example the generated validators reject. + +```bash +(cd pilot/thousandeyes && go generate . && git diff --exit-code -- docs/) +terraform fmt -check -recursive -diff pilot/ +``` + +The `emit` postcheck battery runs the first two of these at generation time (see +[cli.md](cli.md#the-postcheck-battery)), so this gate should only go red when +someone skipped the battery. + +## The other workflows + +| Workflow | Job | What it proves | +|---|---|---| +| `go \| Unit Tests` | 🧪 Run Unit Tests | the toolkit's own suite, with coverage | +| `go \| Linter` | ✨ Run golangci-lint | toolkit lint | +| `go \| Acceptance tests` | 🌍 Acceptance (live tenant) | the generated provider's full lifecycle against the real API | +| `Lint Codebase` (`linter.yml`) | super-linter | markdown, YAML, and everything else non-Go | +| `dependancy-review.yml` | dependency review | no known-vulnerable dependency lands via PR | +| `release-please.yml` | release-please | versioning and CHANGELOG from conventional commits | +| `pr-title-validation.yml` | PR title | conventional-commit PR titles, safely via `env` | +| `auto-merge-dependabot.yml` | auto-merge | patch-level dependabot PRs merge themselves once green | + +## Acceptance + +Acceptance is deliberately **not** a per-PR gate. It runs weekly and on manual +dispatch, gated on a GitHub environment, with a concurrency group so two live +runs can never fight over the same tenant. It creates and destroys real objects, +so it needs `TF_ACC=1` and the provider's live credentials; admin-scoped +resources additionally gate on `TFPFGEN_ACC_ADMIN`. + +Locally, the equivalent is: + +```bash +cd pilot/thousandeyes +TF_ACC=1 go test ./internal/services/... -run 'TestAcc' -timeout 60m -v +``` + +Acceptance is **confirmation, not discovery**: the probe has already rehearsed +the exact lifecycles these tests run (see +[fixtures-and-rehearsal.md](fixtures-and-rehearsal.md)). A red acceptance run +therefore means the evidence is incomplete or the API changed — the fix starts +with `probe`, not with the generated code. + +## Exit codes + +Every command's exit codes are a contract, listed in [cli.md](cli.md#exit-codes). +The one worth repeating here: `probe`'s precedence is **7 > 5 > 3 > 4 > 6 > 1**, +so a run that both exceeded its budget and left an orphan reports the orphan. diff --git a/docs/generated-boundary.md b/docs/generated-boundary.md index a0825f56..e400a615 100644 --- a/docs/generated-boundary.md +++ b/docs/generated-boundary.md @@ -14,6 +14,7 @@ to do when a file genuinely cannot be generated. | `internal/services/**/{modify_plan,validate}.go` | **you** | editing them; `emit` never touches them again | | `internal/provider/{resources,datasources,list_resources,actions}.go` | toolkit | adding a blueprint, then `emit` | | `internal/services/actions/**/{action,model,invoke}.go` | toolkit | editing the blueprint, then `emit` | +| `internal/services/**/testdata/{minimal,maximal}.tf` | toolkit | editing the blueprint's `accFixture` hints, then `emit` — see the note on fixtures | | `internal/provider/interfaces.go` | **you** | it pins that the generated registries are what satisfy each `ProviderWith*` | | `internal/provider/provider.go` | **you** | editing it — authentication is always bespoke | | `internal/client/` | **you** | editing it | @@ -34,6 +35,15 @@ generator eventually overwrites. It becomes partly generated when a blueprint starts declaring where its SDK comes from, since the SDK requirement and any `replace` for local development are then facts the blueprint holds. +The acceptance fixtures (`testdata/minimal.tf`, `testdata/maximal.tf`) moved from +"yours" to generated, and the reversal is worth recording. A hand-maintained +maximal fixture desynchronised from the bodies the probe had rehearsed within one +wave of resources — the values acceptance ran were no longer the values the +evidence vouched for. Both renderings of a fixture now come from one derivation +(see [fixtures-and-rehearsal.md](fixtures-and-rehearsal.md)); what used to be a +hand edit to the fixture is now an `accFixture` hint in the blueprint, which the +probe honours too. + ## The acceptance harness `internal/acceptance/{check,destroy,exists,testlog,types}` is hand-written and @@ -60,7 +70,7 @@ compiles its equivalent into the shipped binary. An external `_test` package can still reach the exported identifiers of a package's own test files, so nothing is lost and no test scaffolding reaches a release build. -## Four ways the boundary is enforced +## Five ways the boundary is enforced **1. A header on every generated file.** @@ -114,6 +124,16 @@ The workflow then builds and tests **both modules**. The explicit would pass on generated code that does not compile — the exact failure it exists to prevent. +**5. `emit` finishes with the tools that gate it.** + +After writing, `emit` runs the postcheck battery — `go build ./...`, `go generate .` +(tfplugindocs), and `terraform fmt` over the output module (see +[cli.md](cli.md#the-postcheck-battery)). The battery is the boundary enforced at +generation time rather than at review time: an emitted tree that does not compile, +or whose registry docs or fixture formatting would drift, fails on the author's +machine instead of in CI. It exists because both failures happened — the gates +caught them, but a gate that fires after the commit is a slower postcheck. + ## State mapping has exactly one call site `crud.go` reads like this, and the shape is the point rather than a style choice: @@ -254,17 +274,6 @@ before doing that work: ## Onboarding a new API -Steps 1 to 3 and the hand-written `common/` packages happen once per API; 4 onward -repeat per resource. - -1. Pin a specification snapshot and commit it *(Phase 2)*. -2. Write the provider blueprint: name, module, SDK dialect, client type, support - package paths. -3. Hand-write `internal/client`, `internal/provider/provider.go` and - `common/{crud,errors,schema}`. Authentication is always bespoke. -4. Write or infer a resource blueprint. Read it like a code review — it is JSON on - purpose. -5. `tfpluginframeworkgen bindings` to confirm every SDK symbol it names exists. -6. `tfpluginframeworkgen emit -dry-run`, then `emit`. Then `go build`. **Fix the - blueprint, not the Go.** -7. Wire `codegen-verify.yml` into the provider's repository. +The full runbook — from pinning a specification to a green live acceptance run — +lives in [onboarding-a-new-api.md](onboarding-a-new-api.md). The one rule that +belongs on this page as well: at every step, **fix the blueprint, not the Go.** diff --git a/docs/interop.md b/docs/interop.md index 1765231d..0a8389e9 100644 --- a/docs/interop.md +++ b/docs/interop.md @@ -95,8 +95,9 @@ conventions, specification provenance, and attribute descriptions — see below. Uniform losses are aggregated per resource with a count; selective ones stay addressed per attribute. Wire bindings and model field names exist on *every* attribute, so -reporting them individually makes the pilot's report sixty-four lines of which -forty-five are identical. Aggregating gives twenty, each saying something distinct. +reporting them individually made even the single-resource pilot's report sixty-four +lines of which forty-five were identical — and the pilot now exports twenty-three +resources. Aggregating gives one line per distinct loss. ### Attribute descriptions diff --git a/docs/onboarding-a-new-api.md b/docs/onboarding-a-new-api.md index f40f693e..a407c3b6 100644 --- a/docs/onboarding-a-new-api.md +++ b/docs/onboarding-a-new-api.md @@ -1,10 +1,12 @@ # Onboarding a new API, end to end The numbered runbook for taking an API from "there is a specification and an SDK" to "a -generated provider with live-proven behaviour". Written from two walked paths — the tag -resource (ingest-assisted, fully probed) and the credential pair (hand-authored, probed, -plus an ephemeral) — so every step below has been done at least twice, and the sharp -edges each one found are recorded where they cut. +generated provider with live-proven behaviour". First written from two walked paths — the +tag resource (ingest-assisted, fully probed) and the credential pair (hand-authored, +probed, plus an ephemeral) — and since walked roughly twenty times across five recorded +waves (tests, alerts, dashboards, account management, endpoint labels), so every step +below has been done repeatedly, and the sharp edges each one found are recorded where +they cut. Throughout: the pipeline's order is **author first, record second**. A resource lands in the blueprint before it has evidence; replay and verify note the gap and continue, and @@ -107,14 +109,33 @@ signatures, return arities, response models, every attribute's `sdkField`, for a block kinds, counted per kind. Run it before ever emitting: it turns a pile of identical compile errors in generated code into one message naming the blueprint field to edit. +Then derive the **static facts** — the behaviour written into the SDK's own struct tags +(a value-typed `omitempty` field cannot send its zero value) — and fold them in: + +```sh +tfpluginframeworkgen bindings -blueprint blueprints/PROVIDER -module pilot/PROVIDER \ + -facts-out blueprints/PROVIDER/static.facts.json +tfpluginframeworkgen merge -blueprint blueprints/PROVIDER \ + -facts blueprints/PROVIDER/static.facts.json +``` + +Commit the document; CI re-derives it with `-facts-check`, so an SDK version bump that +changes a struct tag shows up as drift instead of as a silently wrong fixture. Repeat +both commands whenever the SDK pin moves. + ## 6. Emit, build, and wire the hand-written shell ```sh tfpluginframeworkgen emit -blueprint blueprints/PROVIDER -out pilot/PROVIDER -dry-run tfpluginframeworkgen emit -blueprint blueprints/PROVIDER -out pilot/PROVIDER -cd pilot/PROVIDER && go build ./... && go test ./... +cd pilot/PROVIDER && go test ./... ``` +`emit` finishes with the postcheck battery — compile, tfplugindocs regeneration, +`terraform fmt` over the fixtures — so a tree that would fail the CI gates fails on your +machine at generation time. `go test` is still yours to run; the battery proves the tree +is well-formed, not that it is correct. + The hand-written boundary (once per provider, then stable): `main.go`, `internal/provider/{provider,interfaces}.go`, `internal/client/`, `internal/acceptance/**`, `internal/services/common/**`. Two lessons the first live runs @@ -156,6 +177,14 @@ names all unmet conditions at once. What the fixtures decide: - **Refused enum values escalate** into every other declared fixture before being called rejected; a disagreement one gate cleanly partitions becomes a fact per branch. +A mutating record ends with the **rehearsal fixpoint**: the exact lifecycles the +generated acceptance tests will run, with the fixture values the generator will render, +re-derived and re-run until they converge (see +[fixtures-and-rehearsal.md](fixtures-and-rehearsal.md)). Its refusal notes are the +pre-emit signal — a body the API refuses in rehearsal is an acceptance failure you get +to fix *before* any provider code exists, usually with an `accFixture` hint or omission. +`-no-rehearse` skips it for a cheap targeted re-record. + Then prove purity offline, exactly as CI will: ```sh @@ -186,6 +215,19 @@ Reconcile presence changes by hand where the evidence demands them (the findings pattern: a field the API discards becomes `computed`; a gated field becomes writable with its enum converter), then re-run merge until it reports nothing. +Where the fixture generator refuses to derive a value the plan already knows +(`-promote-plans` names the attributes in its output), promote the plan's values into +`accFixture` wire hints instead of retyping them: + +```sh +tfpluginframeworkgen merge -blueprint blueprints/PROVIDER \ + -facts probe-evidence/PROVIDER/THING/SNAPSHOT/facts.json \ + -promote-plans blueprints/PROVIDER +``` + +Promotion only fills gaps — refused attributes, first fixture, hand-written hints always +win — so re-running it is safe. + ## 9. Acceptance, live ```sh @@ -203,9 +245,11 @@ gated, and the only job that creates real objects. ## 10. The gates that keep it honest -Every PR: regenerate-and-diff, build and test toolkit + pilot, bindings, interop -round-trip, offline fact re-derivation with egress blocked, `merge -check`, terraform -validation of examples and fixtures. A resource with no evidence yet is a stated note; -a verify that proved nothing at all fails. When any of these disagrees with you, the -committed artefact is the arbiter — regenerate, re-derive, or re-record; never edit a -generated file or a cassette by hand. +Every PR runs five jobs in `codegen-verify.yml` — 🔁 Regenerate and diff, 🔗 Verify SDK +bindings (including the static-facts drift check), 🔀 Round-trip through +tfplugingen-framework, 🔬 Re-derive probe facts offline (replay-verify with egress +blocked, then `merge -check` over every facts file), and 🌍 Terraform validates the +examples — each with a local reproduction listed in [gates.md](gates.md). A resource +with no evidence yet is a stated note; a verify that proved nothing at all fails. When +any of these disagrees with you, the committed artefact is the arbiter — regenerate, +re-derive, or re-record; never edit a generated file or a cassette by hand. diff --git a/docs/probing.md b/docs/probing.md index fb0ba7c7..c1a15f16 100644 --- a/docs/probing.md +++ b/docs/probing.md @@ -5,9 +5,9 @@ what an API *does* rather than what its specification claims. Everything it conc own evidence and a confidence level, and everything it does to somebody's tenant is bounded, recorded and swept. -> Status: complete. All fifteen probes are implemented, and the pilot's committed evidence is a -> live mutating run against a real sandbox — 112 requests, 45 objects created, 45 removed, 39 facts, -> all of them re-derivable offline from the transcript. +> Status: complete. All sixteen probes are implemented, and the pilot's committed evidence is +> twenty live mutating runs against a real sandbox, recorded across five waves — every one of +> them re-derivable offline from its transcript, and every one gated in CI by replay-verify. ## What the first live run found @@ -27,6 +27,13 @@ were wrong**, and lost the observations of six probes entirely. None of it was v Each of those is now a guard with a regression test. The one that matters most is the identifier lookup: it failed *silently*, in the direction that leaves objects in somebody's tenant. +Later waves kept finding the same class of thing, which is the evidence the approach generalises +rather than having been lucky once: an API version bump started refusing a synthesised HTTP +header the previous version tolerated (caught by the recorded rehearsal, fixed with one plan +hint); a permissionless role create turned out to be a genuine server-side 500; user emails are +globally unique and deletion is what frees them, which reshaped the whole fixture strategy for +that resource. None of those facts is in any specification. + **The evidence the run produced**, all of it re-derivable offline: - `accessType` and `objectType` are **required by the API**, which its own request schema does not @@ -47,12 +54,29 @@ lookup: it failed *silently*, in the direction that leaves objects in somebody's | Tier | What it does | What it needs | |---|---|---| | Read-only | six probes: list shape, read shape, error envelope, volatility, pagination, unknown-parameter tolerance | credentials | -| Mutating | nine probes: writability, update style, read-your-writes, requiredness, server defaults, immutability, enum boundaries, normalisation, write side effects | credentials, `--allow-mutations`, a sandbox profile, and every gate condition | +| Mutating | ten probes: writability, update style, read-your-writes, requiredness, server defaults, immutability, enum boundaries, normalisation, write side effects, and the rehearsal | credentials, `--allow-mutations`, a sandbox profile, and every gate condition | The two are separate Go interfaces with no overlap, so a read-only probe cannot write. That is a property of the type system rather than a convention: `ReadProbe` is handed a session whose only method is `Get`. +### The rehearsal + +`write.rehearsal` is the tenth mutating probe and the last to run, and it is different in kind +from the other nine: they each isolate one behaviour, while the rehearsal walks the **exact +lifecycles the generated acceptance tests will run** — create minimal → update to maximal → +downgrade → delete, and the reverse — with the same derived fixture values the generator will +render into HCL. Every hop's response is compared field-by-field against what was sent, updates +are contrasted against sibling values to tell "suppressed by interaction" from "never returned", +and a jointly-refused body is bisected one sibling at a time to name the culprit. Its facts are +the ones that used to be discovered by a failing acceptance test. + +After the standard probes, `probe -mode record` re-derives the fixtures from the merged evidence +and re-runs the rehearsal until the derived bodies stop changing, then freezes the converged +bodies as `rehearsal.json` beside the cassette. The full contract — derivation, the fact kinds, +and why the fixpoint exists — is in +[fixtures-and-rehearsal.md](fixtures-and-rehearsal.md). + ## Modes ``` @@ -72,7 +96,29 @@ that can change somebody's tenant has to be spelled out. `verify` is the purity gate, and it runs in CI with egress blocked and no credentials. If derivation ever depended on anything outside the transcript — a clock, an environment variable, map iteration order — the committed facts and the replayed facts would differ, and every fact in -the store would be unreproducible. +the store would be unreproducible. Two refinements keep that gate honest rather than brittle: +a replay that reproduces an error the recording itself ended on still has its facts compared +(a faithful reproduction of a recorded failure is a *pass*, not a crash), and `replay -rederive` +rewrites `facts.json` from the committed cassette when the derivation logic itself improves — +so better inference never requires re-probing a live API. + +## What a snapshot freezes + +A committed evidence snapshot is not just the cassette. It freezes everything replay needs to be +a pure function of the directory: + +| File | What it pins | +|---|---| +| `cassette.json` | the ordered transcript, redacted | +| `facts.json` | the derived facts, which `verify` re-derives and compares | +| `plan.json`, `subject.json` | the plan and blueprint-derived subject *as probed*, so later curation cannot silently change what a replay sends | +| `rehearsal.json` | the converged rehearsal bodies, so the fixpoint's outcome replays instead of re-deriving from a blueprint that has since moved | + +Two version marks make old evidence honest rather than wrong. The subject carries an +`evidenceRev`: observations added to the prober after a snapshot was recorded are only asserted +against evidence recorded at a revision that knew to capture them, so upgrading the toolkit never +turns green history red. And a snapshot recorded with `-only` records that filter in its +metadata, so a filtered recording replays filtered rather than failing over probes it never ran. ## Credentials @@ -183,6 +229,14 @@ with no identifier in it — leaves the intent outstanding. A ledger is clean when it *reconciles*, not when it is empty: forty creates and forty deletes is a clean run. +One more resolution exists, because the conservative classification above manufactures phantoms: +a create refused with a 5xx leaves its intent outstanding — correctly, since a 5xx proves +nothing — but if the server genuinely created nothing, that intent blocks every later record run +over an object that does not exist. The sweeper resolves it by **proven absence**: when a +collection read is *complete* (not truncated, not suspiciously round) and the intent's stamped +name is not in it, the object demonstrably does not exist and the intent is marked rejected. +Absence from a partial read proves nothing and resolves nothing. + ### The sweeper's two passes 1. **By identifier**, from the ledger. Cannot be the whole story: an intent that never learned an @@ -300,6 +354,32 @@ A field whose only acceptable value cannot be discovered belongs on `deny`, with understood: every probe that would have sent it emits a note, and whatever the blueprint claims about it stays unprobed. +`deny` gates **experiments, not sends**. A denied field is never probed — never omitted to test +requiredness, never substituted to test immutability — but if a fixture declares a value for it, +that value still goes into every body that needs it. The distinction matters because the two +readings diverge exactly when a denied field is also required: strip it from the rehearsal's +bodies and every create fails, and the run learns nothing about all the *other* fields. + +## Static facts + +Not every fact needs a network. Some behaviour is written into the SDK's own source: a +value-typed request field tagged `omitempty` is a field whose zero value the SDK is +**structurally unable to send** — `false`, `0` and `""` serialise as an omission, whatever the +API would have said about them. Probing for that would be probing the wrong system. + +`bindings -facts-out` derives these `zeroValueUnsendable` facts by scanning the pinned SDK's +struct tags, and writes them as an ordinary facts document — the pilot commits +`blueprints/thousandeyes/static.facts.json` — that `merge` folds in exactly like probe facts. +They live outside the evidence snapshots because their evidence is a source location, not an +interaction, and they get their own drift gate: `bindings -facts-check` re-derives and fails on +any difference, so an SDK version bump that changes a struct tag cannot leave the committed +facts stale. + +The description marker blocks that `merge` writes keep the two provenances apart: live evidence +writes under its snapshot id, static facts under the id `static`, and within each channel the +newest evidence wins. A fact learned from the SDK's source can never be overwritten by — or +masquerade as — one observed on the wire. + ## What a probe may not do Enforced structurally, not by convention. A `go/ast` test asserts each of these: diff --git a/pilot/thousandeyes/README.md b/pilot/thousandeyes/README.md new file mode 100644 index 00000000..763eb6f1 --- /dev/null +++ b/pilot/thousandeyes/README.md @@ -0,0 +1,96 @@ +# The ThousandEyes pilot + +A fully generated [terraform-plugin-framework](https://developer.hashicorp.com/terraform/plugin/framework) +provider for the ThousandEyes v7 API — the proving ground for the +[toolkit at the repository root](../../README.md). Everything the generator +claims to do has to work here first, against a real API, in CI. + +This module is **not published to the Terraform registry**. It exists to be +generated, built, unit-tested and live-tested; publishing is out of scope. + +## What is in here + +| Path | What it is | Owner | +|---|---|---| +| `internal/services/resources/` | 23 resource packages — CRUD, models, schemas, acceptance tests, fixtures | generated | +| `internal/services/datasources/`, `.../actions/`, `.../ephemeral/` | data sources, the action, the ephemeral credential | generated | +| `internal/provider/{resources,datasources,list_resources,actions}.go` | registration | generated | +| `internal/provider/{provider,interfaces}.go`, `internal/client/` | authentication and wiring | hand-written | +| `internal/services/common/{convert,crud,errors,schema}/` | shared helpers the generated code calls | hand-written | +| `internal/acceptance/` | the acceptance harness (exists/destroy checks, logging) | hand-written | +| `docs/` | registry documentation, rendered by tfplugindocs — **do not hand-edit**; `go generate .` rewrites it | +| `examples/` | emit-scaffolded example configurations a human may enrich | + +The full ownership story, including the per-file header that marks generated +code, is in [docs/generated-boundary.md](../../docs/generated-boundary.md). +The one rule: **fix the blueprint, not the Go** — every generated file here is +overwritten by the next `emit`. + +## The surface + +Every resource is backed by a committed blueprint in +[`blueprints/thousandeyes/`](../../blueprints/thousandeyes/), and every probed +resource by evidence in [`probe-evidence/thousandeyes/`](../../probe-evidence/thousandeyes/): +the test lifecycles were rehearsed against the live API before this code was +generated (see [docs/fixtures-and-rehearsal.md](../../docs/fixtures-and-rehearsal.md)). + +Some acceptance tests gate on environment variables rather than running +unconditionally, because a disposable tenant cannot hold every prerequisite: + +| Gate | Covers | +|---|---| +| `TFPFGEN_ACC_ADMIN` | account groups, roles, users — need an admin-scoped token | +| `TFPFGEN_ACC_ENTERPRISE` | agent-to-agent and voice tests — need enterprise agents (lab hardware) | +| `TFPFGEN_ACC_SIP` | SIP server tests — need a reachable SIP target | + +The hardware-gated variables stay unset in CI, so those tests skip with a stated +reason rather than failing. The dashboard `layout` attribute is dropped pending a +widgets model. + +## Running it + +Unit tests (no credentials, no network — mocks and committed fixtures): + +```bash +go test ./... +``` + +Acceptance (creates and destroys real objects in a live tenant — use a +disposable one): + +```bash +export THOUSANDEYES_BEARER_TOKEN=… # the provider's own auth variable +TF_ACC=1 go test -count=1 -p 1 -run TestAcc ./... +``` + +`-p 1` matters: packages otherwise run concurrently against one tenant. In CI, +acceptance is a weekly and on-dispatch workflow gated on a GitHub environment — +see [docs/gates.md](../../docs/gates.md). + +To try the provider against local Terraform configuration, build it and point +Terraform at the binary with `dev_overrides` (no `terraform init`): + +```bash +go build -o /tmp/terraform-provider-thousandeyes . +cat > /tmp/dev.tfrc <<'EOF' +provider_installation { + dev_overrides { "registry.terraform.io/deploymenttheory/thousandeyes" = "/tmp" } + direct {} +} +EOF +TF_CLI_CONFIG_FILE=/tmp/dev.tfrc terraform plan +``` + +## Regenerating + +From the repository root: + +```bash +go run ./cmd/tfpluginframeworkgen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes +``` + +`emit` finishes with the postcheck battery (compile, tfplugindocs, +`terraform fmt`), so the tree it leaves is the tree CI accepts. The SDK version +everything is checked against is pinned in this module's `go.mod` and named in +the provider blueprint; bump both together and re-run `bindings` with +`-facts-check` (see the [onboarding runbook](../../docs/onboarding-a-new-api.md)).